Sextant

TensorBay SDK supports methods to interact with sextant application. See authorized storage instruction for details about how to start.

Authorize a Sextant Instance

from tensorbay.apps.sextant import Sextant

sextant = Sextant("<YOUR_ACCESSKEY>")

List or get benchmark

# list all benchmarks.
benchmarks = sextant.list_benchmarks()

# get benchmark with given name.
benchmark = sextant.get_benchmark("test_01")

Create a evaluation

A evaluation must be created by one commit.

from tensorbay import GAS

gas = GAS("<YOUR_ACCESSKEY>")
dataset_client = gas.get_dataset("dataset_name")
dataset_client.checkout(revision="branch/tag/commitId")
evaluation = benchmark.create_evaluation(dataset_client.dataset_id, dataset_client.status.commit_id)

List all evaluations

evaluations = benchmark.list_evaluations()
evaluation = evaluations[0]

get evaluation result

result = evaluation.get_result()

The details of the result structure for the evaluation are as follows:

{
    "result": {
        "categories": {
            "aeroplane": {
                "AP": 0.3,
                "averageIoU": 0.71,
                "pr": {
                    "precision": [0.87, 0.8, ...],
                    "recall": [0.001, 0.0045, ...]
                }
            },
            ...
        },
        "overall": {
            "averageIoU": 0.7,
            "mAP": 0.2,
            "pr": {
                "precision": [0.95, 0.91, ...],
                "recall": [0.0036, 0.01, ...]
            }
        }
    }
}

Note

Benchmark can only be created with tensorbay website now.