Sextant
TensorBay SDK supports methods to interact with sextant application. See authorized storage instruction for details about how to start.
List or get benchmark
# list all benchmarks.
benchmarks = sextant.list_benchmarks()
# get benchmark with given name.
benchmark = sextant.get_benchmark("test_01")
Create a evaluation
A evaluation must be created by one commit.
from tensorbay import GAS
gas = GAS("<YOUR_ACCESSKEY>")
dataset_client = gas.get_dataset("dataset_name")
dataset_client.checkout(revision="branch/tag/commitId")
evaluation = benchmark.create_evaluation(dataset_client.dataset_id, dataset_client.status.commit_id)
List all evaluations
evaluations = benchmark.list_evaluations()
evaluation = evaluations[0]
get evaluation result
result = evaluation.get_result()
The details of the result structure for the evaluation are as follows:
{
"result": {
"categories": {
"aeroplane": {
"AP": 0.3,
"averageIoU": 0.71,
"pr": {
"precision": [0.87, 0.8, ...],
"recall": [0.001, 0.0045, ...]
}
},
...
},
"overall": {
"averageIoU": 0.7,
"mAP": 0.2,
"pr": {
"precision": [0.95, 0.91, ...],
"recall": [0.0036, 0.01, ...]
}
}
}
}
Note
Benchmark can only be created with tensorbay website now.