Label Format¶
TensorBay supports multiple types of labels.
Each Data
instance
can have multiple types of label
.
And each type of label
is supported with a specific label
class,
and has a corresponding subcatalog class.
supported label types |
label classes |
subcatalog classes |
---|---|---|
|
Common Label Properties¶
Different types of labels contain differenct aspects of annotation information about the data. Some are more general, and some are unique to a specific label type.
Three common properties of a label will be introduced first, and the unique ones will be explained under the corresponding type of label.
Take a 2D box label as an example:
>>> from tensorbay.label import LabeledBox2D
>>> label = LabeledBox2D(
... 10, 20, 30, 40,
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> label
LabeledBox2D(10, 20, 30, 40)(
(category): 'category',
(attributes): {...},
(instance): 'instance_ID'
)
category¶
Category is a string indicating the class of the labeled object.
>>> label.category
'data_category'
attributes¶
Attributes are the additional information about this data, and there is no limit on the number of attributes.
The attribute names and values are stored in key-value pairs.
>>> label.attributes
{'attribute_name': 'attribute_value'}
instance¶
Instance is the unique id for the object inside of the label, which is mostly used for tracking tasks.
>>> label.instance
"instance_ID"
Common Subcatalog Properties¶
Before creating a label or adding a label to data, it’s necessary to define the annotation rules of the specific label type inside the dataset. This task is done by subcatalog.
Different label types have different subcatalog classes.
Take Box2DSubcatalog
as an example
to describe some common features of subcatalog.
>>> from tensorbay.label import Box2DSubcatalog
>>> box2d_subcatalog = Box2DSubcatalog(is_tracking=True)
>>> box2d_subcatalog
Box2DSubcatalog(
(is_tracking): True
)
tracking information¶
If the label of this type in the dataset has the information of instance IDs, then the subcatalog should set a flag to show its support for tracking information.
Pass True
to the is_tracking
parameter while creating the subcatalog,
or set the is_tracking
attr after initialization.
>>> box2d_subcatalog.is_tracking = True
category information¶
If the label of this type in the dataset has category, then the subcatalog should contain all the optional categories.
Each category of a label appeared in the dataset should be within the categories of the subcatalog.
Category information can be added to the subcatalog.
>>> box2d_subcatalog.add_category(name="cat", description="The Flerken")
>>> box2d_subcatalog.categories
NameList [
CategoryInfo("cat")
]
CategoryInfo
is used to describe
a category.
See details in CategoryInfo
.
attributes information¶
If the label of this type in the dataset has attributes, then the subcatalog should contain all the rules for different attributes.
Each attributes of a label appeared in the dataset should follow the rules set in the attributes of the subcatalog.
Attribute information ca be added to the subcatalog.
>>> box2d_subcatalog.add_attribute(
... name="attribute_name",
... type_="number",
... maximum=100,
... minimum=0,
... description="attribute description"
... )
>>> box2d_subcatalog.attributes
NameList [
AttributeInfo("attribute_name")(...)
]
AttributeInfo
is used to describe the rules of an
attributes, which refers to the Json schema method.
See details in AttributeInfo
.
Other unique subcatalog features will be explained in the corresponding label type section.
Classification¶
Classification is to classify data into different categories.
It is the annotation for the entire file, so each data can only be assigned with one classification label.
Classification labels applies to different types of data, such as images and texts.
The structure of one classification label is like:
{
"category": <str>
"attributes": {
<key>: <value>
...
...
}
}
To create a Classification
label:
>>> from tensorbay.label import Classification
>>> classification_label = Classification(
... category="data_category",
... attributes={"attribute_name": "attribute_value"}
... )
>>> classification_label
Classification(
(category): 'data_category',
(attributes): {...}
)
Classification.attributes¶
The attributes of the entire data file. See attributes for details.
Note
There must be either a category or attributes in one classification label.
ClassificationSubcatalog¶
Before adding the classification label to data,
ClassificationSubcatalog
should be defined.
ClassificationSubcatalog
has categories and attributes information,
see category information and
attributes information for details.
To add a Classification
label to one data:
>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.classification = classification_label
Note
One data can only have one classification label.
Box2D¶
Box2D is a type of label with a 2D bounding box on an image. It’s usually used for object detection task.
Each data can be assigned with multiple Box2D label.
The structure of one Box2D label is like:
{
"box2d": {
"xmin": <float>
"ymin": <float>
"xmax": <float>
"ymax": <float>
},
"category": <str>
"attributes": {
<key>: <value>
...
...
},
"instance": <str>
}
To create a LabeledBox2D
label:
>>> from tensorbay.label import LabeledBox2D
>>> box2d_label = LabeledBox2D(
... xmin, ymin, xmax, ymax,
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> box2d_label
LabeledBox2D(xmin, ymin, xmax, ymax)(
(category): 'category',
(attributes): {...}
(instance): 'instance_ID'
)
Box2D.box2d¶
LabeledBox2D
extends Box2D
.
To construct a LabeledBox2D
instance with only the geometry
information,
use the coordinates of the top-left and bottom-right vertexes of the 2D bounding box,
or the coordinate of the top-left vertex, the height and the width of the bounding box.
>>> LabeledBox2D(10, 20, 30, 40)
LabeledBox2D(10, 20, 30, 40)()
>>> LabeledBox2D.from_xywh(x=10, y=20, width=20, height=20)
LabeledBox2D(10, 20, 30, 40)()
It contains the basic geometry information of the 2D bounding box.
>>> box2d_label.xmin
10
>>> box2d_label.ymin
20
>>> box2d_label.xmax
30
>>> box2d_label.ymax
40
>>> box2d_label.br
Vector2D(30, 40)
>>> box2d_label.tl
Vector2D(10, 20)
>>> box2d_label.area()
400
Box2D.attributes¶
Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.
Box2D.instance¶
Instance is the unique ID for the object inside of the 2D bounding box, which is mostly used for tracking tasks. See instance for details.
Box2DSubcatalog¶
Before adding the Box2D labels to data,
Box2DSubcatalog
should be defined.
Box2DSubcatalog
has categories, attributes and tracking information,
see category information,
attributes information and
tracking information for details.
To add a LabeledBox2D
label to one data:
>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.box2d = []
>>> data.label.box2d.append(box2d_label)
Note
One data may contain multiple Box2D labels,
so the Data.label.box2d
must be a list.
Box3D¶
Box3D is a type of label with a 3D bounding box on point cloud, which is often used for 3D object detection.
Currently, Box3D labels applies to point data only.
Each point cloud can be assigned with multiple Box3D label.
The structure of one Box3D label is like:
{
"box3d": {
"translation": {
"x": <float>
"y": <float>
"z": <float>
},
"rotation": {
"w": <float>
"x": <float>
"y": <float>
"z": <float>
},
"size": {
"x": <float>
"y": <float>
"z": <float>
}
},
"category": <str>
"attributes": {
<key>: <value>
...
...
},
"instance": <str>
}
To create a LabeledBox3D
label:
>>> from tensorbay.label import LabeledBox3D
>>> box3d_label = LabeledBox3D(
... size=[10, 20, 30],
... translation=[0, 0, 0],
... rotation=[1, 0, 0, 0],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> box3d_label
LabeledBox3D(
(size): Vector3D(10, 20, 30),
(translation): Vector3D(0, 0, 0),
(rotation): quaternion(1.0, 0.0, 0.0, 0.0),
(category): 'category',
(attributes): {...},
(instance): 'instance_ID'
)
Box3D.box3d¶
LabeledBox3D
extends Box3D
.
To construct a LabeledBox3D
instance with only the geometry
information,
use the transform matrix and the size of the 3D bounding box,
or use translation and rotation to represent the transform of the 3D bounding box.
>>> LabeledBox3D(
... size=[10, 20, 30],
... transform_matrix=[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]],
... )
LabeledBox3D(
(size): Vector3D(10, 20, 30)
(translation): Vector3D(0, 0, 0),
(rotation): quaternion(1.0, -0.0, -0.0, -0.0),
)
>>> LabeledBox3D(
... size=[10, 20, 30],
... translation=[0, 0, 0],
... rotation=[1, 0, 0, 0],
... )
LabeledBox3D(
(size): Vector3D(10, 20, 30)
(translation): Vector3D(0, 0, 0),
(rotation): quaternion(1.0, 0.0, 0.0, 0.0),
)
It contains the basic geometry information of the 3D bounding box.
>>> box3d_label.transform
Transform3D(
(translation): Vector3D(0, 0, 0),
(rotation): quaternion(1.0, 0.0, 0.0, 0.0)
)
>>> box3d_label.translation
Vector3D(0, 0, 0)
>>> box3d_label.rotation
quaternion(1.0, 0.0, 0.0, 0.0)
>>> box3d_label.size
Vector3D(10, 20, 30)
>>> box3d_label.volumn()
6000
Box3D.attributes¶
Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.
Box3D.instance¶
Instance is the unique id for the object inside of the 3D bounding box, which is mostly used for tracking tasks. See instance for details.
Box3DSubcatalog¶
Before adding the Box3D labels to data,
Box3DSubcatalog
should be defined.
Box3DSubcatalog
has categories, attributes and tracking information,
see category information,
attributes information and
tracking information for details.
To add a LabeledBox3D
label to one data:
>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.box3d = []
>>> data.label.box3d.append(box3d_label)
Note
One data may contain multiple Box3D labels,
so the Data.label.box3d
must be a list.
Keypoints2D¶
Keypoints2D is a type of label with a set of 2D keypoints. It is often used for animal and human pose estimation.
Keypoints2D labels mostly applies to images.
Each data can be assigned with multiple Keypoints2D labels.
The structure of one Keypoints2D label is like:
{
"keypoints2d": [
{ "x": <float>
"y": <float>
"v": <int>
},
...
...
],
"category": <str>
"attributes": {
<key>: <value>
...
...
},
"instance": <str>
}
To create a LabeledKeypoints2D
label:
>>> from tensorbay.label import LabeledKeypoints2D
>>> keypoints2d_label = LabeledKeypoints2D(
... [[10, 20], [15, 25], [20, 30]],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> keypoints2d_label
LabeledKeypoints2D [
Keypoint2D(10, 20),
Keypoint2D(15, 25),
Keypoint2D(20, 30)
](
(category): 'category',
(attributes): {...},
(instance): 'instance_ID'
)
Keypoints2D.keypoints2d¶
LabeledKeypoints2D
extends
Keypoints2D
.
To construct a LabeledKeypoints2D
instance with only the geometry
information,
The coordinates of the set of 2D keypoints are necessary.
The visible status of each 2D keypoint is optional.
>>> LabeledKeypoints2D([[10, 20], [15, 25], [20, 30]])
LabeledKeypoints2D [
Keypoint2D(10, 20),
Keypoint2D(15, 25),
Keypoint2D(20, 30)
]()
>>> LabeledKeypoints2D([[10, 20, 0], [15, 25, 1], [20, 30, 1]])
LabeledKeypoints2D [
Keypoint2D(10, 20, 0),
Keypoint2D(15, 25, 1),
Keypoint2D(20, 30, 1)
]()
It contains the basic geometry information of the 2D keypoints, which can be obtained by index.
>>> keypoints2d_label[0]
Keypoint2D(10, 20)
Keypoints2D.attributes¶
Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.
Keypoints2D.instance¶
Instance is the unique ID for the object inside of the 2D keypoints, which is mostly used for tracking tasks. See instance for details.
Keypoints2DSubcatalog¶
Before adding 2D keypoints labels to the dataset,
Keypoints2DSubcatalog
should be defined.
Besides attributes information,
category information,
tracking information in
Keypoints2DSubcatalog
,
it also has keypoints
to describe a set of keypoints corresponding to certain categories.
>>> from tensorbay.label import Keypoints2DSubcatalog
>>> keypoints2d_subcatalog = Keypoints2DSubcatalog()
>>> keypoints2d_subcatalog.add_keypoints(
... 3,
... names=["head", "body", "feet"],
... skeleton=[[0, 1], [1, 2]],
... visible="BINARY",
... parent_categories=["cat"],
... description="keypoints of cats"
... )
>>> keypoints2d_subcatalog.keypoints
[KeypointsInfo(
(number): 3,
(names): [...],
(skeleton): [...],
(visible): 'BINARY',
(parent_categories): [...]
)]
KeypointsInfo
is used to describe a set of 2D keypoints.
The first parameter of add_keypoints()
is the number of the set of 2D keypoints, which is required.
The names
is a list of string representing the names for each 2D keypoint,
the length of which is consistent with the number.
The skeleton
is a two-dimensional list indicating the connection between the keypoints.
The visible
is the visible status that limits the
v
of Keypoint2D
.
It can only be “BINARY” or “TERNARY”.
See details in Keypoint2D
.
The parent_categories
is a list of categories indicating to which category the keypoints rule
applies.
Mostly, parent_categories
is not given,
which means the keypoints rule applies to all the categories of the entire dataset.
To add a LabeledKeypoints2D
label to one data:
>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.keypoints2d = []
>>> data.label.keypoints2d.append(keypoints2d_label)
Note
One data may contain multiple Keypoints2D labels,
so the Data.label.keypoints2d
must be a list.
Sentence¶
Sentence label is the transcripted sentence of a piece of audio, which is often used for autonomous speech recognition.
Each audio can be assigned with multiple sentence labels.
The structure of one sentence label is like:
{
"sentence": [
{
"text": <str>
"begin": <float>
"end": <float>
}
...
...
],
"spell": [
{
"text": <str>
"begin": <float>
"end": <float>
}
...
...
],
"phone": [
{
"text": <str>
"begin": <float>
"end": <float>
}
...
...
],
"attributes": {
<key>: <value>,
...
...
}
}
To create a LabeledSentence
label:
>>> from tensorbay.label import LabeledSentence
>>> from tensorbay.label import Word
>>> sentence_label = LabeledSentence(
... sentence=[Word("text", 1.1, 1.6)],
... spell=[Word("spell", 1.1, 1.6)],
... phone=[Word("phone", 1.1, 1.6)],
... attributes={"attribute_name": "attribute_value"}
... )
>>> sentence_label
LabeledSentence(
(sentence): [
Word(
(text): 'text',
(begin): 1.1,
(end): 1.6
)
],
(spell): [
Word(
(text): 'text',
(begin): 1.1,
(end): 1.6
)
],
(phone): [
Word(
(text): 'text',
(begin): 1.1,
(end): 1.6
)
],
(attributes): {
'attribute_name': 'attribute_value'
}
Sentence.sentence¶
The sentence
of a
LabeledSentence
is a list of
Word
,
representing the transcripted sentence of the audio.
Sentence.spell¶
The spell
of a
LabeledSentence
is a list of
Word
,
representing the spell within the sentence.
It is only for Chinese language.
Sentence.phone¶
The phone
of a
LabeledSentence
is a list of
Word
,
representing the phone of the sentence label.
Word¶
Word
is the basic component of a phonetic transcription sentence,
containing the content of the word, the start and the end time in the audio.
>>> from tensorbay.label import Word
>>> Word("text", 1.1, 1.6)
Word(
(text): 'text',
(begin): 1,
(end): 2
)
sentence
,
spell
,
and phone
of a sentence label all compose of
Word
.
Sentence.attributes¶
The attributes of the transcripted sentence. See attributes information for details.
SentenceSubcatalog¶
Before adding sentence labels to the dataset,
SetenceSubcatalog
should be defined.
Besides attributes information in
SetenceSubcatalog
,
it also has is_sample
,
sample_rate
and lexicon
.
to describe the transcripted sentences of the audio.
>>> from tensorbay.label import SentenceSubcatalog
>>> sentence_subcatalog = SentenceSubcatalog(
... is_sample=True,
... sample_rate=5,
... lexicon=[["word", "spell", "phone"]]
... )
>>> sentence_subcatalog
SentenceSubcatalog(
(is_sample): True,
(sample_rate): 5,
(lexicon): [...]
)
>>> sentence_subcatalog.lexicon
[['word', 'spell', 'phone']]
The is_sample
is a boolen value indicating whether time format is sample related.
The sample_rate
is the number of samples of audio carried per second.
If is_sample
is Ture, then sample_rate
must be provided.
The lexicon
is a list consists all of text and phone.
Besides giving the parameters while initialing
SetenceSubcatalog
,
it’s also feasible to set them after initialization.
>>> from tensorbay.label import SentenceSubcatalog
>>> sentence_subcatalog = SentenceSubcatalog()
>>> sentence_subcatalog.is_sample = True
>>> sentence_subcatalog.sample_rate = 5
>>> sentence_subcatalog.append_lexicon(["text", "spell", "phone"])
>>> sentence_subcatalog
SentenceSubcatalog(
(is_sample): True,
(sample_rate): 5,
(lexicon): [...]
)
To add a LabeledSentence
label to one data:
>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.sentence = []
>>> data.label.sentence.append(sentence_label)
Note
One data may contain multiple Sentence labels,
so the Data.label.sentence
must be a list.