Label Format

TensorBay supports multiple types of labels.

Each Data instance can have multiple types of label.

And each type of label is supported with a specific label class, and has a corresponding subcatalog class.

Table 7 supported label types

supported label types

label classes

subcatalog classes

Classification

Classification

ClassificationSubcatalog

Box2D

LabeledBox2D

Box2DSubcatalog

Box3D

LabeledBox3D

Box3DSubcatalog

Keypoints2D

LabeledKeypoints2D

Keypoints2DSubcatalog

Polygon

LabeledPolygon

PolygonSubcatalog

MultiPolygon

LabeledMultiPolygon

MultiPolygonSubcatalog

RLE

LabeledRLE

RLESubcatalog

Polyline2D

LabeledPolyline2D

Polyline2DSubcatalog

MultiPolyline2D

LabeledMultiPolyline2D

MultiPolyline2DSubcatalog

Sentence

LabeledSentence

SentenceSubcatalog

SemanticMask

SemanticMask

SemanticMaskSubcatalog

InstanceMask

InstanceMask

InstanceMaskSubcatalog

PanopticMask

PanopticMask

PanopticMaskSubcatalog

Common Label Properties

Different types of labels contain different aspects of annotation information about the data. Some are more general, and some are unique to a specific label type.

Three common properties of a label will be introduced first, and the unique ones will be explained under the corresponding type of label.

Take a 2D box label as an example:

>>> from tensorbay.label import LabeledBox2D
>>> box2d_label = LabeledBox2D(
... 10, 20, 30, 40,
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> box2d_label
LabeledBox2D(10, 20, 30, 40)(
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

category

Category is a string indicating the class of the labeled object.

>>> box2d_label.category
'data_category'

attributes

Attributes are the additional information about this data, and there is no limit on the number of attributes.

The attribute names and values are stored in key-value pairs.

>>> box2d_label.attributes
{'attribute_name': 'attribute_value'}

instance

Instance is the unique id for the object inside of the label, which is mostly used for tracking tasks.

>>> box2d_label.instance
"instance_ID"

Common Subcatalog Properties

Before creating a label or adding a label to data, it’s necessary to define the annotation rules of the specific label type inside the dataset. This task is done by subcatalog.

Different label types have different subcatalog classes.

Take Box2DSubcatalog as an example to describe some common features of subcatalog.

>>> from tensorbay.label import Box2DSubcatalog
>>> box2d_subcatalog = Box2DSubcatalog(is_tracking=True)
>>> box2d_subcatalog
Box2DSubcatalog(
   (is_tracking): True
)

tracking information

If the label of this type in the dataset has the information of instance IDs, then the subcatalog should set a flag to show its support for tracking information.

Pass True to the is_tracking parameter while creating the subcatalog, or set the is_tracking attr after initialization.

>>> box2d_subcatalog.is_tracking = True

category information

common category information

If the label of this type in the dataset has category, then the subcatalog should contain all the optional categories.

Each category of a label appeared in the dataset should be within the categories of the subcatalog.

Common category information can be added to the most subcatalogs except for mask subcatalogs.

>>> box2d_subcatalog.add_category(name="cat", description="The Flerken")
>>> box2d_subcatalog.categories
NameList [
  CategoryInfo("cat")
]

CategoryInfo is used to describe a category. See details in CategoryInfo.

mask category information

If the mask label in the dataset has category information, then the subcatalog should contain all the optional mask categories.

MaskCategory information can be added to the mask subcatalog.

Different from common category, mask category information must have category_id which is the pixel value of this category in all mask images.

>>> semantic_mask_subcatalog.add_category(name="cat", category_id=1, description="Ragdoll")
>>> semantic_mask_subcatalog.categories
NameList [
  MaskCategoryInfo("cat")(...)
]

MaskCategoryInfo is used to describe the category information of pixels in the mask image. See details in MaskCategoryInfo.

attributes information

If the label of this type in the dataset has attributes, then the subcatalog should contain all the rules for different attributes.

Each attributes of a label appeared in the dataset should follow the rules set in the attributes of the subcatalog.

Attribute information ca be added to the subcatalog.

>>> box2d_subcatalog.add_attribute(
... name="attribute_name",
... type_="number",
... maximum=100,
... minimum=0,
... description="attribute description"
... )
>>> box2d_subcatalog.attributes
NameList [
  AttributeInfo("attribute_name")(...)
]

AttributeInfo is used to describe the rules of an attributes, which refers to the Json schema method.

See details in AttributeInfo.

Other unique subcatalog features will be explained in the corresponding label type section.

Classification

Classification is to classify data into different categories.

It is the annotation for the entire file, so each data can only be assigned with one classification label.

Classification labels applies to different types of data, such as images and texts.

The structure of one classification label is like:

{
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    }
}

To create a Classification label:

>>> from tensorbay.label import Classification
>>> classification_label = Classification(
... category="data_category",
... attributes={"attribute_name": "attribute_value"}
... )
>>> classification_label
Classification(
  (category): 'data_category',
  (attributes): {...}
)

Classification.category

The category of the entire data file. See category for details.

Classification.attributes

The attributes of the entire data file. See attributes for details.

Note

There must be either a category or attributes in one classification label.

ClassificationSubcatalog

Before adding the classification label to data, ClassificationSubcatalog should be defined.

ClassificationSubcatalog has categories and attributes information, see common category information and attributes information for details.

To add a Classification label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.classification = classification_label

Note

One data can only have one classification label.

Box2D

Box2D is a type of label with a 2D bounding box on an image. It’s usually used for object detection task.

Each data can be assigned with multiple Box2D labels.

The structure of one Box2D label is like:

{
    "box2d": {
        "xmin": <float>
        "ymin": <float>
        "xmax": <float>
        "ymax": <float>
    },
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    },
    "instance": <str>
}

To create a LabeledBox2D label:

>>> from tensorbay.label import LabeledBox2D
>>> box2d_label = LabeledBox2D(
... xmin, ymin, xmax, ymax,
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> box2d_label
LabeledBox2D(xmin, ymin, xmax, ymax)(
  (category): 'category',
  (attributes): {...}
  (instance): 'instance_ID'
)

Box2D.box2d

LabeledBox2D extends Box2D.

To construct a LabeledBox2D instance with only the geometry information, use the coordinates of the top-left and bottom-right vertexes of the 2D bounding box, or the coordinate of the top-left vertex, the height and the width of the bounding box.

>>> LabeledBox2D(10, 20, 30, 40)
LabeledBox2D(10, 20, 30, 40)()
>>> LabeledBox2D.from_xywh(x=10, y=20, width=20, height=20)
LabeledBox2D(10, 20, 30, 40)()

It contains the basic geometry information of the 2D bounding box.

>>> box2d_label.xmin
10
>>> box2d_label.ymin
20
>>> box2d_label.xmax
30
>>> box2d_label.ymax
40
>>> box2d_label.br
Vector2D(30, 40)
>>> box2d_label.tl
Vector2D(10, 20)
>>> box2d_label.area()
400

Box2D.category

The category of the object inside the 2D bounding box. See category for details.

Box2D.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

Box2D.instance

Instance is the unique ID for the object inside of the 2D bounding box, which is mostly used for tracking tasks. See instance for details.

Box2DSubcatalog

Before adding the Box2D labels to data, Box2DSubcatalog should be defined.

Box2DSubcatalog has categories, attributes and tracking information, see common category information, attributes information and tracking information for details.

To add a LabeledBox2D label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.box2d = []
>>> data.label.box2d.append(box2d_label)

Note

One data may contain multiple Box2D labels, so the Data.label.box2d must be a list.

Box3D

Box3D is a type of label with a 3D bounding box on point cloud, which is often used for 3D object detection.

Currently, Box3D labels applies to point data only.

Each point cloud can be assigned with multiple Box3D label.

The structure of one Box3D label is like:

{
    "box3d": {
        "translation": {
            "x": <float>
            "y": <float>
            "z": <float>
        },
        "rotation": {
            "w": <float>
            "x": <float>
            "y": <float>
            "z": <float>
        },
        "size": {
            "x": <float>
            "y": <float>
            "z": <float>
        }
    },
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    },
    "instance": <str>
}

To create a LabeledBox3D label:

>>> from tensorbay.label import LabeledBox3D
>>> box3d_label = LabeledBox3D(
... size=[10, 20, 30],
... translation=[0, 0, 0],
... rotation=[1, 0, 0, 0],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> box3d_label
LabeledBox3D(
  (size): Vector3D(10, 20, 30),
  (translation): Vector3D(0, 0, 0),
  (rotation): quaternion(1.0, 0.0, 0.0, 0.0),
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

Box3D.box3d

LabeledBox3D extends Box3D.

To construct a LabeledBox3D instance with only the geometry information, use the transform matrix and the size of the 3D bounding box, or use translation and rotation to represent the transform of the 3D bounding box.

>>> LabeledBox3D(
... size=[10, 20, 30],
... transform_matrix=[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]],
... )
LabeledBox3D(
  (size): Vector3D(10, 20, 30)
  (translation): Vector3D(0, 0, 0),
  (rotation): quaternion(1.0, -0.0, -0.0, -0.0),
)
>>> LabeledBox3D(
... size=[10, 20, 30],
... translation=[0, 0, 0],
... rotation=[1, 0, 0, 0],
... )
LabeledBox3D(
  (size): Vector3D(10, 20, 30)
  (translation): Vector3D(0, 0, 0),
  (rotation): quaternion(1.0, 0.0, 0.0, 0.0),
)

It contains the basic geometry information of the 3D bounding box.

>>> box3d_label.transform
Transform3D(
  (translation): Vector3D(0, 0, 0),
  (rotation): quaternion(1.0, 0.0, 0.0, 0.0)
)
>>> box3d_label.translation
Vector3D(0, 0, 0)
>>> box3d_label.rotation
quaternion(1.0, 0.0, 0.0, 0.0)
>>> box3d_label.size
Vector3D(10, 20, 30)
>>> box3d_label.volumn()
6000

Box3D.category

The category of the object inside the 3D bounding box. See category for details.

Box3D.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

Box3D.instance

Instance is the unique id for the object inside of the 3D bounding box, which is mostly used for tracking tasks. See instance for details.

Box3DSubcatalog

Before adding the Box3D labels to data, Box3DSubcatalog should be defined.

Box3DSubcatalog has categories, attributes and tracking information, see common category information, attributes information and tracking information for details.

To add a LabeledBox3D label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.box3d = []
>>> data.label.box3d.append(box3d_label)

Note

One data may contain multiple Box3D labels, so the Data.label.box3d must be a list.

Keypoints2D

Keypoints2D is a type of label with a set of 2D keypoints. It is often used for animal and human pose estimation.

Keypoints2D labels mostly applies to images.

Each data can be assigned with multiple Keypoints2D labels.

The structure of one Keypoints2D label is like:

{
    "keypoints2d": [
        { "x": <float>
          "y": <float>
          "v": <int>
        },
        ...
        ...
    ],
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    },
    "instance": <str>
}

To create a LabeledKeypoints2D label:

>>> from tensorbay.label import LabeledKeypoints2D
>>> keypoints2d_label = LabeledKeypoints2D(
... [[10, 20], [15, 25], [20, 30]],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> keypoints2d_label
LabeledKeypoints2D [
  Keypoint2D(10, 20),
  Keypoint2D(15, 25),
  Keypoint2D(20, 30)
](
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

Keypoints2D.keypoints2d

LabeledKeypoints2D extends Keypoints2D.

To construct a LabeledKeypoints2D instance with only the geometry information, The coordinates of the set of 2D keypoints are necessary. The visible status of each 2D keypoint is optional.

>>> LabeledKeypoints2D([[10, 20], [15, 25], [20, 30]])
LabeledKeypoints2D [
  Keypoint2D(10, 20),
  Keypoint2D(15, 25),
  Keypoint2D(20, 30)
]()
>>> LabeledKeypoints2D([[10, 20, 0], [15, 25, 1], [20, 30, 1]])
LabeledKeypoints2D [
  Keypoint2D(10, 20, 0),
  Keypoint2D(15, 25, 1),
  Keypoint2D(20, 30, 1)
]()

It contains the basic geometry information of the 2D keypoints, which can be obtained by index.

>>> keypoints2d_label[0]
Keypoint2D(10, 20)

Keypoints2D.category

The category of the object inside the 2D keypoints. See category for details.

Keypoints2D.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

Keypoints2D.instance

Instance is the unique ID for the object inside of the 2D keypoints, which is mostly used for tracking tasks. See instance for details.

Keypoints2DSubcatalog

Before adding 2D keypoints labels to the dataset, Keypoints2DSubcatalog should be defined.

Besides attributes information, common category information, tracking information in Keypoints2DSubcatalog, it also has keypoints to describe a set of keypoints corresponding to certain categories.

>>> from tensorbay.label import Keypoints2DSubcatalog
>>> keypoints2d_subcatalog = Keypoints2DSubcatalog()
>>> keypoints2d_subcatalog.add_keypoints(
... 3,
... names=["head", "body", "feet"],
... skeleton=[[0, 1], [1, 2]],
... visible="BINARY",
... parent_categories=["cat"],
... description="keypoints of cats"
... )
>>> keypoints2d_subcatalog.keypoints
[KeypointsInfo(
   (number): 3,
   (names): [...],
   (skeleton): [...],
   (visible): 'BINARY',
   (parent_categories): [...]
 )]

KeypointsInfo is used to describe a set of 2D keypoints.

The first parameter of add_keypoints() is the number of the set of 2D keypoints, which is required.

The names is a list of string representing the names for each 2D keypoint, the length of which is consistent with the number.

The skeleton is a two-dimensional list indicating the connection between the keypoints.

The visible is the visible status that limits the v of Keypoint2D. It can only be “BINARY” or “TERNARY”.

See details in Keypoint2D.

The parent_categories is a list of categories indicating to which category the keypoints rule applies.

Mostly, parent_categories is not given, which means the keypoints rule applies to all the categories of the entire dataset.

To add a LabeledKeypoints2D label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.keypoints2d = []
>>> data.label.keypoints2d.append(keypoints2d_label)

Note

One data may contain multiple Keypoints2D labels, so the Data.label.keypoints2d must be a list.

Polygon

Polygon is a type of label with a polygonal region on an image which contains some semantic information. It’s often used for CV tasks such as semantic segmentation.

Each data can be assigned with multiple Polygon labels.

The structure of one Polygon label is like:

{
    "polygon": [
        {
            "x": <float>
            "y": <float>
        },
        ...
        ...
    ],
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    },
    "instance": <str>
}

To create a LabeledPolygon label:

>>> from tensorbay.label import LabeledPolygon
>>> polygon_label = LabeledPolygon(
... [(1, 2), (2, 3), (1, 3)],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> polygon_label
LabeledPolygon [
  Vector2D(1, 2),
  Vector2D(2, 3),
  Vector2D(1, 3)
](
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

Polygon.polygon

LabeledPolygon extends Polygon.

To construct a LabeledPolygon instance with only the geometry information, use the coordinates of the vertexes of the polygonal region.

>>> LabeledPolygon([(1, 2), (2, 3), (1, 3)])
LabeledPolygon [
  Vector2D(1, 2),
  Vector2D(2, 3),
  Vector2D(1, 3)
]()

It contains the basic geometry information of the polygonal region.

>>> polygon_label.area()
0.5

Polygon.category

The category of the object inside the polygonal region. See category for details.

Polygon.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

Polygon.instance

Instance is the unique id for the object inside of the polygonal region, which is mostly used for tracking tasks. See instance for details.

PolygonSubcatalog

Before adding the Polygon labels to data, PolygonSubcatalog should be defined.

PolygonSubcatalog has categories, attributes and tracking information, see common category information, attributes information and tracking information for details.

To add a LabeledPolygon label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.polygon = []
>>> data.label.polygon.append(polygon_label)

Note

One data may contain multiple Polygon labels, so the Data.label.polygon must be a list.

MultiPolygon

MultiPolygon is a type of label with several polygonal regions which contain same semantic information on an image. It’s often used for CV tasks such as semantic segmentation.

Each data can be assigned with multiple MultiPolygon labels.

The structure of one MultiPolygon label is like:

{
    "multiPolygon": [
        [
            {
                "x": <float>
                "y": <float>
            },
            ...
            ...
        ],
        ...
        ...
    ],
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    }
    "instance": <str>
}

To create a LabeledMultiPolygon label:

>>> from tensorbay.label import LabeledMultiPolygon
>>> multipolygon_label = LabeledMultiPolygon(
... [[(1.0, 2.0), (2.0, 3.0), (1.0, 3.0)], [(1.0, 4.0), (2.0, 3.0), (1.0, 8.0)]],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> multipolygon_label
LabeledMultiPolygon [
  Polygon [...],
  Polygon [...]
](
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

MultiPolygon.multi_polygon

LabeledMultiPolygon extends MultiPolygon.

To construct a LabeledMultiPolygon instance with only the geometry information, use the coordinates of the vertexes of polygonal regions.

>>> LabeledMultiPolygon([[[1.0, 4.0], [2.0, 3.7], [7.0, 4.0]],
... [[5.0, 7.0], [6.0, 7.0], [9.0, 8.0]]])
LabeledMultiPolygon [
  Polygon [...],
  Polygon [...]
]()

MultiPolygon.category

The category of the object inside polygonal regions. See category for details.

MultiPolygon.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

MultiPolygon.instance

Instance is the unique id for the object inside of polygonal regions, which is mostly used for tracking tasks. See instance for details.

MultiPolygonSubcatalog

Before adding the MultiPolygon labels to data, MultiPolygonSubcatalog should be defined.

MultiPolygonSubcatalog has categories, attributes and tracking information, see common category information, attributes information and tracking information for details.

To add a LabeledMultiPolygon label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.multi_polygon = []
>>> data.label.multi_polygon.append(multipolygon_label)

Note

One data may contain multiple MultiPolygon labels, so the Data.label.multi_polygon must be a list.

RLE

RLE, Run-Length Encoding, is a type of label with a list of numbers to indicate whether the pixels are in the target region. It’s often used for CV tasks such as semantic segmentation.

Each data can be assigned with multiple RLE labels.

The structure of one RLE label is like:

{
    "rle": [
        int,
        ...
    ]
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    }
    "instance": <str>
}

To create a LabeledRLE label:

>>> from tensorbay.label import LabeledRLE
>>> rle_label = LabeledRLE(
... [8, 4, 1, 3, 12, 7, 16, 2, 9, 2],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> rle_label
LabeledRLE [
  8,
  4,
  1,
  ...
](
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

RLE.rle

LabeledRLE extends RLE.

To construct a LabeledRLE instance with only the rle format mask.

>>> LabeledRLE([8, 4, 1, 3, 12, 7, 16, 2, 9, 2])
LabeledRLE [
  8,
  4,
  1,
  ...
]()

RLE.category

The category of the object inside the region represented by rle format mask. See category for details.

RLE.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

RLE.instance

Instance is the unique id for the object inside the region represented by rle format mask, which is mostly used for tracking tasks. See instance for details.

RLESubcatalog

Before adding the RLE labels to data, RLESubcatalog should be defined.

RLESubcatalog has categories, attributes and tracking information, see common category information, attributes information and tracking information for details.

To add a LabeledRLE label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.rle = []
>>> data.label.rle.append(rle_label)

Note

One data may contain multiple RLE labels, so the Data.label.rle must be a list.

Polyline2D

Polyline2D is a type of label with a 2D polyline on an image. It’s often used for CV tasks such as lane detection.

Each data can be assigned with multiple Polyline2D labels.

The structure of one Polyline2D label is like:

{
    "polyline2d": [
        {
            "x": <float>
            "y": <float>
        },
        ...
        ...
    ],
    "beizerPointTypes": <str>
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    }
    "instance": <str>
}

Note

When the is_beizer_curve is True in the Polyline2DSubcatalog, beizerPointTypes is mandatory, where each character in the string represents the type of the point (“L” represents the vertex and “C” represents the control point) at the corresponding position in the polyline2d list.

To create a LabeledPolyline2D label:

>>> from tensorbay.label import LabeledPolyline2D
>>> polyline2d_label = LabeledPolyline2D(
... [(1, 2), (2, 3)],
... beizer_point_types="LL",
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> polyline2d_label
LabeledPolyline2D [
  Vector2D(1, 2),
  Vector2D(2, 3)
](
  (beizer_point_types): 'LL',
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

Polyline2D.polyline2d

LabeledPolyline2D extends Polyline2D.

To construct a LabeledPolyline2D instance with only the geometry information, use the coordinates of the vertexes of the polyline.

>>> LabeledPolyline2D([[1, 2], [2, 3]])
LabeledPolyline2D [
  Vector2D(1, 2),
  Vector2D(2, 3)
]()

It contains a series of methods to operate on polyline.

>>> polyline_1 = LabeledPolyline2D([[1, 1], [1, 2], [2, 2]])
>>> polyline_2 = LabeledPolyline2D([[4, 5], [2, 1], [3, 3]])
>>> LabeledPolyline2D.uniform_frechet_distance(polyline_1, polyline_2)
3.6055512754639896
>>> LabeledPolyline2D.similarity(polyline_1, polyline_2)
0.2788897449072021

Polyline2D.category

The category of the 2D polyline. See category for details.

Polyline2D.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

Polyline2D.instance

Instance is the unique ID for the 2D polyline, which is mostly used for tracking tasks. See instance for details.

Polyline2DSubcatalog

Before adding the Polyline2D labels to data, Polyline2DSubcatalog should be defined.

Besides common category information, attributes information and tracking information in Polyline2DSubcatalog, it also has is_beizer_curve to describe the type of the polyline.

>>> from tensorbay.label import Polyline2DSubcatalog
>>> polyline2d_subcatalog = Polyline2DSubcatalog(
... is_beizer_curve=True
... )
>>> polyline2d_subcatalog
Polyline2DSubcatalog(
  (is_beizer_curve): True,
  (is_tracking): False
)

The is_beizer_curve is a boolen value indicating whether the polyline is a Bezier curve.

Besides giving the parameters while initializing Polyline2DSubcatalog, it’s also feasible to set them after initialization.

>>> from tensorbay.label import Polyline2DSubcatalog
>>> polyline2d_subcatalog = Polyline2DSubcatalog()
>>> polyline2d_subcatalog.is_beizer_curve = True
>>> polyline2d_subcatalog
Polyline2DSubcatalog(
  (is_beizer_curve): True,
  (is_tracking): False
)

To add a LabeledPolyline2D label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.polyline2d = []
>>> data.label.polyline2d.append(polyline2d_label)

Note

One data may contain multiple Polyline2D labels, so the Data.label.polyline2d must be a list.

MultiPolyline2D

MultiPolyline2D is a type of label with several 2D polylines which belong to the same category on an image. It’s often used for CV tasks such as lane detection.

Each data can be assigned with multiple MultiPolyline2D labels.

The structure of one MultiPolyline2D label is like:

{
    "multiPolyline2d": [
        [
            {
                "x": <float>
                "y": <float>
            },
            ...
            ...
        ],
        ...
        ...
    ],
    "category": <str>
    "attributes": {
        <key>: <value>
        ...
        ...
    }
    "instance": <str>
}

To create a LabeledMultiPolyline2D label:

>>> from tensorbay.label import LabeledMultiPolyline2D
>>> multipolyline2d_label = LabeledMultiPolyline2D(
... [[[1, 2], [2, 3]], [[3, 4], [6, 8]]],
... category="category",
... attributes={"attribute_name": "attribute_value"},
... instance="instance_ID"
... )
>>> multipolyline2d_label
LabeledMultiPolyline2D [
  Polyline2D [...],
  Polyline2D [...]
](
  (category): 'category',
  (attributes): {...},
  (instance): 'instance_ID'
)

MultiPolyline2D.multi_polyline2d

LabeledMultiPolyline2D extends MultiPolyline2D.

To construct a LabeledMultiPolyline2D instance with only the geometry information, use the coordinates of the vertexes of polylines.

>>> LabeledMultiPolyline2D([[[1, 2], [2, 3]], [[3, 4], [6, 8]]])
LabeledMultiPolyline2D [
  Polyline2D [...],
  Polyline2D [...]
]()

MultiPolyline2D.category

The category of the multiple 2D polylines. See category for details.

MultiPolyline2D.attributes

Attributes are the additional information about this object, which are stored in key-value pairs. See attributes for details.

MultiPolyline2D.instance

Instance is the unique ID for the multiple 2D polylines, which is mostly used for tracking tasks. See instance for details.

MultiPolyline2DSubcatalog

Before adding the MultiPolyline2D labels to data, MultiPolyline2DSubcatalog should be defined.

MultiPolyline2DSubcatalog has categories, attributes and tracking information, see common category information, attributes information and tracking information for details.

To add a LabeledMultiPolyline2D label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.multi_polyline2d = []
>>> data.label.multi_polyline2d.append(multipolyline2d_label)

Note

One data may contain multiple MultiPolyline2D labels, so the Data.label.multi_polyline2d must be a list.

Sentence

Sentence label is the transcripted sentence of a piece of audio, which is often used for autonomous speech recognition.

Each audio can be assigned with multiple sentence labels.

The structure of one sentence label is like:

{
    "sentence": [
        {
            "text":  <str>
            "begin": <float>
            "end":   <float>
        }
        ...
        ...
    ],
    "spell": [
        {
            "text":  <str>
            "begin": <float>
            "end":   <float>
        }
        ...
        ...
    ],
    "phone": [
        {
            "text":  <str>
            "begin": <float>
            "end":   <float>
        }
        ...
        ...
    ],
    "attributes": {
        <key>: <value>
        ...
        ...
    }
}

To create a LabeledSentence label:

>>> from tensorbay.label import LabeledSentence
>>> from tensorbay.label import Word
>>> sentence_label = LabeledSentence(
... sentence=[Word("text", 1.1, 1.6)],
... spell=[Word("spell", 1.1, 1.6)],
... phone=[Word("phone", 1.1, 1.6)],
... attributes={"attribute_name": "attribute_value"}
... )
>>> sentence_label
LabeledSentence(
  (sentence): [
    Word(
      (text): 'text',
      (begin): 1.1,
      (end): 1.6
    )
  ],
  (spell): [
    Word(
      (text): 'text',
      (begin): 1.1,
      (end): 1.6
    )
  ],
  (phone): [
    Word(
      (text): 'text',
      (begin): 1.1,
      (end): 1.6
    )
  ],
  (attributes): {
    'attribute_name': 'attribute_value'
  }

Sentence.sentence

The sentence of a LabeledSentence is a list of Word, representing the transcripted sentence of the audio.

Sentence.spell

The spell of a LabeledSentence is a list of Word, representing the spell within the sentence.

It is only for Chinese language.

Sentence.phone

The phone of a LabeledSentence is a list of Word, representing the phone of the sentence label.

Word

Word is the basic component of a phonetic transcription sentence, containing the content of the word, the start and the end time in the audio.

>>> from tensorbay.label import Word
>>> Word("text", 1.1, 1.6)
Word(
  (text): 'text',
  (begin): 1,
  (end): 2
)

sentence, spell, and phone of a sentence label all compose of Word.

Sentence.attributes

The attributes of the transcripted sentence. See attributes information for details.

SentenceSubcatalog

Before adding sentence labels to the dataset, SentenceSubcatalog should be defined.

Besides attributes information in SentenceSubcatalog, it also has is_sample, sample_rate and lexicon. to describe the transcripted sentences of the audio.

>>> from tensorbay.label import SentenceSubcatalog
>>> sentence_subcatalog = SentenceSubcatalog(
... is_sample=True,
... sample_rate=5,
... lexicon=[["word", "spell", "phone"]]
... )
>>> sentence_subcatalog
SentenceSubcatalog(
  (is_sample): True,
  (sample_rate): 5,
  (lexicon): [...]
)
>>> sentence_subcatalog.lexicon
[['word', 'spell', 'phone']]

The is_sample is a boolen value indicating whether time format is sample related.

The sample_rate is the number of samples of audio carried per second. If is_sample is Ture, then sample_rate must be provided.

The lexicon is a list consists all of text and phone.

Besides giving the parameters while initializing SentenceSubcatalog, it’s also feasible to set them after initialization.

>>> from tensorbay.label import SentenceSubcatalog
>>> sentence_subcatalog = SentenceSubcatalog()
>>> sentence_subcatalog.is_sample = True
>>> sentence_subcatalog.sample_rate = 5
>>> sentence_subcatalog.append_lexicon(["text", "spell", "phone"])
>>> sentence_subcatalog
SentenceSubcatalog(
  (is_sample): True,
  (sample_rate): 5,
  (lexicon): [...]
)

To add a LabeledSentence label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.sentence = []
>>> data.label.sentence.append(sentence_label)

Note

One data may contain multiple Sentence labels, so the Data.label.sentence must be a list.

SemanticMask

SemanticMask is a type of label which is usually used for semantic segmentation task.

In TensorBay, the structure of SemanticMask label is unified as follows:

{
    "localPath": <str>
    "info": [
        {
            "categoryId": <int>
            "attributes": {
                <key>: <value>
                ...
                ...
            }
        },
        ...
        ...
    ]
}

local_path is the storage path of the mask image. TensorBay only supports single-channel, gray-scale png images. If the number of categories exceeds 256, the color depth of this image should be 16 bits, otherwise it is 8 bits.

The gray-scale value of the pixel corresponds to the category id of the categories within the SemanticMaskSubcatalog.

Each data can only be assigned with one SemanticMask label.

To create a SemanticMask label:

>>> from tensorbay.label import SemanticMask
>>> semantic_mask_label = SemanticMask(local_path="/semantic_mask/mask_image.png")
>>> semantic_mask_label
SemanticMask("/semantic_mask/mask_image.png")()

SemanticMask.all_attributes

all_attributes is a dictionary that stores attributes for each category. Each attribute is stored in key-value pairs. See attributes for details.

To create all_attributes:

>>> semantic_mask_label.all_attributes = {1: {"occluded": True}, 2: {"occluded": False}}
>>> semantic_mask_label
SemanticMask("/semantic_mask/mask_image.png")(
  (all_attributes): {
    1: {
      'occluded': True
    },
    2: {
      'occluded': False
    }
  }
)

Note

In SemanticMask, the key of all_attributes is category id which should be an integer.

SemanticMaskSubcatalog

Before adding the SemanticMask labels to data, SemanticMaskSubcatalog should be defined.

SemanticMaskSubcatalog has mask categories and attributes, see mask category information and attributes information for details.

To add a SemanticMask label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.semantic_mask = semantic_mask_label

Note

One data can only have one SemanticMask label, See Data.label.semantic_mask for details.

InstanceMask

InstanceMask is a type of label which is usually used for instance segmentation task.

In TensorBay, the structure of InstanceMask label is unified as follows:

{
    "localPath": <str>
    "info": [
        {
            "instanceId": <int>
            "attributes": {
                <key>: <value>
                ...
                ...
            }
        },
        ...
        ...
    ]
}

local_path is the storage path of the mask image. TensorBay only supports single-channel, gray-scale png images. If the number of categories exceeds 256, the color depth of this image should be 16 bits, otherwise it is 8 bits.

There are pixels in the InstanceMask that do not represent the instance, such as backgrounds or borders. This information is written to the categories within the InstanceMaskSubcatalog.

Each data can only be assigned with one InstanceMask label.

To create a InstanceMask label:

>>> from tensorbay.label import InstanceMask
>>> instance_mask_label = InstanceMask(local_path="/instance_mask/mask_image.png")
>>> instance_mask_label
InstanceMask("/instance_mask/mask_image.png")()

InstanceMask.all_attributes

all_attributes is a dictionary that stores attributes for each instance. Each attribute is stored in key-value pairs. See attributes for details.

To create all_attributes:

>>> instance_mask_label.all_attributes = {1: {"occluded": True}, 2: {"occluded": True}}
>>> instance_mask_label
InstanceMask("/instance_mask/mask_image.png")(
  (all_attributes): {
    1: {
      'occluded': True
    },
    2: {
      'occluded': True
    }
  }
)

Note

In InstanceMask, the key of all_attributes is instance id which should be an integer.

InstanceMaskSubcatalog

Before adding the InstanceMask labels to data, InstanceMaskSubcatalog should be defined.

InstanceMaskSubcatalog has mask categories and attributes, see mask category information and attributes information for details.

To add a InstanceMask label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.instance_mask = instance_mask_label

Note

One data can only have one InstanceMask label, See Data.label.instance_mask for details.

PanopticMask

PanopticMask is a type of label which is usually used for panoptic segmentation task.

In TensorBay, the structure of PanopticMask label is unified as follows:

{
    "localPath": <str>
    "info": [
        {
            "instanceId": <int>
            "categoryId": <int>
            "attributes": {
                <key>: <value>
                ...
                ...
            }
        }
        ...
        ...
    ],
}

local_path is the storage path of the mask image. TensorBay only supports single-channel, gray-scale png images. If the number of categories exceeds 256, the color depth of this image should be 16 bits, otherwise it is 8 bits.

The gray-scale value of the pixel corresponds to the category id of the categories within the PanopticMaskSubcatalog.

Each data can only be assigned with one PanopticMask label.

To create a PanopticMask label:

>>> from tensorbay.label import PanopticMask
>>> panoptic_mask_label = PanopticMask(local_path="/panoptic_mask/mask_image.png")
>>> panoptic_mask_label.all_category_ids = {1: 2, 2: 2}
>>> panoptic_mask_label
PanopticMask("/panoptic_mask/mask_image.png")(
  (all_category_ids): {
    1: 2,
    2: 2
  }
)

Note

In PanopticMask, the key and value of all_category_ids are instance id and category id, respectively, which both should be integers.

PanopticMask.all_attributes

all_attributes is a dictionary that stores attributes for each instance. Each attribute is stored in key-value pairs. See attributes for details.

To create all_attributes:

>>> panoptic_mask_label.all_attributes = {1: {"occluded": True}, 2: {"occluded": True}}
>>> panoptic_mask_label
PanopticMask("/panoptic_mask/mask_image.png")(
  (all_category_ids): {
    1: 2,
    2: 2
  },
  (all_attributes): {
    1: {
      'occluded': True
    },
    2: {
      'occluded': True
    }
  }
)

Note

In PanopticMask, the key of all_attributes is instance id which should be integer.

PanopticMaskSubcatalog

Before adding the PanopticMask labels to data, PanopticMaskSubcatalog should be defined.

PanopticMaskSubcatalog has mask categories and attributes, see mask category information and attributes information for details.

To add a PanopticMask label to one data:

>>> from tensorbay.dataset import Data
>>> data = Data("local_path")
>>> data.label.panoptic_mask = panoptic_mask_label

Note

One data can only have one PanopticMask label, See Data.label.panoptic_mask for details.