THCHS-30

This topic describes how to manage the THCHS-30 Dataset, which is a dataset with Sentence label

Authorize a Client Instance

An accesskey is needed to authenticate identity when using TensorBay.

from tensorbay import GAS

ACCESS_KEY = "Accesskey-*****"
gas = GAS(ACCESS_KEY)

Create Dataset

gas.create_dataset("THCHS-30")

Organize Dataset

It takes the following steps to organize the “THCHS-30” dataset by the Dataset instance.

Step 1: Write the Catalog

A Catalog contains all label information of one dataset, which is typically stored in a json file. However the catalog of THCHS-30 is too large, instead of reading it from json file, we read it by mapping from subcatalog that is loaded by the raw file. Check the dataloader below for more details.

Important

See catalog table for more catalogs with different label types.

Step 2: Write the Dataloader

A dataloader is needed to organize the dataset into a Dataset instance.

#!/usr/bin/env python3
#
# Copyright 2021 Graviti. Licensed under MIT License.
#
# pylint: disable=invalid-name
# pylint: disable=missing-module-docstring

import os
from itertools import islice
from typing import List

from ...dataset import Data, Dataset
from ...label import LabeledSentence, SentenceSubcatalog, Word
from .._utility import glob

DATASET_NAME = "THCHS-30"
_SEGMENT_NAME_LIST = ("train", "dev", "test")


def THCHS30(path: str) -> Dataset:
    """Dataloader of the `THCHS-30`_ dataset.

    .. _THCHS-30: http://166.111.134.19:7777/data/thchs30/README.html

    The file structure should be like::

        <path>
            lm_word/
                lexicon.txt
            data/
                A11_0.wav.trn
                ...
            dev/
                A11_101.wav
                ...
            train/
            test/

    Arguments:
        path: The root directory of the dataset.

    Returns:
        Loaded :class:`~tensorbay.dataset.dataset.Dataset` instance.

    """
    dataset = Dataset(DATASET_NAME)
    dataset.catalog.sentence = _get_subcatalog(os.path.join(path, "lm_word", "lexicon.txt"))
    for segment_name in _SEGMENT_NAME_LIST:
        segment = dataset.create_segment(segment_name)
        for filename in glob(os.path.join(path, segment_name, "*.wav")):
            data = Data(filename)
            label_file = os.path.join(path, "data", os.path.basename(filename) + ".trn")
            data.label.sentence = _get_label(label_file)
            segment.append(data)
    return dataset


def _get_label(label_file: str) -> List[LabeledSentence]:
    with open(label_file, encoding="utf-8") as fp:
        labels = ((Word(text=text) for text in texts.split()) for texts in fp)
        return [LabeledSentence(*labels)]


def _get_subcatalog(lexion_path: str) -> SentenceSubcatalog:
    subcatalog = SentenceSubcatalog()
    with open(lexion_path, encoding="utf-8") as fp:
        for line in islice(fp, 4, None):
            subcatalog.append_lexicon(line.strip().split())
    return subcatalog

See Sentence annotation for more details.

Note

Since the THCHS-30 dataloader above is already included in TensorBay, so it uses relative import. However, the regular import should be used when writing a new dataloader.

from tensorbay.dataset import Data, Dataset
from tensorbay.label import LabeledSentence, SentenceSubcatalog, Word

There are already a number of dataloaders in TensorBay SDK provided by the community. Thus, instead of writing, importing an available dataloadert is also feasible.

from tensorbay.opendataset import THCHS30

dataset = THCHS30("path/to/dataset/directory")

Note

Note that catalogs are automatically loaded in available dataloaders, users do not have to write them again.

Important

See dataloader table for dataloaders with different label types.

Visualize Dataset

Optionally, the organized dataset can be visualized by Pharos, which is a TensorBay SDK plug-in. This step can help users to check whether the dataset is correctly organized. Please see Visualization for more details.

Upload Dataset

The organized “THCHS-30” dataset can be uploaded to TensorBay for sharing, reuse, etc.

dataset_client = gas.upload_dataset(dataset, jobs=8)
dataset_client.commit("initial commit")

Similar with Git, the commit step after uploading can record changes to the dataset as a version. If needed, do the modifications and commit again. Please see Version Control for more details.

Read Dataset

Now “THCHS-30” dataset can be read from TensorBay.

dataset = Dataset("THCHS-30", gas)

In dataset “THCHS-30”, there are three Segments: dev, train and test. Get the segment names by listing them all.

dataset.keys()

Get a segment by passing the required segment name.

segment = dataset["dev"]

In the dev segment, there is a sequence of data, which can be obtained by index.

data = segment[0]

In each data, there is a sequence of Sentence annotations, which can be obtained by index.

labeled_sentence = data.label.sentence[0]
sentence = labeled_sentence.sentence
spell = labeled_sentence.spell
phone = labeled_sentence.phone

There is only one label type in “THCHS-30” dataset, which is Sentence. It contains sentence, spell and phone information. See Sentence label format for more details.

Delete Dataset

gas.delete_dataset("THCHS-30")