This package is the entry point for the prepared-token reader. The module docstring expressly limits its responsibility to reading already prepared shards, rather than downloading a corpus or tokenizing text source L1–L5.
Export contract
from .dataset import ShardedTokenDataset, TokenDataLoader
__all__ = ["ShardedTokenDataset", "TokenDataLoader"]ShardedTokenDataset validates the manifest and reads uint16 token ranges; TokenDataLoader iterates over causal batches with a resumable cursor. Their I/O, shape and state contracts belong to data/dataset.py, not to this export layer. The explicit __all__ controls star imports; it does not construct a dataset, allocate token buffers or change a cursor.