Source: Hugging Face Datasets docs: Arrow and Streaming
Datasets library docs on its memory-mapped Apache Arrow cache and on streaming mode (IterableDataset, shuffle buffers, sharding, resumable state).
Contents
- What: library documentation (datasets v5.2.0 docs)
- Publisher: Hugging Face
- Date: accessed 2026-11-14
- Original: Datasets and Arrow, Stream
Summary
The datasets library loads data in one of two ways. Map-style datasets are memory-mapped Arrow tables on disk with random access. Streaming IterableDatasets read lazily from remote or local files without converting them first.
Key claims
- Arrow is a columnar, zero-copy, language-agnostic format.
datasetsmemory-maps its on-disk Arrow cache, so loading English Wikipedia (about 18 GB) costs about 50 MB of RAM, and iterating it runs at multiple Gbit/s on a laptop. load_dataset(..., streaming=True)returns anIterableDataset. This suits data larger than the disk, such as FineWeb’s 45 TB English split. Parquet sources can stream only some columns and push down row filters.- An IterableDataset has no random access, so reaching the last example means iterating through all the earlier ones.
shuffle(buffer_size=...)(default buffer 1,000) samples from a rolling buffer and shuffles shard order, so it is approximate.set_epochreseeds each epoch. Calltakeandskipafter shuffling, because they lock the shard order.to_iterable_dataset(num_shards=...)is faster than streaming for local data and parallelizes across DataLoader workers.state_dict()/load_state_dict()resume iteration at the shard level. Examples sitting in shuffle buffers are lost on resume.mapandfilterapply lazily while streaming.push_to_hubwrites Parquet shards.