Bucket Integrations

Most tools can already use a bucket, in one of two ways:

• Python libraries built on fsspec (pandas, Dask, PyArrow, and others) read and write hf://buckets/ paths through the huggingface_hub filesystem interface.
• Tools that talk to S3 (boto3, the AWS CLI, rclone, DVC, and others) work against the S3-compatible API.

This page covers libraries with built-in bucket support, plus examples for common data libraries. For mounts and volume mounts, see Access Patterns.

pandas

import pandas as pd

df = pd.read_parquet("hf://buckets/username/my-bucket/data.parquet")
df.to_parquet("hf://buckets/username/my-bucket/output.parquet")

Dask

Dask reads and writes lazily, one partition per file, so you can process datasets larger than memory straight from a bucket:

import dask.dataframe as dd

df = dd.read_parquet("hf://buckets/username/my-bucket/data/")
df.to_parquet("hf://buckets/username/my-bucket/output/")

See Dask on the Hub for more.

Daft

Daft supports hf://buckets/ paths natively, with Xet-accelerated reads enabled by default:

import daft
from daft.io import IOConfig, HuggingFaceConfig
from huggingface_hub import get_token

io_config = IOConfig(hf=HuggingFaceConfig(token=get_token()))
df = daft.read_parquet("hf://buckets/username/my-bucket/data.parquet", io_config=io_config)

PyArrow

import pyarrow.parquet as pq

table = pq.read_table("hf://buckets/username/my-bucket/data.parquet")

PySpark

With pyspark_huggingface installed:

df = (
    spark.read.format("huggingface")
    .option("data_files", '["data.parquet"]')
    .load("buckets/username/my-bucket")
)

See PySpark on the Hub for more.

🤗 Datasets

from datasets import load_dataset

ds = load_dataset("buckets/username/my-bucket", data_files=["data.parquet"])

datatrove

datatrove readers and writers accept hf://buckets/ paths. HuggingFaceBucketWriter uploads large outputs with Xet:

from datatrove.pipeline.readers import ParquetReader
from datatrove.pipeline.writers import HuggingFaceBucketWriter

reader = ParquetReader(data_folder="hf://buckets/username/my-bucket/raw/")
writer = HuggingFaceBucketWriter(bucket="username/my-bucket", prefix="filtered")

See the datatrove README for more.

LeRobot

LeRobot can stream a training dataset from a bucket. The bucket must hold the full dataset (meta/, Parquet and video files) at its root:

lerobot-train \
  --dataset.repo_id=username/my-bucket \
  --dataset.repo_type=bucket \
  --dataset.streaming=true \
  --policy.type=act

Diffusers

diffusers-cli run uploads its outputs to a bucket with --push-to. The bucket is created if it does not exist:

diffusers-cli run --model black-forest-labs/FLUX.1-dev --dtype bf16 \
  --pipeline-kwargs '{"prompt": "an astronaut riding a horse"}' \
  --push-to hf://buckets/username/my-bucket/outputs

See the Diffusers CLI guide for more.

Trackio

When Trackio logs to a Space, it stores run data in a bucket created from the Space ID. Pass bucket_id to trackio.init() to use a bucket you choose.

Inspect AI

Inspect AI can write evaluation logs directly to a bucket — point its log directory at an hf://buckets/ path (requires huggingface_hub>=1.6.0). Create the bucket and authenticate first (Inspect won’t create it for you):

hf auth login
hf buckets create username/my-bucket --private

export INSPECT_LOG_DIR=hf://buckets/username/my-bucket/eval-logs
inspect eval popularity.py --model openai/gpt-4
inspect view

See Inspect’s eval logs guide for details.

SkyPilot

SkyPilot runs AI workloads across 20+ clouds, Kubernetes, and on-prem, and can use Hugging Face storage as a backend — so one bucket is readable from every cloud with no per-cloud copies. Set store: hf on a file_mounts entry to mount a bucket read-write or a repo read-only:

# qwen-sft.yaml — launch anywhere: sky launch qwen-sft.yaml --infra aws|gcp|...
resources:
  accelerators: H100:1

file_mounts:
  /base-model:
    source: hf://Qwen/Qwen2.5-3B           # model repo, read-only
    store: hf
    mode: MOUNT
  /data:
    source: hf://datasets/username/my-data@v1.0   # dataset repo, pinned to a tag, read-only
    store: hf
    mode: MOUNT
  /checkpoints:
    source: hf://buckets/username/qwen-sft   # bucket, read-write — checkpoints sync back
    store: hf
    mode: MOUNT

run: |
  python train.py --model /base-model --output_dir /checkpoints

Authenticate once - hf auth login (or export HF_TOKEN=<your-token>) is all SkyPilot needs. It forwards your local Hugging Face token to every cloud, so the bucket and repo mounts authenticate automatically:

pip install "skypilot[huggingface]"
hf auth login                              # or: export HF_TOKEN=<your-token>
sky launch qwen-sft.yaml

If your own run code pulls gated repos, add --secret HF_TOKEN to the launch command to also expose the token as an env var.

MOUNT and MOUNT_CACHED behave identically for hf and use the hf-mount FUSE backend, which needs a base image with glibc ≥ 2.34 and /dev/fuse. Bare-VM clouds provide both. SkyPilot’s default Kubernetes image ships older glibc, so set a newer image_id (e.g. docker:mirror.gcr.io/ubuntu:22.04). See the SkyPilot storage docs for the current environment requirements.

See the SkyPilot + Hugging Face storage blog post for benchmarks and a full walkthrough.

Filesystem operations

For direct file operations, huggingface_hub exposes a pre-instantiated filesystem object, hffs:

from huggingface_hub import hffs

with hffs.open("buckets/username/my-bucket/hello.txt", "w") as f:
    f.write("Hello world!")

hffs.cp("buckets/username/my-bucket/hello.txt", "buckets/username/my-bucket/hello2.txt")
hffs.rm("buckets/username/my-bucket/hello2.txt")
files = hffs.ls("buckets/username/my-bucket")
text_files = hffs.glob("buckets/username/my-bucket/*.txt")

DuckDB

DuckDB 2.0 and later read hf://buckets/ paths natively through the httpfs extension (and with the hf extension for writing), from the CLI and every client:

SELECT * FROM 'hf://buckets/username/my-bucket/data/**/*.parquet' LIMIT 10;
COPY tbl TO 'hf://buckets/username/my-bucket/out.parquet';

See Query with DuckDB for private buckets and older DuckDB versions.

Other languages

OpenDAL provides a similar filesystem interface for Rust, Java, Go, JavaScript, and more.

Coming soon

Native hf:// URL support is on the way for more libraries — including WebDataset. In the meantime, all of these already work today through the S3-compatible API.