Datasets

Upload data to C3 once, then mount it in any job. Data moves between C3 storage and the machine over fast links; only the first upload from your laptop is slow.

Workspace or dataset?

c3 deploy already uploads your project folder, so small inputs can simply sit next to your script. Use a dataset instead when the data is:

• large: more than a few hundred megabytes. The workspace is re-hashed on every deploy and staged onto every machine; a dataset is uploaded once.
• shared: used by more than one project, or by a colleague. A dataset mounts into any job; a workspace belongs to one project.
• versioned: something you will update and want to reproduce. Datasets keep every version; a workspace is replaced on each deploy.
• produced by a job: mount /jobs/<job-id> directly rather than downloading and re-uploading. See Artifacts.

If large data must stay in the project folder, exclude it from the upload with .c3ignore (see Project configuration) and upload it as a dataset instead.

Upload

c3 data cp ./my-data/ /datasets/my-data/

Every file is hashed before upload and skipped if C3 already has it, so re-uploading a dataset with a few changed files transfers only those files. Each upload creates a new immutable version.

Mount in a job

datasets:
  - ref: /datasets/my-data
    mount: /data/my-data

Your script reads the files at the mount path like local files:

data = np.loadtxt("/data/my-data/measurements.csv", delimiter=",")

Rules:

• mount must be absolute. If omitted, it is /data/<dataset-name>.
• ref can be a local directory, such as ./my-data. C3 uploads it as a dataset before submitting.
• ref can be a previous job's results, /jobs/<job-id>. See Artifacts.
• Without a version, a job gets the latest version at the moment it is submitted. Later uploads do not change a queued job.
• The same datasets: block works for autoresearch: every evaluation job of a run mounts the datasets, pinned when the run is submitted.

Versions

Every upload creates a new version. Without a version, a job mounts whichever version is latest when it is submitted. Pin one to make a run reproducible:

c3 data log /datasets/my-data/           # versions, newest first, with their ids

datasets:
  - ref: /datasets/my-data/@a1b2c3d4e5f6   # an id from c3 data log
    mount: /data/my-data

The id is the start of the version's content hash. C3 expands it before submitting and refuses the deploy if it matches no version or more than one. @latest means the same as no version. Coding agents pass the same id in the version field of the MCP deploy tool.

Older versions can be inspected and downloaded the same way:

c3 data ls -l /datasets/my-data/@a1b2c3d4e5f6/
c3 data cp /datasets/my-data/@a1b2c3d4e5f6/ ./

Browse and download

c3 data ls /datasets/                       # all datasets
c3 data ls /datasets/my-data/               # versions
c3 data ls -l /datasets/my-data/@latest/    # files in the latest version
c3 data cp /datasets/my-data/@latest/ ./    # download
c3 data du /datasets/my-data/               # storage used
c3 data rm -r /datasets/my-data/            # delete the whole dataset

Deleting the latest version makes the previous one current. Deleting the last version removes the dataset.

Datasets can also be scoped to a project at /projects/<project>/data/<name>/, with the same commands. The dashboard shows the same data.

How storage works

Datasets, uploaded workspaces and job results all use one content-addressed store. Every file is a blob keyed by its SHA-256 hash; a manifest lists the blobs in each version. Identical files across datasets, jobs and workspaces are stored once, and storage usage is charged on unique bytes. Listings still show every file in every version.