Datasets

Upload data to C3 once, then mount it in any job. Data moves between C3 storage and the machine over fast links; only the first upload from your laptop is slow.

Workspace or dataset?

c3 deploy already uploads your project folder, so small inputs can simply sit next to your script. Use a dataset instead when the data is:

• large: more than a few hundred megabytes. The workspace is re-hashed on every deploy and staged onto every machine; a dataset is uploaded once.
• shared: used by more than one project, or by a colleague. A dataset mounts into any job; a workspace belongs to one project.
• versioned: something you will update and want to reproduce. Datasets keep every version; a workspace is replaced on each deploy.
• produced by a job: mount /jobs/<job-id> directly rather than downloading and re-uploading. See Artifacts.

If large data must stay in the project folder, exclude it from the upload with .c3ignore (see Project configuration) and upload it as a dataset instead.

Upload

c3 data cp ./my-data/ /datasets/my-data/

Every file is hashed before upload and skipped if you have already uploaded it, so re-uploading a dataset with a few changed files transfers only those files. Only your own earlier uploads count: a file someone else uploaded is transferred again and stored once. Each upload creates a new immutable version.

Large files are sent as 32 MiB parts, several at a time, and each part is retried on a fresh connection if the network drops it. A status line shows bytes sent, rate and retries; a retry is routine and only a part that keeps failing is reported. An interrupted upload resumes: run the same command again and the parts already stored are skipped. Set C3_CLI_MULTIPART_CONCURRENCY (1 to 16, default 4) to change how many parts are in flight.

Mount in a job

Dataset-version mounts require an explicit compatibility choice. Add this block to .c3, then submit with c3 deploy --storage=legacy. For the default JuiceFS pathway, use persistent data resources.

datasets:
  - ref: /datasets/my-data
    mount: /data/my-data

Your script reads the files at the mount path like local files:

data = np.loadtxt("/data/my-data/measurements.csv", delimiter=",")

Rules:

• mount must be absolute. If omitted, it is /data/<dataset-name>.
• ref can be a local directory, such as ./my-data. C3 uploads it as a dataset before submitting.
• ref can be a previous job's results, /jobs/<job-id>. See Artifacts.
• Without a version, a job gets the latest version at the moment it is submitted. Later uploads do not change a queued job.
• The same datasets: block works for autoresearch: every evaluation job of a run mounts the datasets, pinned when the run is submitted.

Versions

Every upload creates a new version. Without a version, a job mounts whichever version is latest when it is submitted. Pin one to make a run reproducible:

c3 data log /datasets/my-data/           # versions, newest first, with their ids

datasets:
  - ref: /datasets/my-data/@a1b2c3d4e5f6   # an id from c3 data log
    mount: /data/my-data

The id is the start of the version's content hash. C3 expands it before submitting and refuses the deploy if it matches no version or more than one. @latest means the same as no version. Coding agents pass the same id in the version field of the MCP deploy tool.

Older versions can be inspected and downloaded the same way:

c3 data ls -l /datasets/my-data/@a1b2c3d4e5f6/
c3 data cp /datasets/my-data/@a1b2c3d4e5f6/ ./

Browse and download

c3 data ls /datasets/                       # all datasets
c3 data ls /datasets/my-data/               # versions
c3 data ls -l /datasets/my-data/@latest/    # files in the latest version
c3 data cp /datasets/my-data/@latest/ ./    # download
c3 data du /datasets/my-data/               # storage used
c3 data rm -r /datasets/my-data/            # delete the whole dataset

Deleting the latest version makes the previous one current. Deleting the last version removes the dataset.

Datasets can also be scoped to a project at /projects/<project>/data/<name>/, with the same commands. The dashboard shows the same data.

How storage works

Datasets, uploaded workspaces and job results all use one content-addressed store. Every file is a blob keyed by its SHA-256 hash; a manifest lists the blobs in each version. Identical files across datasets, jobs and workspaces are stored once, and storage usage is charged on unique bytes. Listings still show every file in every version.

Persistent workspaces and working data

The default JuiceFS pathway keeps workspace and working-data writes between jobs. It is enabled only for qualified runtime offerings. Existing immutable datasets and c3 pull <job> results keep their current meaning.

c3 deploy --storage=juicefs --workspace default
c3 data resource ls my-project
c3 data resource create my-project data training
c3 data resource upload <resource-id> ./training 0
c3 data resource operation <operation-id>
c3 deploy --storage=juicefs --data-resource training=<resource-id>

A checkout records its resource ID, synchronized revision and local content baseline in .c3-runtime-sync-<name>.json. Keep that file locally; it is excluded from uploads. Re-deploying unchanged code reuses the current remote workspace. Local edits conflict if a job changed the workspace after the last synchronization. Fetch an export and reconcile deliberately, or choose a new workspace name. Missing local paths never delete remote checkpoints.

--data-resource name=<resource-id> mounts working data at /data/name with write access; append :ro for read-only access. A resource permits one writer or shared readers. Jobs waiting for the complete resource set do not hold GPUs. Filesystem writes made by a failed job can still persist; failures do not restore an earlier workspace, and an automatic retry after a lost machine continues from whatever the interrupted attempt wrote.

c3 data resource files <resource-id> [prefix] [cursor]
c3 data resource export <resource-id> [prefix]
c3 data resource download <export-operation-id> <new-folder>
c3 data resource rm <resource-id> <relative-path> <expected-revision>
c3 data resource operation <operation-id>

Listings and exports return operation IDs. Inspect the operation to obtain its bounded file listing or immutable export manifest. Download a completed export into a new local directory with c3 data resource download; existing directories are never overwritten. The browser downloads bounded exports as a tar archive. Listings across separately requested pages are not an atomic snapshot. Deletes require an explicit path and current revision. An uncertain operation retains ownership until recovery; retrying with a different operation ID does not bypass it.

The dashboard Data page and resource_* MCP tools use the same resource APIs. MCP deploy accepts storage: "juicefs" and explicit runtime_storage resource revisions. MCP callers import local edits explicitly; a hosted server cannot infer a local checkout's synchronization baseline.