> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.prolific.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.prolific.com/_mcp/server.

# Working with Datasets

A dataset contains the data that participants will annotate in a Task Builder Batch. This page covers dataset creation, upload, and advanced configuration options.

For the complete batch workflow, see [Working with Batches](/api-reference/task-builder/batches).

## Dataset versions

There are two dataset versions:

| Version         | Formats          | Upload tracking              | Schema support                |
| --------------- | ---------------- | ---------------------------- | ----------------------------- |
| **V3** (legacy) | `.csv`, `.zip`   | No — poll dataset status     | No                            |
| **V4**          | `.csv`, `.jsonl` | Yes — per-upload import jobs | Yes — named fields with types |

V4 datasets are created by passing a `schema` when calling `POST /datasets`. See [Defining a dataset schema](/api-reference/task-builder/defining-a-dataset-schema) for details.

## Creating a dataset

```bash
POST /api/v1/data-collection/datasets
```

**V3 dataset (no schema):**

```json
{
  "name": "Product reviews Q4 2024",
  "workspace_id": "6278acb09062db3b35bcbeb0"
}
```

**V4 dataset (with schema):**

```json
{
  "name": "Product reviews Q4 2024",
  "workspace_id": "6278acb09062db3b35bcbeb0",
  "schema": {
    "strict": false,
    "fields": {
      "review_text": { "type": "text", "label": "Review" },
      "product_name": { "type": "text", "label": "Product" },
      "source": { "type": "metadata" }
    }
  }
}
```

| Field          | Type   | Required | Description                                                                                                                                |
| -------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `name`         | string | Yes      | A name for your dataset                                                                                                                    |
| `workspace_id` | string | Yes      | The ID of the Prolific workspace                                                                                                           |
| `schema`       | object | No       | Field schema. When provided, creates a V4 dataset. See [Defining a dataset schema](/api-reference/task-builder/defining-a-dataset-schema). |

## Uploading data

### V4 datasets (JSONL or CSV)

V4 upload uses a three-step flow with import job tracking.

**Step 1: Request a presigned URL**

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/upload-url/{filename}
```

For example:

```bash
GET /api/v1/data-collection/datasets/0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d/upload-url/reviews.jsonl
```

The response includes a presigned S3 URL, the `content_type` to use for the upload, and an `import_id` to track this upload:

```json
{
  "upload_url": "https://s3.amazonaws.com/bucket/...",
  "http_method": "PUT",
  "content_type": "application/x-ndjson",
  "import_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f"
}
```

**Step 2: Upload to S3**

Use the `content_type` value from the response as the `Content-Type` header.

```bash
curl -X PUT \
  -H "Content-Type: {content_type}" \
  --data-binary @reviews.jsonl \
  "{upload_url}"
```

**Step 3: Poll for import status**

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/imports/{import_id}
```

See [Monitoring an import](#monitoring-an-import) for status details.

### V3 datasets (CSV or ZIP)

**Step 1: Request a presigned URL**

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/upload-url/{filename}
```

**Step 2: Upload to S3**

```bash
curl -X PUT \
  -H "Content-Type: text/csv" \
  --data-binary @reviews.csv \
  "{upload_url}"
```

**Step 3: Poll dataset status**

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/status
```

Wait until status reaches `READY` before creating a batch.

## JSONL format

For V4 datasets, upload data as newline-delimited JSON. Each line is one datapoint. Field names must match the keys defined in the dataset schema.

```
{"review_text": "Great product!", "product_name": "Widget Pro", "source": "amazon"}
{"review_text": "Arrived damaged", "product_name": "Widget Pro", "source": "trustpilot"}
{"review_text": "Works as expected", "product_name": "Basic Widget", "source": "amazon"}
```

In strict mode (`"strict": true`), every record must include all schema fields. In non-strict mode, missing fields are allowed and treated as absent.

## CSV format

Both V3 and V4 datasets accept CSV uploads. Each row is one datapoint.

```csv
id,review_text,product_name,rating
1,"Great product, exactly what I needed!",Widget Pro,5
2,"Arrived damaged, very disappointed",Widget Pro,1
3,"Works as expected, nothing special",Basic Widget,3
```

For V4 CSV uploads, column names must match schema field keys.

## Metadata columns

Columns prefixed with `META_` are not displayed to participants. Use these for internal data you need in your results but don't want participants to see.

```csv
id,review_text,META_source,META_timestamp
1,"Great product!",amazon,2024-01-15T10:30:00Z
2,"Not worth it",trustpilot,2024-01-16T14:22:00Z
```

In V4 datasets, use fields of type `metadata` instead of the `META_` prefix.

## Custom task grouping

By default, tasks are grouped randomly when you set up a batch (using the `tasks_per_group` parameter). To define your own groupings, include a `META_TASK_GROUP_ID` column in your CSV (V3) or a field of type `task_group_id` in your schema (V4).

Rows with the same group ID value are grouped together into a single task group. Participants complete all tasks within a group in one submission.

```csv
id,review_text,product_name,META_TASK_GROUP_ID
1,"Great product!",Widget Pro,widget_pro_reviews
2,"Excellent quality",Widget Pro,widget_pro_reviews
3,"Not worth the price",Basic Widget,basic_widget_reviews
4,"Does the job",Basic Widget,basic_widget_reviews
```

> **Note**
>
> If your dataset includes `META_TASK_GROUP_ID` (V3) or a `task_group_id` field (V4), these groupings take precedence over the `tasks_per_group` parameter during batch setup.

## Monitoring an import

For V4 datasets, each upload is tracked as an import job.

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/imports/{import_id}
```

| Status           | Description                                                        |
| ---------------- | ------------------------------------------------------------------ |
| `uninitialised`  | Import job created; file not yet uploaded to S3                    |
| `queued`         | File uploaded; queued for processing                               |
| `processing`     | Extraction in progress                                             |
| `complete`       | All records accepted — `accepted_count` is populated               |
| `partial`        | Some records accepted, some rejected — see `errors`                |
| `failed`         | Extraction failed entirely — see `reason`                          |
| `pending_schema` | Dataset has no schema set; upload paused until a schema is defined |

A `partial` response includes per-record errors:

```json
{
  "import_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f",
  "status": "partial",
  "accepted_count": 498,
  "rejected_count": 2,
  "errors": [
    { "record_index": 47, "field": "review_text", "reason": "Value exceeds maximum length" },
    { "record_index": 312, "field": null, "reason": "Record missing required field: product_name" }
  ]
}
```

Accepted records are available immediately even if some records were rejected.

## Dataset status (V3 only)

Poll the status endpoint to check processing status for V3 datasets.

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/status
```

| Status          | Description                                |
| --------------- | ------------------------------------------ |
| `UNINITIALISED` | Dataset created but no data uploaded       |
| `PROCESSING`    | Dataset is being processed                 |
| `READY`         | Dataset is ready to be attached to a batch |
| `ERROR`         | Something went wrong during processing     |

Wait for the status to reach `READY` before creating a batch with this dataset.