> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.prolific.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.prolific.com/_mcp/server.

# Migrating a dataset schema

A dataset's [schema](/api-reference/ai-task-builder/dataset-schema) rarely stays fixed for the life of a project. You add a field for a new annotation, relabel a column, or tighten validation. Changing the schema of a **V4 dataset** that already holds data starts a **schema migration**: an asynchronous job that re-normalises every existing datapoint against the new schema version, so old and new data stay consistent.

This guide covers what triggers a migration, how to track it, and how to handle the outcomes.

## When a migration runs

You update a schema by patching the dataset:

```bash
PATCH /api/v1/data-collection/datasets/{dataset_id}
```

```json
{
  "schema": {
    "strict": false,
    "fields": {
      "review_text": { "type": "text", "label": "Review" },
      "product_name": { "type": "text", "label": "Product" },
      "source": { "type": "metadata" },
      "rating": { "type": "text", "label": "Rating" }
    }
  }
}
```

Every successful `PATCH` writes a **new schema version**. Whether that triggers a migration depends on whether the dataset already had a schema:

| Situation                                             | Result                                                                                                                     |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| The dataset had **no schema** yet (this is its first) | The schema is set. No datapoints exist to re-normalise, so **no migration runs** and no `migration_job_id` is returned.    |
| The dataset **already had a schema**                  | The version is bumped and an async migration re-normalises existing datapoints. The response carries a `migration_job_id`. |

Setting the first schema also releases any uploads that were parked waiting for one — imports left in `pending_schema` are re-enqueued and processed against the new schema.

### Requirements

The `PATCH` is rejected before any version is written if:

| Condition                                                                             | Response                                                  |
| ------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| The dataset is not V4                                                                 | `400` — schema updates are only supported for V4 datasets |
| Changing which column is the `task_group_id` on a dataset that already has datapoints | `400` — the grouping column is frozen once data exists    |
| The caller lacks access to the dataset's workspace                                    | `403`                                                     |
| The dataset does not exist                                                            | `404`                                                     |

The `task_group_id` designation is frozen because existing datapoints are keyed at ingestion and a migration never re-keys them — changing the grouping column would silently split grouping between old and new data. Other field changes (adding, removing, relabelling, retyping) are fine.

## Tracking the migration

When a migration starts, capture the `migration_job_id` from the `PATCH` response:

```json
{
  "id": "0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d",
  "schema_version": 4,
  "schema": { "strict": false, "fields": { "...": "..." } },
  "migration_job_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f"
}
```

Then poll the migration job until it reaches a terminal status:

```bash
GET /api/v1/data-collection/datasets/{dataset_id}/schema-migrations/{job_id}
```

### Migration status

| Status       | Description                                                                                                                                                          |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `processing` | The migration is underway — keep polling                                                                                                                             |
| `complete`   | Every datapoint was reprocessed. `reprocessed_count` is populated                                                                                                    |
| `partial`    | Some datapoints were reprocessed, but one or more fields could not be re-normalised and were nulled. `reprocessed_count`, `failed_count`, and `errors` are populated |
| `failed`     | The migration could not run at all — nothing was reprocessed. `reason` is populated                                                                                  |

A `complete` job:

```json
{
  "dataset_id": "0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d",
  "job_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f",
  "schema_version": 4,
  "status": "complete",
  "created_at": "2026-07-16T10:30:00.000Z",
  "updated_at": "2026-07-16T10:30:07.000Z",
  "reprocessed_count": 1000
}
```

## Handling a partial migration

A migration is **partial** when the datapoints are reprocessed but some field values are invalid under the new schema — for example, a column newly typed as a number that holds non-numeric text. The migration does not fail the whole job or block the change: it **nulls** the offending field on that datapoint, records it, and carries on.

```json
{
  "dataset_id": "0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d",
  "job_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f",
  "schema_version": 4,
  "status": "partial",
  "reprocessed_count": 998,
  "failed_count": 2,
  "errors": [
    {
      "datapoint_id": "0192a3c1-1111-7a0b-1c2d-3e4f5a6b7c8d",
      "field": "rating",
      "reason": "Value is not a valid number"
    }
  ]
}
```

| Error field    | Meaning                                              |
| -------------- | ---------------------------------------------------- |
| `datapoint_id` | The datapoint whose field was nulled                 |
| `field`        | The schema field key that could not be re-normalised |
| `reason`       | Human-readable explanation                           |

`reprocessed_count` counts cleanly-migrated datapoints; `failed_count` counts datapoints that had at least one nulled field. To recover a nulled field, fix the source value and re-upload or [append](/api-reference/ai-task-builder/appending-data) the corrected datapoint.

A `failed` job means the migration could not proceed — for instance the target schema version could not be resolved — and carries a `reason`. Nothing was reprocessed; the dataset's datapoints remain on the previous version.

```json
{
  "status": "failed",
  "reason": "Schema version 4 not found for dataset 0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d"
}
```

## Concurrency with imports, appends, and syncs

A migration re-normalises the whole dataset in place, so it does not run alongside other writes to the same dataset:

* While a migration is `processing`, an [append](/api-reference/ai-task-builder/appending-data) (`POST .../datapoints`) is rejected with `409` — retry once the migration completes.
* A file import in progress and a schema migration are likewise mutually exclusive on a dataset.
* A [batch sync](/api-reference/ai-task-builder/syncing-a-batch) that overlaps a dataset write can end `failed` with a reason telling you to retry.

The migration itself is idempotent: it only reprocesses datapoints below the target version and retains each datapoint's id, so a retried or redelivered run resumes from where it left off rather than double-writing.

## Recipe: change a schema and track the migration

Patch the dataset with the new schema: `PATCH /datasets/{dataset_id}` with `{ "schema": { ... } }`. Read `migration_job_id` from the response — if it's absent, this was the dataset's first schema and there's nothing to migrate.

Poll `GET /datasets/{dataset_id}/schema-migrations/{job_id}` while `status` is `processing`.

On `complete`, confirm `reprocessed_count`. On `partial`, inspect `errors` to see which fields were nulled and on which datapoints.

For any partial failures, fix the source values and re-upload or append the corrected datapoints once the migration has finished.

***

By using AI Task Builder, you agree to our [AI Task Builder Terms](https://prolific.notion.site/Researcher-Terms-7787f102f0c541bdbe2c04b5d3285acb).