Migrating a dataset schema

A dataset’s schema rarely stays fixed for the life of a project. You add a field for a new annotation, relabel a column, or tighten validation. Changing the schema of a V4 dataset that already holds data starts a schema migration: an asynchronous job that re-normalises every existing datapoint against the new schema version, so old and new data stay consistent.

This guide covers what triggers a migration, how to track it, and how to handle the outcomes.

When a migration runs

You update a schema by patching the dataset:

$
PATCH /api/v1/data-collection/datasets/{dataset_id}

1
{
2
  "schema": {
3
    "strict": false,
4
    "fields": {
5
      "review_text": { "type": "text", "label": "Review" },
6
      "product_name": { "type": "text", "label": "Product" },
7
      "source": { "type": "metadata" },
8
      "rating": { "type": "text", "label": "Rating" }
9
    }
10
  }
11
}

Every successful PATCH writes a new schema version. Whether that triggers a migration depends on whether the dataset already had a schema:

Requirements

The PATCH is rejected before any version is written if:

The task_group_id designation is frozen because existing datapoints are keyed at ingestion and a migration never re-keys them — changing the grouping column would silently split grouping between old and new data. Other field changes (adding, removing, relabelling, retyping) are fine.

Tracking the migration

When a migration starts, capture the migration_job_id from the PATCH response:

1
{
2
  "id": "0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d",
3
  "schema_version": 4,
4
  "schema": { "strict": false, "fields": { "...": "..." } },
5
  "migration_job_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f"
6
}

Then poll the migration job until it reaches a terminal status:

$
GET /api/v1/data-collection/datasets/{dataset_id}/schema-migrations/{job_id}

Migration status

A complete job:

1
{
2
  "dataset_id": "0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d",
3
  "job_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f",
4
  "schema_version": 4,
5
  "status": "complete",
6
  "created_at": "2026-07-16T10:30:00.000Z",
7
  "updated_at": "2026-07-16T10:30:07.000Z",
8
  "reprocessed_count": 1000
9
}

Handling a partial migration

A migration is partial when the datapoints are reprocessed but some field values are invalid under the new schema — for example, a column newly typed as a number that holds non-numeric text. The migration does not fail the whole job or block the change: it nulls the offending field on that datapoint, records it, and carries on.

1
{
2
  "dataset_id": "0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d",
3
  "job_id": "01935c2d-1a2b-3c4d-5e6f-7a8b9c0d1e2f",
4
  "schema_version": 4,
5
  "status": "partial",
6
  "reprocessed_count": 998,
7
  "failed_count": 2,
8
  "errors": [
9
    {
10
      "datapoint_id": "0192a3c1-1111-7a0b-1c2d-3e4f5a6b7c8d",
11
      "field": "rating",
12
      "reason": "Value is not a valid number"
13
    }
14
  ]
15
}

reprocessed_count counts cleanly-migrated datapoints; failed_count counts datapoints that had at least one nulled field. To recover a nulled field, fix the source value and re-upload or append the corrected datapoint.

A failed job means the migration could not proceed — for instance the target schema version could not be resolved — and carries a reason. Nothing was reprocessed; the dataset’s datapoints remain on the previous version.

1
{
2
  "status": "failed",
3
  "reason": "Schema version 4 not found for dataset 0192a3b5-e8f9-7a0b-1c2d-3e4f5a6b7c8d"
4
}

Concurrency with imports, appends, and syncs

A migration re-normalises the whole dataset in place, so it does not run alongside other writes to the same dataset:

• While a migration is processing, an append (POST .../datapoints) is rejected with 409 — retry once the migration completes.
• A file import in progress and a schema migration are likewise mutually exclusive on a dataset.
• A batch sync that overlaps a dataset write can end failed with a reason telling you to retry.

The migration itself is idempotent: it only reprocesses datapoints below the target version and retains each datapoint’s id, so a retried or redelivered run resumes from where it left off rather than double-writing.

Recipe: change a schema and track the migration

By using AI Task Builder, you agree to our AI Task Builder Terms.