Replicating Tables

Incrementally replicate table records to your destination system

Datagrid tables provide a cursor-based pagination system that enables efficient incremental replication of records. This is useful when you want to sync table data to an external system while only fetching new or updated records since your last replication.

How It Works

The tables.records.list endpoint returns a cursor with each response. By storing this cursor and passing it as the next parameter in subsequent requests, you can fetch only the records that have changed since your last replication.

Incremental Replication

The following example demonstrates how to implement incremental replication:

JavaScript

import Datagrid from "datagrid-ai";

const datagrid = new Datagrid();

// Fetch the last sync cursor from your storage
const cursor = await fetchLastSyncCursor(tableId);

const response = await datagrid.knowledge.tables.records.list(tableId, {
    next: cursor,
    limit: 500,
});

for await (const page of response.iterPages()) {
    // Push record batch to your destination system
    await pushRecords(page.data);

    // Store the cursor, indicating that you have replicated the data up to this point
    await storeCursor(tableId, page.cursor);
}

Cursor Persistence

Always store the cursor after successfully processing each batch of records. This ensures that if your replication process fails mid-way, you can resume from the last successfully processed batch.