# Incremental loading (CDC) Source: https://docs.file.ai/deliver/incremental-loading > Sync new and changed files to a data warehouse with timestamp filters on GET /v1/files. fileScout # Incremental loading (CDC) Copy pageCopy page Sync new and changed files to a data warehouse with timestamp filters on GET /v1/files. Copy pageCopy page `GET /v1/files` accepts `updatedAfter` and `createdAfter` filters, so each sync fetches only the files that changed since the last one. Use it to keep Snowflake, BigQuery or any other warehouse in step with your fileAI workspace without re-reading everything. ## [​ ](#when-to-use-it) When to use it * **Scheduled syncs** to a warehouse or data lake: use this page. * **Reacting to each upload as it finishes**: use [processing callbacks](/deliver/callbacks) instead, and keep a periodic incremental sync as a safety net. This is insert-and-update capture only. Deleted files never appear in `GET /v1/files`, so you cannot see deletions this way. See [Deletions](#deletions). ## [​ ](#how-it-works) How it works 1 Store a watermark After each successful sync, save the largest `updatedAt` you loaded. On the first run there is no watermark: fetch everything. 2 Fetch everything changed since the watermark Call `GET /v1/files?updatedAfter=`, sorted by `updatedAt` ascending, and page through every result. 3 Deduplicate and merge The filter is inclusive, so the boundary file comes back again. Merge on `fileId`, keeping the row with the newest `updatedAt`. 4 Advance the watermark Only after every page has loaded, save the new maximum `updatedAt`. ### [​ ](#filters) Filters Parameter Matches files where Format `updatedAfter` `updatedAt` is **on or after** the value (`>=`) ISO 8601, e.g. `2025-01-01T00:00:00.000Z` `createdAfter` `createdAt` is **on or after** the value (`>=`) ISO 8601 Both are **inclusive**: a file with `updatedAt = 2025-01-01T12:00:00.000Z` is returned for `updatedAfter=2025-01-01T12:00:00.000Z`. A value that is not a valid timestamp is rejected with `422`. Send timestamps in UTC; `createdAt` and `updatedAt` come back in UTC with millisecond precision. You can combine the filters with `fileIds` and `directoryId`. ### [​ ](#choosing-a-pagination-mode) Choosing a pagination mode `GET /v1/files` supports both [pagination modes](/api/pagination). **Use cursor mode** with `sortBy=updatedAt&sortOrder=ASC`: pages are keyed on the sort value and the file’s id, so a file written mid-walk does not shift rows you have not read yet. A file edited during your walk can still turn up again on the next sync; the inclusive filter plus deduplication covers this. If you must use legacy `page`/`limit`, always send both `sortBy` and `sortOrder`, or the order is unspecified. [Pagination](/api/pagination) explains why. ### [​ ](#what-changes-updatedat) What changes `updatedAt` Any change to a file’s record moves its `updatedAt`, including: * metadata changes, status changes and reprocessing (including OCR) * extracted values and schema updates * classification, tag, contact or vendor changes * workflow step changes ### [​ ](#deletions) Deletions Deleted files are excluded from every `GET /v1/files` response, including filtered ones. To detect them, run a periodic full sync and remove rows from your warehouse that no longer come back. ## [​ ](#example) Example The first page of everything changed since the watermark: ``` curl "$FILEAI_BASE_URL/files?updatedAfter=2025-01-01T00:00:00.000Z&sortBy=updatedAt&sortOrder=ASC&limit=100&cursor=" \ -H "x-api-key: $FILEAI_API_KEY" ``` Follow `pagination.nextCursor` until `pagination.hasMore` is `false`, then merge and advance the watermark. The [warehouse sync recipe](/examples/warehouse-sync) has the full loop in Python and Node, and a Snowflake `MERGE`. ## [​ ](#reference) Reference ### [​ ](#file-fields) File fields Each file in `data` (cursor mode) or `files` (legacy mode) carries: Field Description `fileId` The document’s id. Use this as your primary key. `uploadId` The upload the document came from. `fileName`, `fileType`, `fileSize`, `fileStoragePath`, `fileHash` File details. `status` Processing status, e.g. `processed`, `action_required`. `fileClass` The document’s classification. `schemaId`, `blueprintId` The AI Schema and file type used. `isFormFilled` Whether values have been extracted. `fileContactId`, `fileContactName` The matched contact or vendor. `currency`, `summary` Detected currency and generated summary. `isDuplicate`, `duplicateToFileId`, `referenceId` Duplicate detection. `isEmbedded`, `isCentralAIEmbedded`, `isDocumentExampleAdded` Indexing flags. `directoryIds` Folders the file is in. `exportInfo` Latest integration export: `integrationExportStatus`, `integrationExportMessages`, `integrationExportUpdatedAt`. `createdAt`, `updatedAt` UTC timestamps, millisecond precision. Legacy mode also returns `count` (total matches) and `currentPage`. Cursor mode returns `pagination` instead. ### [​ ](#quick-reference) Quick reference Behaviour Value Filter comparison `>=` (inclusive) Invalid timestamp `422` Deleted files Never returned Recommended sort `sortBy=updatedAt&sortOrder=ASC` Deduplicate on `fileId`, keeping the newest `updatedAt` ## [​ ](#related) Related ## Pagination Cursor and legacy modes, limits and cursor rules. ## Warehouse sync recipe A complete sync job, end to end. ## Processing callbacks Get told when an upload finishes instead of polling. ## Errors What a `422` or `400` from this endpoint looks like. [Previous](/deliver/callbacks)[ Schema locking Stop fileAI from creating new file types, so every document lands in a schema you have already approved. Next ](/extract/schema-locking) ⌘I ## On this page * [When to use it](#when-to-use-it) * [How it works](#how-it-works) * [Filters](#filters) * [Choosing a pagination mode](#choosing-a-pagination-mode) * [What changes updatedAt](#what-changes-updatedat) * [Deletions](#deletions) * [Example](#example) * [Reference](#reference) * [File fields](#file-fields) * [Quick reference](#quick-reference) * [Related](#related)