---
metadata:
  - name: generator
    content: Diplodoc Platform v5.64.0
alternate:
  - https://ydb-platform--ydb.viewer.diplodoc.com/en/devops/async-replication-initial-sync.md
  - https://ydb-platform--ydb.viewer.diplodoc.com/ru/devops/async-replication-initial-sync.md
  - href: https://ydb-platform--ydb.viewer.diplodoc.com/en/devops/async-replication-initial-sync.md
    type: text/markdown
    title: Markdown version
  - href: https://ydb-platform--ydb.viewer.diplodoc.com/en/llms.txt?revision=22190017956a6e86201b27edd728b76dadd9ca61&version=v26.3
    rel: describedby
sourcePath: en/core/devops/async-replication-initial-sync.md
---
> **Documentation Index:** Fetch the complete configuration index at https://ydb-platform--ydb.viewer.diplodoc.com/en/llms.txt?revision=22190017956a6e86201b27edd728b76dadd9ca61&version=v26.3

# Initial Synchronization for Asynchronous Replication Using Backup and Restore

With this initial synchronization method, the source data is transferred to the target from a backup in S3-compatible storage. [Asynchronous replication](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md) then applies accumulated changes and continues synchronizing the databases. Applications can keep writing to the source during the transfer.

This method uses server-side [backup](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/export-s3.md) and [restore](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/import-s3.md) operations to transfer large datasets. With an [initial CDC scan](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#initial-scan), all source data passes through changefeeds, and a separate backup is not required. For differences in resource and operational costs, see the [comparison of initial synchronization methods](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#initial-sync-comparison).

## Prerequisites {#prerequisites}

In this example, the `original_table` table in the `/Root/source` database is transferred to the `replica_table` table in the `/Root/target` database. The `my_replication` replication instance is created on the target. The commands use the `source` and `target` [YDB CLI profiles](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/profile/index.md) to connect to the respective databases.

Before you begin:

- Configure the profiles and check connectivity to both databases using the [`scheme ls` command](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/commands/scheme-ls.md).
- Set up [permissions for asynchronous replication](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#init), export on the source, and import on the target. The user creating the replication instance must be able to alter the restored tables to attach them as replicas.
- On the target, [create a secret](https://ydb-platform--ydb.viewer.diplodoc.com/en/yql/reference/syntax/create-secret.md) named `source_token` containing the token that replication will use to connect to the source.
- Prepare the `mybucket` bucket and [S3 authentication](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/auth-s3.md). The examples use the `storage.yandexcloud.net` endpoint and read access keys from environment variables or the `~/.aws/credentials` file. The endpoint must be accessible from the servers of both databases: the source writes the backup, and the target reads it.
- Choose an unused prefix, `replication-bootstrap`, in the bucket. Do not modify the exported files until the import is complete.
- Make sure that `replica_table` does not exist on the target. The `import s3` command creates the table together with its data and does not overwrite existing tables. Use `scheme ls` to check the path.

Keep the set of replicated tables, their columns, and their indexes unchanged during initial synchronization. The table schemas in the backup must match the source schemas when replication starts.

{% note warning %}

Before creating replication, choose the [availability period for unacknowledged changes](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#availability-period). With `AVAILABILITY_PERIOD = 'UNLIMITED'`, changes are retained until their processing is acknowledged; provide enough space for them to accumulate in the source. With a finite period, plan the export, import, and processing of accumulated changes so that replication can acknowledge their processing before this period expires. The `PAUSED` state does not stop messages from aging.

{% endnote %}

## 1. Create Replication Without an Initial Scan {#create-replication}

On the target, run a [CREATE ASYNC REPLICATION](https://ydb-platform--ydb.viewer.diplodoc.com/en/yql/reference/syntax/create-async-replication.md) statement with `INITIAL_SCAN = FALSE`. You can run the YQL statements in this guide using the [`sql` command](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/sql.md) with the `target` profile.

```yql
CREATE ASYNC REPLICATION my_replication
FOR original_table AS replica_table
WITH (
    CONNECTION_STRING = 'grpcs://source.example.com:2135/?database=/Root/source',
    TOKEN_SECRET_PATH = 'source_token',
    INITIAL_SCAN = FALSE,
    AVAILABILITY_PERIOD = 'UNLIMITED'
);
```

The `source.example.com:2135` address in the example is the source endpoint. The `CONNECTION_STRING` and `TOKEN_SECRET_PATH` parameters specify the connection that the target will use to create and read changefeeds.

The example explicitly selects the `'UNLIMITED'` availability period, which is also the default. For a finite period, specify a positive value of type `Interval`, for example, `AVAILABILITY_PERIOD = Interval('PT48H')`. This parameter cannot be changed using `ALTER ASYNC REPLICATION` after creation.

Replication is created in the `PAUSED` state and prepares changefeeds in the source without exporting the initial data. The `replica_table` table is not created or attached to replication yet. The import operation will create it.

## 2. Wait for the Changefeeds to Be Ready {#wait-for-changefeeds}

Changefeeds are created asynchronously. Before starting the export, check that they are ready for all replicated tables, including index implementation tables if they are part of the replication.

Get the [replication description](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/commands/scheme-describe.md) in JSON format:

```bash
ydb -p target scheme describe my_replication --format proto-json-base64
```

Check the following fields:

- `paused`: replication is waiting to be started after the import. This field alone does not confirm that the changefeeds are ready.
- `items`: the list of objects with their `source_path` and `destination_path` paths. Wait until all expected objects appear.
- `source_changefeed_name` for each object: the changefeed name. Names are assigned automatically; do not set them manually.

For `original_table`, check the changefeed state on the source:

```bash
ydb -p source scheme describe original_table --format proto-json-base64
```

In the `changefeeds` array, find the changefeed with the name from `source_changefeed_name`. Its `state` field must be `STATE_ENABLED`.

For an item corresponding to a synchronous or unique index, `source_path` points to the index. Its implementation table is located at `source_path/indexImplTable`. Check the `changefeeds` array in the description of that table.

Also use [`scheme describe`](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/commands/scheme-describe.md) to check the changefeed topic settings: the presence of the `replicationConsumer` consumer, its availability policy, and the `RetentionPeriod` value. The changefeed path depends on the object:

| Object | Changefeed path |
| --- | --- |
| Table | `source_path/source_changefeed_name` |
| Index implementation table | `source_path/indexImplTable/source_changefeed_name` |

Use the field values from the replication description in these paths. To check the settings, run:

```bash
ydb -p source scheme describe <changefeed-path>
```

Where:

- `<changefeed-path>`: the changefeed path, for example, `/Root/source/original_table/stream-name`. Replace `stream-name` with the name from the replication description.

In the consumer table, check the following for `replicationConsumer`:

- With `AVAILABILITY_PERIOD = 'UNLIMITED'`, the `Important` field must be `Yes`.
- With a finite period, the `Important` field must be `No`, and `Availability period` must match the selected interval.

Repeat the checks for every object in `items`. Unacknowledged messages are protected by the consumer policy, so `RetentionPeriod` alone does not determine the availability of accumulated changes. For more information, see [Storing Changes in the Source](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#availability-period).

{% note warning %}

Start the export only after all changefeeds are ready. If the backup is created earlier, replication may miss changes made between the backup and the creation of the changefeeds. You cannot use an arbitrary pre-existing backup for this procedure.

{% endnote %}

## 3. Export the Source Data {#export}

Start an [export to S3](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/export-s3.md) using the source profile:

```bash
ydb -p source export s3 \
  --s3-endpoint storage.yandexcloud.net \
  --bucket mybucket \
  --destination-prefix replication-bootstrap \
  --include original_table \
  --include-index-data true
```

The table export is stored under the `replication-bootstrap/original_table` prefix. The `--include-index-data true` option also saves index table data if the source table has indexes. This data will be loaded from the backup during the import.

For multiple tables, specify `--include` for each table in a single export operation: the server creates a consistent copy of the tables. The exported objects must match the set of replication objects.

The command returns a server-side operation ID. Wait for the operation to complete successfully using [`operation get`](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/export-s3.md#status):

```bash
ydb -p source operation get <export-operation-id>
```

Where:

- `<export-operation-id>`: the ID returned when you started the export.

Proceed to the import only when the table output shows `ready = true`, `status = SUCCESS`, and `progress = Done`. Successfully starting the export command does not mean that the export has finished.

Applications can continue changing the source data. Changefeeds retain these changes according to the selected `AVAILABILITY_PERIOD`, including while replication is in the `PAUSED` state.

## 4. Import the Data into the Target {#import}

After the export succeeds, start an [import from S3](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/import-s3.md) using the target profile:

```bash
ydb -p target import s3 \
  --s3-endpoint storage.yandexcloud.net \
  --bucket mybucket \
  --item src=replication-bootstrap/original_table,dst=replica_table \
  --index-population-mode import
```

The `--item` option maps the table prefix in the backup to the `replica_table` target path specified in `CREATE ASYNC REPLICATION`. The `--index-population-mode import` mode loads the index data saved in the previous step. For multiple tables, specify a separate `--item` for each pair of paths.

Wait for the operation to complete successfully using [`operation get`](https://ydb-platform--ydb.viewer.diplodoc.com/en/reference/ydb-cli/export-import/import-s3.md#status):

```bash
ydb -p target operation get <import-operation-id>
```

Where:

- `<import-operation-id>`: the ID returned when you started the import.

Make sure that `ready = true`, `status = SUCCESS`, and `progress = Done`. Check the created table:

```bash
ydb -p target scheme describe replica_table
```

Until replication starts, the restored table is a regular table. Do not write application data to it: these writes are not synchronized with the source and may be overwritten when changes are applied.

## 5. Start Change Data Replication {#start-replication}

After all table imports are complete, run [ALTER ASYNC REPLICATION](https://ydb-platform--ydb.viewer.diplodoc.com/en/yql/reference/syntax/alter-async-replication.md) on the target:

```yql
ALTER ASYNC REPLICATION my_replication SET (STATE = 'STANDBY');
```

The target checks that the restored table schemas match the source schemas and attaches the tables as replica objects. The check covers the primary key, columns, their types, column families, and indexes. Attached tables become read-only, and replication starts applying accumulated changes.

Synchronous secondary indexes, including unique indexes, are supported. Restored vector indexes are deleted when tables are attached to replication. For more information, see [Restrictions](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#restrictions).

Records are read from the changefeeds created before the export. They may include changes already reflected in the backup. Until the accumulated records have been processed, the target data may change and does not yet match the current source state.

Check the state and lag:

```bash
ydb -p target scheme describe my_replication --stats
```

After replication starts, the `paused` field in the description is replaced by `running`. Statistics include lag for replication and individual objects. Wait until the accumulated changes have been processed; a missing lag value does not mean zero lag. The value may be unavailable if a consumer stops, loses its session, or does not report fresh statistics. For more information, see [Change Data Replication](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#replication-of-changes).

Replication then continues transferring new changes. If the transfer is part of an application migration, follow the separate procedure for [completing asynchronous replication](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#done) before switching writes to the target.

## Errors and Retries {#errors}

- If an export or import fails, check its status and error description using `operation get`. Keep replication in the `PAUSED` state until all table imports succeed. When retrying an import, remember that target tables must not exist; a failed operation does not confirm that no objects were created.
- If starting replication fails, get its JSON description and check the `error` field. In particular, check that all target tables exist, their schemas match, and permissions are sufficient. Do not import over tables already attached as replicas.
- If required CDC records have been deleted because their retention period expired, continuing to read cannot ensure data completeness. Repeat initial synchronization with new changefeeds and a new backup.
- Restarting the replication controller preserves the `PAUSED` state; it does not replace an explicit transition to `STANDBY`. If a CLI session is interrupted, check previously started server-side operations by their IDs before starting new ones.

For more information about temporary and critical errors, see [Error Handling During Asynchronous Replication](https://ydb-platform--ydb.viewer.diplodoc.com/en/concepts/async-replication.md#error-handling).