Backup concepts

YDB ensures data safety during hardware failures through replication and fault tolerance. However, replication does not protect against logical errors: an accidental DROP TABLE, an erroneous mass UPDATE, or DELETE will be replicated to all replicas. To protect against such scenarios, backup is required — a separate copy of data that can be restored from.

Full backup

A full backup is a snapshot of table data at a specific point in time. YDB provides several ways to create full backups, from simple to more functional.

Copying tables within a cluster

The simplest way is to create a copy of a table (or several tables) within the same cluster using the ydb tools copy command. The copy is created atomically from a consistent snapshot and uses a copy-on-write mechanism, so the operation is fast.

Suitable for:

• Quickly creating a safety copy before a dangerous operation.
• Cloning data for testing.

Dump to file system

The ydb tools dump and ydb tools restore commands allow you to dump data to a local file system and restore it back.

Suitable for:

• Local development and testing.
• Small databases.
• Creating a copy on a separate medium.

Export to S3-compatible storage

The ydb export s3 and ydb import s3 commands allow you to export and import data to an external S3-compatible storage.

Suitable for:

• Disaster recovery (data is stored outside the cluster).
• Data migration between clusters.
• Long-term archiving.

Export to NFS

The ydb export nfs and ydb import nfs commands allow you to export and import data to a network file system (NFS) mounted on all hosts of the YDB cluster. Unlike dump to file system, the export is performed on the server side and does not require data transfer via CLI.

Suitable for:

• Disaster recovery (data is stored outside the cluster).
• Data migration between clusters.
• Long-term archiving.

For more details on configuring NFS for backup and recovery, see the recipe Backup and recovery via NFS.

Incremental backup

When working with large tables, repeatedly creating full backups can be too costly. Incremental backup solves this problem: after an initial full copy, each subsequent increment captures only the changes (inserts, updates, deletes) that occurred since the previous backup.

Incremental backups are organized into a chain:

Full backup → Increment₁ → Increment₂ → ... → Incrementₙ

To restore, the entire chain is needed: first the full copy is applied, then all increments sequentially. Restoration is performed to the state at the time of the last increment in the chain.

Incremental backup is implemented using backup collections.

Backup of system tablets

The system tablet backup mechanism provides incremental copying of cluster metadata — such as Hive, BSController, and SchemeShard — to the local file system of cluster hosts.

This mechanism is used to restore cluster metadata when restoring from database backups is technically possible but not suitable in terms of time or effort. A typical scenario is when the total volume of databases in the cluster is large due to their number, the size of individual databases, or a combination of both; a full import/restore of all data into a new cluster in such a case leads to prolonged downtime. In this scenario, you can restore system tablets and bring the cluster back to a working state without performing a bulk restore of user data on the new cluster.

If the volume of databases allows for a standard restore, use export/import or dump/restore first. System tablet backup should be used as a special mechanism for situations where you need to restore cluster metadata and reduce the volume of restore operations.

How it works

Backup consists of two components:

• State snapshot — at each start, the tablet scans all its tables and writes its full state to a backup, including the data schema. The scan is based on a state snapshot and does not block tablet operation.
• Change log — with each change to data or schema, the tablet asynchronously writes the change to the log in parallel with writing to the distributed storage. When the log size exceeds the snapshot size, the tablet automatically creates a new snapshot.

Backups are created locally on the host where the tablet is currently running. Therefore, the most recent copy is on the host where the tablet was running just before the failure.

The number of stored backups on a host is limited in the configuration. After a successful snapshot, the oldest copy is automatically deleted when the limit is exceeded. Incomplete copies (without a fully written snapshot) are deleted when a new backup is created.

Comparison of approaches

See also

• Backup and restore — practical guide

• Backup collections — architecture and limitations

• Recipes for system tablet backup — enabling and restoring

• Backup and restore — practical guide

• Export/import reference — CLI commands

• YQL reference:

  • CREATE BACKUP COLLECTION
  • BACKUP
  • RESTORE
  • DROP BACKUP COLLECTION