Metrics reference

This document provides a reference for the main metrics of the YDB system, which are used for cluster state monitoring, performance diagnostics, and load analysis.

How to view current metric values in the built-in web interface and configure their collection in Prometheus and Grafana is described in the section Configuring monitoring of the YDB cluster.

Resource usage metrics

Metric name Type, units	Description Labels
resources.storage.used_bytes IGAUGE, bytes	Size of user and service data stored in the distributed network storage. resources.storage.used_bytes = resources.storage.table.used_bytes + resources.storage.topic.used_bytes.
resources.storage.table.used_bytes IGAUGE, bytes	Size of user and service data stored by tables in the distributed network storage. Service data includes data of primary, secondary indexes, vector indexes, full-text indexes, and local Bloom indexes.
resources.storage.topic.used_bytes IGAUGE, bytes	Size of the distributed network storage used by topics. Equal to the sum of topic.storage_bytes values of all topics.
resources.storage.limit_bytes IGAUGE, bytes	Limit on the size of user and service data that the database can store in the distributed network storage.

Common gRPC API metrics

Metric name Type, units	Description Labels
api.grpc.request.bytes RATE, bytes	Size of requests received by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table or data_streams. - method – name of the gRPC API service method, for example ExecuteDataQuery (for the table service), or PutRecord, GetRecords (for the data_streams service).
api.grpc.request.dropped_count RATE, units	Number of requests whose processing was terminated at the transport (gRPC) level due to an error. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.request.inflight_count IGAUGE, units	Number of requests being processed simultaneously by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.request.inflight_bytes IGAUGE, bytes	Size of requests being processed simultaneously by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.response.bytes RATE, bytes	Size of responses sent by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.response.count RATE, units	Number of responses sent by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery. - status – request execution status, more details on statuses are described in the section Error handling.
api.grpc.response.dropped_count RATE, units	Number of responses whose sending was terminated at the transport (gRPC) level due to an error. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.response.issues RATE, units	Number of errors of a certain type that occurred during query execution in a certain time period. Labels: - issue_type – error type, the only value is optimistic_locks_invalidation, more details on lock invalidation are described in the section Transactions and queries to YDB.

gRPC API metrics for topics

Metric name Type, units	Description Labels
grpc.topic.stream_read.commits RATE, units	Number of commits of the Ydb::TopicService::StreamRead method. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.bytes RATE, units	Number of bytes read by the Ydb::TopicService::StreamRead method. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.messages RATE, units	Number of messages read by the Ydb::TopicService::StreamRead method. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.errors RATE, units	Number of errors when working with a partition. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.started RATE, pieces	Number of sessions started per unit time. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.stopped RATE, pieces	Number of sessions stopped per unit time. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.starting_count RATE, pieces	Number of sessions being started (i.e., the client received a command to start a session, but the client has not started the session yet). Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.stopping_count RATE, pieces	Number of sessions being stopped. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.count RATE, pieces	Number of partition_session. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_write.bytes RATE, bytes	Number of bytes written by the Ydb::TopicService::StreamWrite method. Labels: - topic – topic name.
grpc.topic.stream_write.uncommitted_bytes RATE, bytes	Number of bytes written by the Ydb::TopicService::StreamWrite method within uncommitted transactions. Labels: - topic – topic name.
grpc.topic.stream_write.errors RATE, pieces	Number of errors when calling the Ydb::TopicService::StreamWrite method. Labels: - topic – topic name.
grpc.topic.stream_write.messages RATE, pieces	Number of messages written by the Ydb::TopicService::StreamWrite method. Labels: - topic – topic name.
grpc.topic.stream_write.uncommitted_messages RATE, pieces	Number of messages written by the Ydb::TopicService::StreamWrite method within uncommitted transactions. Labels: - topic – topic name.
grpc.topic.stream_write.partition_throttled_milliseconds HIST_RATE, pieces	Histogram counter. Intervals are specified in milliseconds. Shows the number of messages waiting on quota. Labels: - topic – topic name.
grpc.topic.stream_write.sessions_active_count GAUGE, pieces	Number of open write sessions. Labels: - topic – topic name.
grpc.topic.stream_write.sessions_created RATE, pieces	Number of created write sessions. Labels: - topic – topic name.

HTTP API metrics

Metric name Type, units	Description Labels
api.http.data_streams.request.count RATE, pieces	Number of requests over HTTP. Labels: - method – HTTP API service method name, e.g., PutRecord, GetRecords. - topic – topic name.
api.http.data_streams.request.bytes RATE, bytes	Total size of requests over HTTP. Labels: - method – HTTP API service method name, in this case only PutRecord. - topic – topic name.
api.http.data_streams.response.count RATE, pieces	Number of responses over HTTP. Labels: - method – HTTP API service method name, e.g., PutRecord, GetRecords. - topic – topic name. - code – HTTP response code.
api.http.data_streams.response.bytes RATE, bytes	Total size of responses over HTTP. Labels: - method – HTTP API service method name, in this case only GetRecords. - topic – topic name.
api.http.data_streams.response.duration_milliseconds HIST_RATE, pieces	Histogram counter. Intervals are specified in milliseconds. Shows the number of responses whose execution time falls within a certain interval. Labels: - method – HTTP API service method name. - topic – topic name.
api.http.data_streams.get_records.messages RATE, pieces	Number of messages read by the GetRecords method. Labels: - topic – topic name.
api.http.data_streams.put_record.messages RATE, pieces	Number of messages written by the PutRecord method (always =1). Labels: - topic – topic name.
api.http.data_streams.put_records.failed_messages RATE, pieces	Number of messages sent by the PutRecords method that were not written. Labels: - topic – topic name.
api.http.data_streams.put_records.successful_messages RATE, pieces	Number of messages sent by the PutRecords method that were successfully written. Labels: - topic – topic name.
api.http.data_streams.put_records.total_messages RATE, pieces	Number of messages sent by the PutRecords method. Labels: - topic – topic name.

Kafka API metrics

Metric name Type, units	Description Labels
api.kafka.request.count RATE, pieces	Number of requests over the Kafka protocol per unit time. Labels: - method – Kafka API service method name, e.g., PRODUCE, SASL_HANDSHAKE.
api.kafka.request.bytes RATE, bytes	Total size of requests over the Kafka protocol per unit time. Labels: - method – Kafka API service method name, e.g., PRODUCE, SASL_HANDSHAKE.
api.kafka.response.count RATE, pieces	Number of responses per unit time via the Kafka protocol. Labels: - method – name of the Kafka API service method, e.g. PRODUCE, SASL_HANDSHAKE. - error_code – Kafka response code.
api.kafka.response.bytes RATE, bytes	Total size of responses per unit time via the Kafka protocol. Labels: - method – name of the Kafka API service method, e.g. PRODUCE, SASL_HANDSHAKE.
api.kafka.response.duration_milliseconds HIST_RATE, items	Histogram counter. Defines a set of intervals in milliseconds and shows the number of requests whose execution time falls within each interval. Labels: - method – name of the Kafka API service method.
api.kafka.produce.failed_messages RATE, items	Number of messages per unit time sent by the PRODUCE method that were not written. Labels: - topic – topic name.
api.kafka.produce.successful_messages RATE, items	Number of messages per unit time sent by the PRODUCE method that were successfully written. Labels: - topic – topic name.
api.kafka.produce.total_messages RATE, items	Number of messages per unit time sent by the PRODUCE method. Labels: - topic – topic name.

Session metrics

Metric name Type, units	Description Labels
table.session.active_count IGAUGE, items	Number of sessions currently open by clients.
table.session.closed_by_idle_count RATE, items	Number of sessions closed by the database server in a given time period due to exceeding the time allocated for an inactive session.

Transaction processing metrics

Transaction execution duration can be analyzed using a histogram counter. Intervals are set in milliseconds. The chart shows the number of transactions whose duration falls within a specific time interval.

Metric name Type, units	Description Labels
table.transaction.total_duration_milliseconds HIST_RATE, items	Number of transactions with a specific execution duration on the server and client. Execution duration is the time from the explicit or implicit start of a transaction to the commit or rollback. It includes the transaction processing time on the server and the time on the client between sending different requests within the same transaction. Labels: - tx_kind – transaction type, possible values read_only, read_write, write_only, pure.
table.transaction.server_duration_milliseconds HIST_RATE, items	Number of transactions with a specific execution duration on the server. Execution duration is the time of executing queries within a transaction on the server. It does not include the waiting time on the client between sending individual requests in the same transaction. Labels: - tx_kind – transaction type, possible values read_only, read_write, write_only, pure.
table.transaction.client_duration_milliseconds HIST_RATE, items	Number of transactions with a specific execution duration on the client. Execution duration is the waiting time on the client between sending individual requests in the same transaction. It does not include the execution time of queries on the server. Labels: - tx_kind – transaction type, possible values read_only, read_write, write_only, pure.

Query processing metrics

Metric name Type, units	Description Labels
table.query.request.bytes RATE, bytes	Size of YQL query text and parameter values for queries that arrived at the database in a given time period.
table.query.request.parameters_bytes RATE, bytes	Size of parameters for queries that arrived at the database in a given time period.
table.query.response.bytes RATE, bytes	Size of responses sent by the database in a given time period.
table.query.compilation.latency_milliseconds HIST_RATE, items	Histogram counter. Intervals are set in milliseconds. Shows the number of successfully completed compilation requests whose duration falls within a specific time interval.
table.query.compilation.active_count IGAUGE, items	Number of compilations currently in progress.
table.query.compilation.count RATE, items	Number of compilations that completed successfully in a given time period.
table.query.compilation.errors RATE, items	Number of compilations that completed with an error in a given time period.
table.query.compilation.cache_hits RATE, items	Number of queries in a given time period that did not require compilation because a previously created plan existed in the prepared query cache.
table.query.compilation.cache_misses RATE, items	Number of queries in a certain period of time that required compilation to execute.
table.query.execution.latency_milliseconds HIST_RATE, items	Histogram counter. Intervals are specified in milliseconds. Shows the number of queries whose execution time falls within a certain interval.

Row table partition metrics

Metric name Type, units	Description Labels
table.datashard.row_count GAUGE, items	Number of rows in all row tables of the database.
table.datashard.size_bytes GAUGE, bytes	Data size in all row tables of the database.
table.datashard.used_core_percents HIST_GAUGE, %	Histogram counter. Intervals are specified in percent. Shows the number of row table partitions that use compute resources in a proportion falling within a certain interval.
table.datashard.read.rows RATE, items	Number of rows read by all partitions of all row tables in the database in a certain period of time.
table.datashard.read.bytes RATE, bytes	Size of data read by all partitions of all row tables in the database in a certain period of time.
table.datashard.write.rows RATE, items	Number of rows written by all partitions of all row tables in the database in a certain period of time.
table.datashard.write.bytes RATE, bytes	Size of data written by all partitions of all row tables in the database in a certain period of time.
table.datashard.scan.rows RATE, items	Number of rows read via gRPC API calls StreamExecuteScanQuery or StreamReadTable by all partitions of all row tables in the database in a certain period of time.
table.datashard.scan.bytes RATE, bytes	Size of data read via gRPC API call StreamExecuteScanQuery or StreamReadTable by all partitions of all row tables in the database in a certain period of time.
table.datashard.bulk_upsert.rows RATE, items	Number of rows added via gRPC API call BulkUpsert to all partitions of all row tables in the database in a certain period of time.
table.datashard.bulk_upsert.bytes RATE, bytes	Size of data added via gRPC API call BulkUpsert to all partitions of all row tables in the database in a certain period of time.
table.datashard.erase.rows RATE, items	Number of rows deleted from all row tables in the database in a certain period of time.
table.datashard.erase.bytes RATE, bytes	Size of data deleted from all row tables in the database in a certain period of time.
table.datashard.cache_hit.bytes RATE, bytes	Total volume of row table data successfully retrieved from memory (cache). A larger volume of data retrieved from cache indicates efficient cache usage without accessing the distributed storage.
table.datashard.cache_miss.bytes RATE, bytes	Total volume of row table data that was requested but not found in memory (cache) and was read from the distributed storage. Indicates potential areas for cache optimization.

Column table partition metrics

Metric name Type, units	Description Labels
table.columnshard.write.rows RATE, items	Number of rows written by all partitions of all column tables in the database in a certain period of time.
table.columnshard.write.bytes RATE, bytes	Size of data written by all partitions of all column tables in the database in a certain period of time.
table.columnshard.scan.rows RATE, items	Number of rows read via gRPC API calls StreamExecuteScanQuery or StreamReadTable by all partitions of all column tables in the database in a certain period of time.
table.columnshard.scan.bytes RATE, bytes	Size of data read via gRPC API call StreamExecuteScanQuery or StreamReadTable by all partitions of all column tables in the database in a certain period of time.
table.columnshard.bulk_upsert.rows RATE, items	Number of rows added via gRPC API call BulkUpsert to all partitions of all column tables in the database in a certain period of time.
table.columnshard.bulk_upsert.bytes RATE, bytes	Size of data added via gRPC API call BulkUpsert to all partitions of all column tables in the database in a certain period of time.

Resource usage metrics (Dedicated mode only)

Metric name Type units	Description Labels
resources.cpu.used_core_percents RATE, %	CPU usage. A value of 100 means that one of the cores is used at 100%. The value can be greater than 100 for configurations with more than 1 core. Labels: - pool – compute pool, possible values user, system, batch, io, ic.
resources.cpu.limit_core_percents IGAUGE, %	CPU available to the database in percent. For example, for a database of three nodes with 4 cores in pool=user in each node, the value of this sensor will be 1200. Labels: - pool – compute pool, possible values user, system, batch, io, ic.
resources.memory.used_bytes IGAUGE, bytes	RAM used by the database nodes.
resources.memory.limit_bytes IGAUGE, bytes	RAM available to the database nodes.

Query processing metrics (Dedicated mode only)

Metric name Type Unit	Description Labels
table.query.compilation.cache_evictions RATE, count	Number of queries evicted from the prepared query cache in a certain time period.
table.query.compilation.cache_size_bytes IGAUGE, bytes	Size of the prepared query cache.
table.query.compilation.cached_query_count IGAUGE, count	Number of queries in the prepared query cache.

Topic metrics

Metric name Type, unit	Description Labels
topic.producers_count GAUGE, count	Number of unique topic sources. Labels: - topic – topic name.
topic.storage_bytes GAUGE, bytes	Topic size in bytes. Labels: - topic – topic name.
topic.read.bytes RATE, bytes	Number of bytes read from the topic. Labels: - topic – topic name. - consumer – consumer name.
topic.read.messages RATE, count	Number of messages read from the topic. Labels: - topic – topic name. - consumer – consumer name.
topic.read.lag_messages RATE, count	Total number of messages not yet read by this consumer across the topic. Labels: - topic – topic name. - consumer – consumer name.
topic.read.lag_milliseconds HIST_RATE, count	Histogram counter. Intervals are specified in milliseconds. Shows the number of messages for which the difference between the read time and the message creation time falls within the specified interval. Labels: - topic – topic name. - consumer – consumer name.
topic.write.bytes RATE, bytes	Size of written data. Labels: - topic – topic name.
topic.write.uncommited_bytes RATE, bytes	Size of data written as part of not yet completed transactions. Labels: - topic – topic name.
topic.write.uncompressed_bytes RATE, bytes	Size of decompressed written data. Labels: - topic – topic name.
topic.write.messages RATE, count	Number of written messages. Labels: - topic – topic name.
topic.write.uncommitted_messages RATE, count	Number of messages written as part of not yet completed transactions. Labels: - topic – topic name.
topic.write.message_size_bytes HIST_RATE, count	Histogram counter. Intervals are specified in bytes. Shows the number of messages whose size matches the interval boundaries. Labels: - topic – topic name.
topic.write.lag_milliseconds HIST_RATE, count	Histogram counter. Intervals are specified in milliseconds. Shows the number of messages for which the difference between the write time and the message creation time falls within the specified interval. Labels: - topic – topic name.

Aggregated topic partition metrics

The following table lists aggregated partition metrics for a topic. Maximum and minimum values are calculated across all partitions of the specified topic.

Metric name Type, unit	Description Labels
topic.partition.init_duration_milliseconds_max GAUGE, milliseconds	Maximum partition initialization delay. Labels: - topic – topic name.
topic.partition.producers_count_max GAUGE, count	Maximum number of sources in a partition. Labels: - topic – topic name.
topic.partition.storage_bytes_max GAUGE, bytes	Maximum partition size in bytes. Labels: - topic – topic name.
topic.partition.uptime_milliseconds_min GAUGE, count	Minimum partition uptime after restart. Normally, during a rolling restart, topic.partition.uptime_milliseconds_min is close to 0; after the rolling restart finishes, the value of topic.partition.uptime_milliseconds_min should increase to infinity. Labels: - topic – topic name.
topic.partition.total_count GAUGE, count	Total number of partitions in the topic. Labels: - topic – topic name.
topic.partition.alive_count GAUGE, count	Number of partitions sending their metrics. Labels: - topic – topic name.
topic.partition.committed_end_to_end_lag_milliseconds_max GAUGE, milliseconds	Maximum (across all partitions) difference between the current time and the creation time of the last committed message. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.committed_lag_messages_max GAUGE, items	Maximum (across all partitions) difference between the last partition offset and the committed partition offset. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.committed_read_lag_milliseconds_max GAUGE, milliseconds	Maximum (across all partitions) difference between the current time and the write time of the last committed message. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.end_to_end_lag_milliseconds_max GAUGE, milliseconds	Difference between the current time and the minimum creation time among all messages read in the last minute across all partitions. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.lag_messages_max GAUGE, items	Maximum difference (across all partitions) between the last offset in a partition and the last read offset. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.read.idle_milliseconds_max GAUGE, milliseconds	Maximum idle time (how long a partition has not been read from) across all partitions. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.read.lag_milliseconds_max GAUGE, milliseconds	Difference between the current time and the minimum write time among all messages read in the last minute across all partitions. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.write.lag_milliseconds_max GAUGE, milliseconds	Maximum difference between write time and creation time among all messages written in the last minute. Labels: - topic – topic name.
topic.partition.write.speed_limit_bytes_per_second GAUGE, bytes per second	Write quota in bytes per second per partition. Labels: - topic – topic name.
topic.partition.write.throttled_nanoseconds_max GAUGE, nanoseconds	Maximum write throttling time (waiting on quota) across all partitions. In the limit, if topic.partition.write.throttled_nanoseconds_max = 10^9, it means that the entire second was spent waiting on quota. Labels: - topic – topic name.
topic.partition.write.bytes_per_day_max GAUGE, bytes	Maximum number of bytes written in the last 24 hours across all partitions. Labels: - topic – topic name.
topic.partition.write.bytes_per_hour_max GAUGE, bytes	Maximum number of bytes written in the last hour across all partitions. Labels: - topic – topic name.
topic.partition.write.bytes_per_minute_max GAUGE, bytes	Maximum number of bytes written in the last minute across all partitions. Labels: - topic – topic name.
topic.partition.write.idle_milliseconds_max GAUGE, milliseconds	Maximum partition idle time for writes. Labels: - topic – topic name.

Resource pool metrics

Metric name Type, units	Description Labels
kqp.workload_manager.CpuQuotaManager.AverageLoadPercentage RATE, items	Average database load, DATABASE_LOAD_CPU_THRESHOLD operates based on this metric.
kqp.workload_manager.InFlightLimit GAUGE, items	Limit on the number of concurrently running queries.
kqp.workload_manager.GlobalInFly GAUGE, items	Current number of concurrently running queries. Displayed only for pools with CONCURRENT_QUERY_LIMIT or DATABASE_LOAD_CPU_THRESHOLD enabled.
kqp.workload_manager.QueueSizeLimit GAUGE, items	Size of the queue of queries waiting to be executed.
kqp.workload_manager.GlobalDelayedRequests GAUGE, items	Number of queries waiting in the execution queue. Displayed only for pools with CONCURRENT_QUERY_LIMIT or DATABASE_LOAD_CPU_THRESHOLD enabled.

See also

• Grafana dashboards for monitoring YDB metrics