Reading messages from a topic

The topic read command reads messages from a topic and outputs them to a file or the command-line terminal:

ydb [connection options] topic read <topic-path> --consumer STR \
  [--format STR] [--wait] [--limit INT] \
  [--transform STR] [--file STR] [--commit BOOL] \
  [additional parameters...]

where [connection options] are database connection options

Three command modes are supported:

  1. Single message. No more than one message is read from a topic.
  2. Batch mode. Messages are read from a topic until it runs out of messages for processing or their number exceeds the limit that must be set.
  3. Streaming mode. Messages are read from a topic as they appear while waiting for new messages to arrive until you terminate the command with Ctrl+C or the number of messages exceeds the limit that is set optionally.

Parameters

Required parameters

Name Description
<topic-path> Topic path
-c VAL, --consumer VAL Topic consumer name.
Message consumption starts from the current offset for this consumer (if the --timestamp parameter is not specified).
The current offset is shifted as messages are consumed and output (if --commit=false is not set).

Basic optional parameters

--format STR: Output format.

  • Specifies how to format messages at the output. Some formats don't support streaming mode.

  • List of supported formats:

    Name Description Is
    streaming mode supported?
    single-message
    (default)    		 The contents of no more than one message are output without formatting. -
    pretty Output to a pseudo-graphic table with columns containing message metadata. The message itself is output to the body column. No
    newline-delimited Messages are output with a delimiter (0x0A newline character) added after each message. Yes
    concatenated Messages are output one after another with no delimiter added. Yes

--wait (-w): Waiting for new messages to arrive.

  • Enables waiting for the first message to appear in a topic. If not set and the topic has no messages to handle, the command is terminated once started. If the flag is set, the started read message command waits for the first message to arrive to be processed.
  • Enables streaming selection mode for the formats that support it, or else batch mode is used.

--limit INT: The maximum number of messages that can be consumed from a topic.

  • The default and acceptable values depend on the selected output format:

    Does the format
    support streaming selection mode?    		 Default limit value Acceptable values
    No 10 1-500
    Yes 0 (no limit) 0-500

--transform VAL: Method for transforming messages.

  • Defaults to none.
  • Possible values:
    base64: A message is transformed into Base64
    none: The contents of a message are output byte by byte without transforming them.

--file VAL (-f VAL): Write the messages read to the specified file. If not set, messages are output to stdout.

--commit BOOL: Commit message reads.

  1. If true (by default), a consumer's current offset is shifted as topic messages are consumed.
  2. Possible values: true or false.

Other optional parameters

Name Description
--idle-timeout VAL Timeout for deciding if a topic is empty, meaning that it contains no messages for processing.
The time is counted from the point when a connection is established once the command is run or when the last message is received. If no new messages arrive from the server during the specified timeout, the topic is considered to be empty.
Defaults to 1s (1 second).
--timestamp VAL Message consumption starts from the point in time specified in UNIX timestamp format.
If not set, messages are consumed starting from the consumer's current offset in the topic.
If set, consumption starts from the first message received after the specified time.
--metadata-fields VAL List of message attributes whose values should be output in columns with metadata in pretty format. If not set, columns with all attributes are output.
Possible values:
--partition-ids VAL Comma-separated list of partition identifiers to read from.
If not specified, messages are read from all partitions.

Examples

Note

The examples use the quickstart profile. To learn more, see Creating a profile to connect to a test database.

In all the examples below, a topic named topic1 and a consumer named c1 are used.

  • Reading a single message with output to the terminal: If the topic doesn't contain new messages for this consumer, the command terminates with no data output:

    ydb -p quickstart topic read topic1 -c c1

  • Waiting for and reading a single message written to a file named message.bin. The command keeps running until new messages appear in the topic for this consumer. However, you can terminate it with Ctrl+C:

    ydb -p quickstart topic read topic1 -c c1 -w -f message.bin

  • Viewing information about messages waiting to be handled by the consumer without committing them. Up to 10 first messages are output:

    ydb -p quickstart topic read topic1 -c c1 --format pretty --commit false

  • Output messages to the terminal as they appear, using newline delimiter characters and transforming messages into Base64. The command will be running until you terminate it with Ctrl+C:

    ydb -p quickstart topic read topic1 -c c1 -w --format newline-delimited --transform base64

  • Track when new messages with the ERROR text appear in the topic and output them to the terminal once they arrive:

    ydb -p quickstart topic read topic1 -c c1 --format newline-delimited -w | grep ERROR

  • Receive another non-empty batch of no more than 150 messages transformed into base64, delimited with newline characters, and written to the batch.txt file:

    ydb -p quickstart topic read topic1 -c c1 \
  --format newline-delimited -w --limit 150 \
  --transform base64 -f batch.txt

  • Examples of YDB CLI command integration

Previous
Saving a consumer offset
Next
Writing messages to a topic