Amazon SQS API
YDB supports working with topics via the Amazon SQS protocol.
Note
Working with topics via the Amazon SQS protocol is supported only for Amazon JSON protocol.
Work with a single topic can be done simultaneously using multiple protocols. For example, writing can be done using the Topic API, and reading using the Amazon SQS API, and vice versa.
When creating a topic using the CreateQueue Amazon SQS API command, the topic is created with auto-partitioning enabled: with one partition and the automatic ability to increase up to 10 active partitions. Auto-partitioning parameters can be changed via YQL or YDB CLI.
Reading via Amazon SQS API
For reading via Amazon SQS API, a shared (common) reader is used, which must be created on the topic before reading via the Amazon SQS protocol. If the topic is created using the CreateQueue Amazon SQS API command, a shared (common) reader named ydb-sqs-consumer is automatically created.
Messages written via the Topic API may be compressed using gzip, lzop, or zstd algorithms. When reading via the Amazon SQS protocol, the server does not decompress them but transmits them as base64: the reader must perform base64 decoding and then decompression.
Messages written via the Topic API without compression may contain binary data. When reading via the Amazon SQS protocol, the server also encodes them in base64 — the reader must perform base64 decoding.
To allow the reader to determine the compression algorithm, an attribute BodyEncoding is provided with each message. If the BodyEncoding attribute is missing, decompressing the message is not required. Possible attribute values: gzip, lzop, zstd, or base64.
Example of a message whose content is compressed:
{
"Messages": [
{
"MessageId": "D523647F-5E89-560B-B1D0-152201831603",
"ReceiptHandle": "CAAQAA==",
"MD5OfBody": "d620a162c499920254054f78eac4feed",
"Body": "KLUv/QBYaQAAeWRiIHdyaXRlZCAyCg==",
"Attributes": {
"SentTimestamp": "1780660726888",
"BodyEncoding": "zstd"
}
}
]
}
Writing via Amazon SQS API
When writing to a topic via the Amazon SQS protocol, messages are evenly distributed across partitions. This guarantees that all messages with the same MessageGroupId end up in the same partition.
For writing via the Amazon SQS protocol, message deduplication is supported by DeduplicationMessageId, and if DeduplicationMessageId is not provided, by message content. Deduplication by content can be enabled using the CreateQueue and SetQueueAttributes commands by specifying the ContentBasedDeduplication parameter.
Deduplication by content is implemented with a 5-minute window: a message with a duplicate DeduplicationMessageId can be written again after 5 minutes or more.
There is a limit on the number of messages that can be written to a topic partition with deduplication by content enabled: 500 messages per second. If you need to write more messages to the topic, you should increase the number of partitions. The limit for a topic is calculated as 500 messages/sec/partition × number of partitions. For example, to write 10,000 messages per second, create a topic with 20 partitions.