# How I built the queue contract in Go Hypercube

> How I use one queue contract across SQL, Redis, Valkey, Amazon SQS, RabbitMQ, and memory.

Author: Nidal Bakir

Published: 2026-10-16T12:04:53.055Z

Canonical URL: https://portfolio.nidal-bakir.workers.dev/blog/go-hypercube-queue-contract/

I built queue drivers for `database/sql`, Redis, Valkey, Amazon SQS, RabbitMQ and memory in Go Hypercube. I want a job to keep the same attempt limit and backoff policy across those drivers, without moving its retry rules into each backend.

That is the boundary I built into Go Hypercube. The `queue` package defines how messages move through the transport. It has no dependency on the application, job or plugin packages. The worker uses that contract to tell a driver what to do.

The main interface has five methods:

```go
type Queue interface {
    Push(ctx context.Context, msgs ...*Message) error
    Pop(ctx context.Context, queueName string, maxMessages int) ([]*Message, error)
    Ack(ctx context.Context, msgs ...*Message) error
    Retry(ctx context.Context, msgs ...*Message) error
    DeadLetter(ctx context.Context, msgs ...*Message) error
}
```

The declarations are short. Most of the contract lives in the behaviour each method has to preserve, especially when a worker stops or an operation only partly succeeds. The full rules are in [the queue package](https://github.com/go-hypercube/go-hypercube/blob/a1dc78839bb79d52231f88988e6cc229e7671539/queue/queue.go).

## The worker decides what a failure means

`Push` stores a message. `Pop` gives a worker a delivery. From there, the worker runs the job and decides whether to call `Ack`, `Retry` or `DeadLetter`.

The driver increments `Attempt` on each delivery, starting at one. The worker reads that count and applies the job's `MaxAttempts` and `Backoff`. The driver does not invent an attempt limit of its own.

There is a small detail around delay too. `Pop` resets `Delay` to zero. If the worker wants a delayed retry, it must set a new delay before calling `Retry`. An initial delay should not accidentally become the delay for every later attempt.

Before moving a failed job out of the active queue, the worker records the handler error in `Extra`, preserving the previous metadata inside it. An explicit `DeadLetter` call must preserve the message state it receives. In the SQS and RabbitMQ drivers, that means publishing the updated message before deleting or acknowledging the original. A failure between those steps can leave a duplicate, so the handler still has to tolerate repeated delivery.

Broker configuration can still introduce its own automatic redrive behaviour. The contract requires drivers to document that. It should not silently become a second application retry policy.

## A message ID is not enough to acknowledge it

A worker can finish the job and crash before acknowledging the message. The queue cannot know that the work finished, so it may deliver the message again. Handlers have to tolerate that repeat.

There is another case to account for in drivers that use visibility leases. A worker can outlive its visibility window while another worker claims the same logical message. If the first worker wakes up and acknowledges by message ID alone, it could remove work that the second worker now owns.

Each delivery therefore gets its own opaque state in `DriverData`. Settlement uses that state to identify the delivery. A stale acknowledgement must leave the current delivery alone and report `ErrNotFound` for the failed item.

The application preserves `DriverData` unchanged. Behind it, the SQL, Redis and Valkey drivers use delivery tokens, SQS uses receipt handles, and RabbitMQ tracks a delivery tag with its channel. The worker can settle a delivery through the contract without unpacking those details.

## Ownership includes the byte slices

Passing a pointer into `Push` does not mean the queue can keep sharing that pointer with the caller. Before reporting success, the driver takes a snapshot, including separate copies of `Payload` and `Extra`.

After `Push` returns, the caller can reuse its original message without changing the queued version. Every successful `Pop` also returns an independently owned message, even when it is another delivery of the same logical ID.

While processing a delivery, the caller can change its payload, metadata and retry delay. Identity, routing and delivery fields must stay unchanged. Once `Ack`, `Retry` or `DeadLetter` succeeds, the caller gives up that delivery and must stop accessing it.

I wanted these rules to hold for the memory driver too. Otherwise, code could appear to work locally because it shared a slice with the queue, then behave differently after switching to a remote backend.

## A batch can partly succeed

The variadic methods accept multiple messages, but a batch is not promised to be one transaction. Some items can succeed while others fail.

`BatchError` reports the failed items, with each item's original input index, message pointer and cause. That gives the caller enough information to handle the failures without blindly submitting the successful items again. A nil message is an individual argument error; it should not stop other valid items from being processed.

Backend request limits belong inside the driver too. My SQS driver splits send batches at ten entries and accounts for the combined message body size. The caller can pass a larger batch through the contract and still get failures reported against the original input positions.

Cancellation also needs care. A context error from a remote operation does not prove that the backend made no change. A command may have reached the server before the caller stopped waiting. Treating cancellation as a guaranteed rollback would make retries unsafe.

## Keep backend differences visible

The [SQL driver](https://github.com/go-hypercube/hypercube-queue-sql/blob/4e44acbc94e6ef19c15fab1705d67732f79ed8ed/sql.go) works through `database/sql`, with PostgreSQL, MySQL and SQLite dialects. It uses database transactions and dialect specific claim statements to give workers ownership of rows. The [Redis driver](https://github.com/go-hypercube/hypercube-queue-redis/blob/0cf2ffec9d21a46b84c05283e2f8d0c75679a9c0/redis.go) and [Valkey driver](https://github.com/go-hypercube/hypercube-queue-valkey/blob/16952e7eb7a65a1e7cb6cd3847d2b675a3b7713f/valkey.go) use atomic Lua transitions. Valkey has its own implementation using the native Valkey Go client.

The [Amazon SQS driver](https://github.com/go-hypercube/hypercube-queue-sqs/blob/26b525f920a136b66e0e89e5bef0734a72c58204/sqs.go) works with existing Standard queues and uses long polling. Logical ID reservations go through a separate `IDStore`. Its default store is local to the process, so an application using multiple processes needs a shared implementation to enforce ID uniqueness across them.

The [RabbitMQ driver](https://github.com/go-hypercube/hypercube-queue-rabbitmq/blob/fedc9a12f02ab3dbfe01f0d979b8f6c45b3be7b9/delivery.go) keeps an AMQP channel open for outstanding deliveries. RabbitMQ does not provide a visibility deadline for an unacknowledged delivery. The driver preserves `VisibilityTimeout` in the message, but does not use it to schedule redelivery. Unsettled work can return when the channel or connection closes.

Those differences are why visibility extension, moving failed messages back into the queue, and statistics are separate optional interfaces. SQL, Redis, Valkey and memory implement all three. SQS implements visibility extension and redrive, without statistics. RabbitMQ implements redrive, without visibility extension or statistics. I can check for a capability instead of asking a driver to pretend it supports one.

Durability also stays a backend concern. The [memory driver](https://github.com/go-hypercube/hypercube-queue-memory/blob/851cf943d0341ba166247adc273d731f101a58d6/memory.go) loses its messages when the process exits. SQL, Redis and Valkey depend on their database and server configuration, while SQS and RabbitMQ have their own broker requirements and limits. I keep those documented with each driver so choosing a backend includes understanding how it behaves when something fails.