Reliability

What pigeon guarantees, what the caller has to do, and the client to copy.

This is the page that keeps mail from getting lost. If you are about to write a fetch towards /api/send-email, you write it the way it is written here.

What pigeon guarantees

The row always exists. It is written pending before pigeon contacts the SMTP server, and it becomes sent or failed at the end. A process that dies halfway leaves a visible pending row, which is the truth, instead of no trace at all of an attempt that was made.

One retry on the connection that was never opened. If the SMTP server refuses the connection or its name does not resolve, pigeon tries again once after a second, inside the same request: that is the case of the mail server restarted exactly while your call was going through.

It does not try again in any other case. Once the message has been handed to the server, a second attempt risks sending it twice with nobody able to notice. Nor after a connection timeout: those are ten seconds already spent, and your backoff makes more sense than another immediate attempt.

Shutdown waits for the sends in flight. When the container is replaced, the process stops accepting new requests and finishes the open ones, with forty-five seconds of grace before Docker takes it away.

What it does not guarantee

There is no queue. Sending happens inside the HTTP request, not after it. If the call fails the mail does not go out, and nobody sends it again on your behalf: failed is a final state and there is no button in the panel that sends it again.

There is no protection against duplicates unless you ask for it. Two identical requests are two emails, unless you send the same Idempotency-Key: then it goes out once and the second call receives the same emailId. See Sending.

During an update there is a window. The container is replaced, and for those seconds the proxy holds the requests waiting instead of refusing them. If the wait is not enough, what you get is a 503, and it is the main reason a caller has to retry.

The timeout is a case of its own

pigeon talks to the SMTP server inside the request: it does not accept and send afterwards, it accepts by sending. A send towards a slow server gets to some forty seconds before giving up, so your timeout has to sit above that threshold. A five second timeout cuts off sends that would have worked.

If it fires anyway, you do not know what state the send is in: the mail may have gone out all the same.

With Idempotency-Key the problem disappears: you retry the same call with the same key, and if the first attempt had arrived pigeon does not send again, it returns the id of that one. It is the reason the client below always sends it.

If for some reason you cannot use it, the choice comes back to you and depends on who receives. Towards an internal recipient, say the contact form that lands in the newsroom, retry: a duplicate gets read and thrown away, a lost message cannot be recovered. Towards a customer, for a confirmation or a link, do not retry on a timeout: record what happened and look at the row in the panel, which exists either way.

Reference client

To copy into your own repository, not to import: there is no shared package, and for eighty lines it would not be worth publishing one. It works on any runtime with global fetch, AbortSignal and crypto.randomUUID, so Node 19 and above and the edge runtimes of Vercel and Cloudflare. On Node 18 the only difference is importing randomUUID from crypto.

// Sending client for pigeon. Copied from https://pigeon.esse.dev/docs/reliability
// If you change it, look first at whether the change holds for everybody and belongs there.

const ENDPOINT = "https://pigeon.esse.dev/api/send-email";

// Above pigeon's own timings, since it talks to the SMTP server inside the request: below
// forty seconds you cut off a send that would have worked.
const ATTEMPT_TIMEOUT_MS = 45_000;
const MAX_ATTEMPTS = 3;
const BASE_DELAY_MS = 1_000;

export interface SendEmailInput {
	to: string | string[];
	subject: string;
	templateId: number;
	templateData?: Record<string, string>;
	replyTo?: string;
	// One per message, not per attempt: it is what makes retrying safe. If you do not
	// pass one it gets generated, which is fine as long as you do not retry from another
	// process.
	idempotencyKey?: string;
}

export type SendEmailResult =
	| { ok: true; emailId: number; attempts: number }
	| { ok: false; code: string; message: string; attempts: number };

// What is worth sending again. Everything else is final: see the table of errors.
function isRetryable(status: number, code?: string): boolean {
	if (status === 429 || status === 503 || status === 504) return true;
	if (status === 502) return code === "SMTP_DELIVERY_FAILED";
	// Another attempt with our own key is still talking to the server.
	if (status === 409) return code === "IDEMPOTENCY_IN_PROGRESS";
	return false;
}

// Exponential with a bit of randomness: without it, all the callers that failed together
// come back together on the retry as well.
function backoffMs(attempt: number, retryAfter?: string | null): number {
	const header = retryAfter ? Number.parseInt(retryAfter, 10) : Number.NaN;
	if (Number.isFinite(header)) return Math.min(header * 1000, 30_000);
	return BASE_DELAY_MS * 2 ** (attempt - 1) + Math.floor(Math.random() * 250);
}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export async function sendEmail(
	input: SendEmailInput,
	apiKey: string,
): Promise<SendEmailResult> {
	const { idempotencyKey = crypto.randomUUID(), ...payload } = input;
	let last = { code: "UNKNOWN", message: "No attempt was made" };
	let attempt = 0;

	while (attempt < MAX_ATTEMPTS) {
		attempt++;
		try {
			const response = await fetch(ENDPOINT, {
				method: "POST",
				headers: {
					"Content-Type": "application/json",
					Authorization: `Bearer ${apiKey}`,
					// The same one on every attempt: generating it inside the loop would
					// mean not having it at all.
					"Idempotency-Key": idempotencyKey,
				},
				body: JSON.stringify(payload),
				signal: AbortSignal.timeout(ATTEMPT_TIMEOUT_MS),
			});

			if (response.ok) {
				const { emailId } = (await response.json()) as { emailId: number };
				return { ok: true, emailId, attempts: attempt };
			}

			// The body of a 503 during a deploy is HTML, not JSON: read whatever is
			// there without making the parsing fail.
			const body = (await response.json().catch(() => ({}))) as {
				error?: string;
				code?: string;
			};
			last = {
				code: body.code ?? `HTTP_${response.status}`,
				message: body.error ?? response.statusText,
			};

			if (!isRetryable(response.status, body.code)) break;
			if (attempt === MAX_ATTEMPTS) break;
			await sleep(backoffMs(attempt, response.headers.get("retry-after")));
		} catch (error) {
			// Network down, DNS, or the timeout that fired. All three are retried: with
			// the idempotency key a second attempt cannot duplicate a send that already
			// worked.
			const timedOut = error instanceof Error && error.name === "TimeoutError";
			last = {
				code: timedOut ? "TIMEOUT" : "NETWORK_ERROR",
				message: error instanceof Error ? error.message : String(error),
			};

			if (attempt === MAX_ATTEMPTS) break;
			await sleep(backoffMs(attempt));
		}
	}

	return { ...last, ok: false, attempts: attempt };
}

The result is a value and not an exception: the caller has to decide what to tell the user, and a throw invites a catch that swallows. Always record the code of a failure: it is the only thing that later lets you tell "it was our deploy" from "that address does not exist".

What not to do

Do not retry blindly. Three attempts on a 401 are the same error three times, and on an SMTP_DELIVERY_REJECTED they are three failed rows for an address that does not exist.

Do not block the response to the user on the outcome of the send, when the product allows it. A contact form that saves first and sends afterwards loses nothing if pigeon is unreachable; one that only sends loses the message, and tells the user so on top of that.

Do not send the key to the browser. /api/send-email answers without CORS headers on purpose: it is a server-only route, and the key belongs in the environment variables of the server.

What is still missing

A real queue. Today failed is a final state: if the send does not work after your attempts, nobody tries again later on your behalf and there is no button in the panel that sends it again. With the idempotency key you can retry yourself though, even hours later, without risking a duplicate.