Delivery lifecycle and recovery

Availability

The advanced delivery pipeline requires an explicitly activated deployment. If your dashboard shows the previous delivery history, ask your administrator about rollout status. Do not assume the new recovery windows apply to a legacy worker.

Separate the guarantees

Secure V2 authenticates and decrypts a packet, resolves its immutable schema slot and decodes each reading. PostgreSQL commits the sample, a durable event identity and eligible destination-specific delivery records together. Only after the ingestion RPC succeeds can the gateway build the authenticated device receipt. A failed database or outbox transaction cannot produce a successful commit receipt.

Received, authenticated and decoded describe gateway evidence, while Stored and Queued describe database evidence. The delivery timeline records database transaction time and worker attempt times. It does not invent an arrival time or show a missing UDP packet as rejected. Authenticated gateway capture, when enabled before sending, adds actual processing stages. Failed authentication is not attributed to a customer device merely from the packet header.

Signing and duplicate requests

Each webhook retains the existing raw-body HMAC-SHA256 contract: x-struct-signature is sha256= followed by the hexadecimal digest. Verify the exact raw bytes before parsing; see the complete Node receiver in Webhooks. body.id and x-struct-event-id identify the original reading. x-struct-delivery-id identifies its destination delivery and stays stable across automatic retries and manual replay. x-struct-attempt increases across recovery cycles.

Return any HTTP 2xx only after safely accepting the event. Struct records the response, not the completion of every later process in your application. A request can succeed remotely before a timeout or worker crash prevents Struct recording it. Atomically deduplicate event IDs in your own database. A replay retains the original event ID, so deliberate reprocessing needs an explicit receiver policy. The legacy signature has no signed delivery timestamp; maintain durable deduplication and protect/rotate the destination secret using your existing controls.

Automatic retry policy

When the new delivery pipeline is enabled, HTTP 408, 429 and 500–599 and connection/DNS/timeouts retry. Most other HTTP errors, redirects and invalid public-HTTPS destination configuration are terminal. Redirects are never followed. The timeout is eight seconds. Backoff starts at five seconds, doubles per attempt, adds bounded zero-to-five-second jitter and is bounded at one hour. Retry-After seconds or dates are honored up to one hour when inside the retry deadline.

Automatic delivery stops after eight attempts or 24 hours per cycle. The worker stores attempts when claimed and records status, safe error category, response code, elapsed time and next retry. A 60-second lease recovers abandoned claims with an unknown remote outcome. Endpoint dispatch is limited to one active request and ten starts/second; a tenant has up to eight active claims. The process concurrency setting can further reduce those limits.

Use Delivery to investigate

Open Delivery, filter by device, destination, event UUID, status, error category, time range, attempt count or protocol. Results are queried server-side, 50 per page. Trace shows the reading, schema slot, sequence, batch index, queue record, attempts, response metadata and manual actions. Historical events may lack gateway/schema metadata; that absence is labeled.

Destination health shows backlog, last success and failed/unknown attempts in the last 24 hours. Pause retains work and blocks new claims; an already-started request may finish. Disable/removal is a configuration change that can dead-letter work. Fix the receiver before resuming or recovering failures.

Retry, replay and cancel

Retry a failed or skipped delivery while its payload remains available. Replay a delivered event only when you intend another request. Both keep the original reading and delivery identities. Use Retry eligible matches to preview the oldest 100 eligible failures across the current destination, device, event and time filters (the status filter is replaced by failed/skipped eligibility), or select individual rows. Select up to 100 deliveries per recovery request; the UI previews eligibility and requires duplicate-processing confirmation. Bulk operations are all-or-nothing and are limited to twenty actions/minute/workspace. Each delivery supports three manual recovery cycles.

Cancel applies to queued work, not a request already in flight. Every accepted action is audited and scheduling uses a request UUID for idempotency. The confirmation message means queued, not delivered. Owner/admin permissions are checked again in PostgreSQL; viewers can inspect permitted evidence but cannot schedule recovery.

Retention and overload

The new pipeline keeps delivery recovery payloads for seven days independently of telemetry sample retention. At expiry, remaining work becomes a recorded payload-expired dead letter and cannot be replayed. Delivery metadata, attempts and event identities last thirty days. Workspace deletion still removes its data. These are bounded operational recovery windows, not a promise of indefinite delivery or new subscription benefits.

A service-controlled pending-work ceiling (default 100,000 per workspace) prevents unlimited backlog. If atomic enqueue would exceed it, ingestion rolls back and cannot issue a commit receipt. Firmware should retain and retry the original frame according to its existing policy. Queue and retention policy must be reviewed during rollout. Historical events are not automatically sent when the migration is installed.

Troubleshoot by evidence

No stored reading: check Wi-Fi, endpoint/UDP port, current credentials, schema slot, receipt result and authenticated diagnostics. Unknown receipt can mean accepted storage with a lost reply. Stored but no webhook: inspect destination scope/event types/routing and the delivery queue. HTTP 500/503: wait for retry or repair your receiver. HTTP 401/403: fix receiver authorization/signing configuration, then explicitly retry.

Repeated timeout: make the receiver accept promptly and process expensive work asynchronously after durable acceptance. Dead letter: inspect the terminal reason and payload expiry before replay. A missing payload cannot be reconstructed by claiming the old event was sent again. Worker/database outages leave durable work for lease recovery; remote duplicates remain possible.