Gotify Reliability Runbook: Testing the Full Path from HTTP Alert to Android

Gotify is easy to deploy and easy to misunderstand. A server can accept a test message, show it in the web interface, and still be an unreliable Android alert path a week later. The failure is usually not the container itself. It is an untested chain: an application has a token, the reverse proxy has to preserve the streaming endpoint, the server must retain its data, the phone must keep a background service alive, and somebody must notice when any link stops working.

This guide treats Gotify as a notification dependency rather than a messaging toy. The goal is not to make every alert impossible to miss. The goal is to define what delivery means, measure the parts you can measure, rehearse recovery, and keep a second path for incidents where Gotify is the thing that failed. That makes it useful for a homelab, a small internal platform, scheduled jobs, backups, and personal infrastructure without pretending it is a replacement for carrier or platform push services.

Start with a delivery contract, not a Docker command

Write down the promise before installing anything. For example: “A routine job reports success within two minutes; a critical event appears on the registered Android device within one minute when the device is online; a missed connection is visible in the monitoring system; no event is considered delivered merely because the HTTP request returned 200.” This wording separates acceptance by the server from arrival at the phone.

A useful contract has four states:

Accepted: the producer received an HTTP success response from Gotify and recorded the message identifier or request timestamp.
Stored: the Gotify instance can show the message after a restart, proving that the data path is not only in memory.
Streamed: a connected client receives the event over the WebSocket stream. This is the part most affected by proxy and idle-timeout mistakes.
Acknowledged: a human or test device confirms that the Android notification was visible and usable. This must be tested separately from the first three states.

Use different checks for different promises. An HTTP health check can prove that the process is alive; it cannot prove that the Android service is connected. A synthetic notification can prove the end-to-end route only if the device-side check is real. Keeping these claims separate prevents a green dashboard from creating false confidence.

The reliability boundary: what Gotify does and does not guarantee

Gotify provides a self-hosted server, a web interface, an HTTP API, and real-time delivery to connected clients through WebSocket. Its Android client maintains a connection to the server and presents new messages as notifications. That design avoids a third-party notification provider, but it also means the device needs a functioning background service and a reachable network path.

It is not FCM, APNs, or a carrier-grade paging system. Android power management can stop a long-running application, and a phone can be offline, behind a captive portal, asleep without the right app exception, or disconnected from the configured host. The official Android documentation explicitly calls out disabling battery optimization for Gotify; vendor-specific power managers may require additional treatment. Record this as a device prerequisite, not as a footnote.

For a critical production incident, send a compact event through a second provider or an out-of-band channel as well. Gotify can remain the private, detailed stream while the fallback carries only the service name, severity, and runbook link. The fallback is not redundant decoration: it is how you learn that the primary notification path is broken.

Model the path as five independently failing links

Producer to HTTP endpoint: DNS, TLS, authentication, request timeout, and application-side retry behavior.
Reverse proxy to Gotify: upstream reachability, request buffering, WebSocket upgrade headers, idle timeouts, and path rewriting.
Gotify process to persistent storage: mounted data directory, filesystem permissions, free space, and database consistency.
Server to device: DNS on the phone, TLS trust, firewall rules, mobile-network reachability, and a live WebSocket.
Device to human: battery policy, notification permission, channel importance, Do Not Disturb, sound/vibration settings, and whether the message is actionable.

When a notification is missing, test in this order. First determine whether the producer got an error. Then inspect the server logs and message list. Next inspect the client connection state. Only then investigate Android notification channels and power policy. This order avoids wasting time changing phone settings for a request that never reached the server.

Deploy the smallest reversible topology

Keep the Gotify container private and put the public hostname on a deliberate edge. A common layout is Gotify listening on an internal container port, a reverse proxy terminating HTTPS, and a single persistent host directory mounted at /app/data. Do not expose an unencrypted host port just because the first browser test is convenient.

services:
  gotify:
    image: gotify/server
    restart: unless-stopped
    volumes:
      - ./gotify-data:/app/data
    expose:
      - "80"
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1/health"]
      interval: 30s
      timeout: 5s
      retries: 3

The exact image tag and runtime settings should be pinned in the deployment repository rather than silently following a moving tag. Keep the compose file, environment configuration, proxy configuration, and backup procedure together. A rollback is only reversible if the operator can identify the previous image and knows which data directory belongs to it.

Do not infer that a health check makes the service reliable. It only gives the supervisor a local signal. Add a separate external check through the public hostname, and test an authenticated message path from a controlled synthetic producer. Avoid putting a secret token in a public uptime monitor URL.

Reverse proxy correctness is a functional requirement

Browser access through HTTPS is not enough. A proxy can render the web UI while breaking the long-lived stream used by clients. The proxy must pass the WebSocket upgrade, preserve the intended path, and allow a connection to remain open. A short read timeout can create a particularly misleading failure: messages work immediately after opening the app, then disappear when the connection is recycled.

For Nginx, the relevant shape is an explicit HTTP/1.1 upgrade and a long read timeout. The Gotify documentation provides the authoritative proxy examples; adapt them to your topology rather than copying directives that refer to a different path.

location / {
    proxy_pass http://gotify:80;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 1h;
}

After changing the proxy, test both ordinary API requests and the streaming connection. Check the server logs for a client that stays connected, then restart the proxy and confirm the Android client reconnects. A configuration review is not a substitute for this test.

Separate producer credentials from reader credentials

Use an Application Token for a producer that posts messages. Use a Client Token for a reader such as the Android app or a controlled dashboard. Do not reuse an administrator password in scripts, and do not put a reader token into a shell script that only needs to send.

Create one application per meaningful producer or trust boundary: backup jobs, monitoring, CI, and personal scripts do not need to share a token.
Give an application a descriptive name that identifies its owner and rotation date.
Store tokens in the producer's secret manager or a root-readable environment file, not in a repository, command history, or public monitoring configuration.
When a token may have leaked, revoke it and create a replacement. Do not rely on deleting a message or renaming the application.
Keep the administrator account for administration. It should not be the credential used by a cron job.

Assume notification bodies can contain secrets. URLs with query parameters, backup paths, customer names, and stack traces may all be sensitive. Redact tokens and credentials before posting, and set a retention policy for old messages. Reliability that creates a permanent copy of confidential data is not a free improvement.

Make producers idempotent and failure-aware

A producer should distinguish transport failure from business failure. On a timeout, the server may have accepted the message even though the producer did not receive the response. Blind retries can create duplicates. Include a compact event identifier in the title or message, such as a job run ID, and make the receiving workflow tolerant of repeated alerts.

curl --fail-with-body --silent --show-error \
  --connect-timeout 5 --max-time 15 \
  -X POST \
  -H 'Content-Type: application/json' \
  -H "X-Gotify-Key: ${GOTIFY_APP_TOKEN}" \
  -d '{"title":"backup failed","message":"job=nightly-2026-09-22 run=8f31; inspect the backup log","priority":8}' \
  'https://push.example.com/message'

Keep retry policy outside the notification server. Retry a small number of times with backoff for connection failures and 5xx responses; do not retry authentication errors or malformed requests. If a job cannot send its failure notification, emit an independent local log or fallback alert. Otherwise the failure path becomes silent at exactly the moment it matters.

Use priority as a response rule, not a decoration

Define a small severity policy that operators can remember. A low-priority message can be a daily summary; a medium-priority message may require attention during working hours; a high-priority message should include the affected service, the first safe action, and a link to the runbook. Do not mark everything urgent. If every notification is loud, none of them communicates triage.

Titles should be scannable on a locked phone. Put the system and state first, then the event ID: “Postgres backup failed · prod-db · run 8f31”. Put stack traces and verbose context behind a link or in the body, not in a title that gets truncated. Use the same vocabulary in the fallback channel so an operator can correlate both alerts.

Back up the data directory, then prove that restore works

The persistent directory is part of the service. Back it up on a schedule that matches the cost of losing message history and configuration. Include the database, uploaded assets, and any certificates or configuration stored there according to the chosen deployment model. Protect the backup itself because it may contain message content and credential-related metadata.

set -eu
stamp=$(date -u +%Y%m%dT%H%M%SZ)
tmp="/var/backups/gotify/${stamp}.tar.zst"
mkdir -p /var/backups/gotify
# Quiesce or use the documented database-safe procedure for your version.
tar --zstd -cpf "$tmp" -C /srv/gotify gotify-data
sha256sum "$tmp" > "$tmp.sha256"
find /var/backups/gotify -type f -mtime +30 -delete

That example is a file backup, not a complete recovery design. Test extraction into a disposable directory, start an isolated Gotify instance against the restored data, log in with a test account, confirm an old message is present, and send a new test message. Record the restore duration and the result. A backup that has never been restored is an assumption.

Before upgrades, take a labeled backup and record the image digest or release tag. Never test an upgrade directly against the only copy of the data. Keep the previous image available until the new instance has passed API, WebSocket, message persistence, and client reconnection checks.

Build synthetic checks around the actual failure modes

Use at least three probes. A liveness probe checks the public health endpoint or a non-destructive endpoint. A send probe posts a uniquely labeled low-noise message with a dedicated application token. A client probe verifies that a controlled reader receives it, either through a device test procedure or a small always-on client in the same operational environment.

Measure timestamps at the producer and reader. Track request latency, non-2xx responses, reconnect count, time since last client connection, and synthetic delivery age. Set a stale-client alert when a registered critical reader has not connected within its normal interval. A server with zero errors but no connected Android clients is not healthy for Android delivery.

Keep synthetic messages easy to identify and expire them under the retention policy. Never use a real incident notification as the only test; it produces ambiguous results and conditions people to ignore noise. The test should also exercise the public TLS hostname, not only the container network.

Android acceptance must be device-specific

On each critical device, perform the acceptance test after installation and after a major Android or vendor firmware change:

Grant notification permission and verify the relevant notification channel is enabled at an appropriate importance.
Disable battery optimization for Gotify and check the vendor's background-start or auto-launch controls when present.
Connect the app through both Wi-Fi and mobile data if both networks matter.
Lock the screen, wait beyond the proxy's normal idle window, and send a synthetic message.
Force a network transition, then confirm the client reconnects and reports missed messages according to its behavior.
Enable Do Not Disturb temporarily to document what the team should expect; do not mistake a policy-suppressed notification for a server failure.

The foreground connection notification may be visually minimized on supported Android versions, but it should not be treated as evidence that delivery is guaranteed. Capture the device model, Android version, battery policy, test timestamp, and result in the runbook. “Works on my phone” is not a fleet-wide guarantee.

Observe the notifier itself and keep an escape route

Monitor process restarts, container health, disk usage, database errors, certificate expiry, proxy 4xx/5xx rates, and active or recently active clients. Alert on a sudden fall in accepted messages as well as a rise in failures; a quiet producer can be a business incident, not a healthy notification system.

Route the highest-severity Gotify failures outside Gotify. Options include email, a second push provider, an on-call service, or a direct local console for a small homelab. The fallback message should say that the notification path failed, identify the affected host, and link to the diagnostic runbook. Do not duplicate every routine message into the fallback channel.

Keep logs structured enough to correlate a producer request, server acceptance, and client test. Avoid logging bearer tokens. Retain enough history to answer “did the server accept this event?” without retaining more message content than your privacy policy permits.

Run a fault-injection checklist before calling it production-ready

Stop the Gotify container and verify the producer gets a failure or timeout rather than reporting success.
Restart only the reverse proxy and verify an existing client reconnects.
Break the WebSocket upgrade temporarily and confirm the synthetic client alert fires.
Fill a disposable filesystem or simulate low disk space and verify the storage alert.
Revoke a producer token and verify the producer reports authentication failure clearly.
Disable the Android network or battery exception and document the resulting symptom and recovery step.
Restore a backup into an isolated instance and compare message persistence before and after restart.
Let a certificate approach its alert threshold in a staging hostname or synthetic rule, not in the only production endpoint.

For each exercise, record detection time, diagnosis, recovery action, and what the operator could not see. The purpose is not to prove that failures never happen; it is to reduce the time between failure and a useful decision.

A practical incident decision tree

No notification arrived: check whether the producer got a 2xx response and whether the message exists in the web UI. If it does not exist, inspect token, URL, TLS, request body, and server logs. If it exists but the client is absent, inspect the client connection and proxy stream. If the client received it but the phone did not show it, inspect notification permission, channel settings, battery policy, Do Not Disturb, and vendor background restrictions.

Messages arrive in batches: suspect a disconnected or sleeping client, proxy idle timeout, mobile-network transition, or battery manager. Compare the message IDs and timestamps with the client reconnect time. A batch after reconnection means the server may have accepted messages correctly while the device path was unavailable.

The web UI works but Android is silent: stop changing Docker settings first. Test the public WebSocket path, check the Android foreground connection state, verify the device can resolve and reach the hostname, and rerun the battery-optimization acceptance. The web UI and Android path share the server but do not exercise exactly the same client behavior.

Everything is slow: check disk space, database growth, proxy buffering and timeouts, container CPU/memory pressure, and the producer's retry storm. Do not raise every timeout before finding the bottleneck; that can turn a visible failure into a queue of delayed duplicates.

Upgrade with a small rollback window

Read the server and Android release notes before upgrading. Record the current image reference, data backup checksum, proxy configuration, and the last successful synthetic delivery. Upgrade one component at a time where practical. After the server change, test login, message creation, message persistence after restart, public API access, WebSocket streaming, and client reconnection. After an Android change, rerun the device acceptance on every device class that matters.

Keep rollback criteria explicit: restore the previous image if messages cannot be accepted, persistent data cannot be read, the public stream repeatedly disconnects, or the synthetic delivery budget is exceeded. Do not roll back by deleting the data directory. Stop the service, preserve logs, use the labeled backup only when necessary, and document whether the newer version changed data in a way that affects downgrade safety.

Decide whether Gotify is the right boundary

Gotify is a strong fit when you want control over message storage, a simple HTTP producer interface, an open-source server, and a small number of known reader devices. It is especially useful beside self-hosted monitoring and automation where keeping routine events inside your own infrastructure is valuable.

Choose another or additional system when you need guaranteed wake-up while an app is not running, a managed on-call escalation policy, large fleets of mobile clients, formal delivery receipts, regulatory audit controls, or a provider-supported availability target. The answer is often hybrid: Gotify for private operational detail, an external escalation route for high-severity events, and ordinary logs or metrics for everything that does not need a phone notification.

First-day release checklist

Define accepted, stored, streamed, and acknowledged delivery states.
Pin the server image and keep compose, proxy, secrets, and recovery instructions together.
Mount and back up the data directory; complete one isolated restore test.
Put the service behind HTTPS and test both API requests and the WebSocket stream.
Create separate application and client credentials; record rotation ownership.
Add producer timeouts, bounded retries, event IDs, redaction, and a fallback path.
Configure health, synthetic send, client freshness, certificate, disk, and restart checks.
Complete Android notification, battery, locked-screen, mobile-network, and reconnect tests.
Inject at least one proxy, storage, token, server, and device failure.
Write the incident decision tree and name the person responsible for each alert.

The reliable Gotify installation is not the one that has the most containers or the longest configuration file. It is the one whose operator can answer, with evidence, where a message stopped, whether the data can be restored, and how a critical incident reaches a human when Gotify itself is unavailable.

Authoritative references

Gotify documentation — server concepts and configuration.
Gotify Nginx reverse proxy guide — HTTPS and streaming proxy considerations.
Gotify Android README — background service and battery-optimization requirements.
Gotify server repository — releases, source, and deployment context.