Skip to content

Troubleshooting

Symptoms and their fixes, by area. Each entry starts with what you see: an error text, a state, a log line. For how the stack is set up, start from the docs index.

Sync server

  • docker compose up -d fails, the Sync server stays down, with ERROR: SYNC_X and X are both set .... A setting has both its old and its new name, in .env or in the environment docker compose runs in. Run ./prepare_environment.sh --upgrade for .env, or unset the old name in your shell. See When both names are set.
  • The init container refuses to start on an empty value, or on a <REPLACE_WITH_...> placeholder. The public URL or a database password was never filled in. Set it in .env (Settings).
  • The init container logs a warning listing old setting names, and the setup check "Old setting names (.env, environment)" fails. Run ./prepare_environment.sh --upgrade: see Renamed settings.
  • 413 Request Entity Too Large in about:sync-log. The reverse proxy's limit of the request body is lower than the Sync server's: raise it, see Request size. With the Helm chart, the Ingress annotation: see Ingress.
  • Anything else in SYNC_MAX_USERS (empty, not a number) stops the init container, and with it the Sync server. The forms it takes: Limits.
  • The account limit does not change after editing .env. With initial:N the value is set only once; after that the admin UI manages it. See Limits.

Databases

  • ERROR 1419 ... You do not have the SUPER privilege and binary logging is enabled (init container or fxa-policy). Set log_bin_trust_function_creators=ON on the server: see Server requirements.
  • ERROR 2026 ... self-signed certificate in certificate chain, then "Timeout waiting for database connection" in the init container. Set SYNC_DB_TLS_VERIFY=false (Helm: database.external.tlsVerify: false).
  • "Timeout waiting for database connection", or the migrations never finish. The containers cannot reach the server; the init container's message ends with the database client's last error (a name that does not resolve, a refused connection, a login). Check the host (not localhost), the port, the server's listen address, the firewall and the users' host part. Test from the stack's network: docker compose run --rm --no-deps --entrypoint mariadb tokenserver_db_init --host=... --user=... -p --skip-ssl-verify-server-cert -e 'SELECT 1'. With the Helm chart: kubectl logs deploy/<release>-firefox-sync-syncstorage -c wait-db (the server answers) and -c db-bootstrap (the logins).
  • the Sync databases on your server lack what this release needs (or the accounts' databases), in db-bootstrap or fxa-db-bootstrap. A user, grant or database of database-setup.sql is missing, or a login is refused: the error names the statement. setup migrations NNN not applied: this release needs a file your server has not run, its SQL follows in the log. See When a release needs new users or grants.
  • setup migrations of ... were applied as another file than this release's, in db-bootstrap. A file of db/*/setup/ differs from the one that ran on your server: an edited checkout or image. Use this release's files; see Setup migrations.
  • Access denied from PostgreSQL, or no pg_hba.conf entry. Add a host line for the containers' addresses: see Server requirements.
  • A MariaDB or MySQL container does not start, with keep-major.sh: this database is MySQL 8, the image is MySQL 9 (or MariaDB). Its image setting (SYNC_MARIADB_IMAGE, SYNC_MYSQL_IMAGE, FXA_MARIADB_IMAGE or FXA_MYSQL_IMAGE) is not set, and its lts default is another major than the data. ./prepare_environment.sh --upgrade pins it (The database majors of an install from before). To move the data to the new major instead: MySQL 8 to 9.
  • Plugin 'mysql_native_password' is not loaded (MySQL 9, also for root). A user is still on mysql_native_password: for example an install from before moved to mysql:lts without step 2 of MySQL 8 to 9. Back on FXA_MYSQL_IMAGE=mysql:8 the data no longer opens, because MySQL 9 upgraded it. Either restore the backup of that procedure (Restore), or start MySQL 9 once with --skip-grant-tables, run FLUSH PRIVILEGES and the ALTER USER of step 2.
  • The PostgreSQL container does not start, with keep-major.sh: this database is PostgreSQL 18 ... Not started. The image is another major than the data: SYNC_POSTGRES_IMAGE is unset and postgres:latest moved on, or the pin changed. Set it to the major the data has (the message names it): see PostgreSQL.
  • The setup check of the event log fails, and the admin UI's log says the event log is in the tokenserver database. An install from before ADMIN_DB_PASSWORD: see The admin UI's own database.

Self-hosted accounts

  • ER_NOT_SUPPORTED_AUTH_MODE from fxa-migrations or fxa-auth. The accounts image has no mozilla/fxa#21709 (Mozilla's images, or ours from before). Then FXA_MYSQL_USER must use mysql_native_password, on MySQL 8.4 with mysql_native_password=ON (fxa-db: FXA_MYSQL_IMAGE=mysql:8 with FXA_MYSQL_ARGS). Use our current images for MySQL 9: see MySQL or MariaDB.
  • ER_CANT_DROP_FIELD_OR_KEY: Can't DROP FOREIGN KEY ..._ibfk_1 from fxa-migrations. The server is MariaDB 12 and the accounts image does not have mozilla/fxa#21494 (Mozilla's images, or ours from before). Use our current images, or MariaDB 11.4 to 11.8 on a new database: see MariaDB 12.
  • fxa-migrations stops at once with Access denied for user and not retried. The accounts database refused FXA_MYSQL_USER: a wrong FXA_MYSQL_PASSWORD, or missing grants. Unlike a server that does not answer yet, which is retried for 5 minutes, this stops at the first try. Fix the password or the grants (Your own server), then start again. With the Helm chart the auth pod stays in its init containers until then.
  • docker compose up -d stops with an error after an upgrade. fxa-migrations failed: docker compose logs fxa-migrations says why. To go back to the older release, restore the backup: see Self-hosted accounts.
  • fxa-auth stops at once with Config 'mfa.jwt.secretKey' must be set in production (or must be at least 32 bytes), and stays unhealthy. The accounts image is v1.349.0 or later, the checkout older: since v1.349.0 the auth server needs its own key for the tokens that guard the password, email and 2-step changes. Update the checkout (git pull), then docker compose up -d (Quadlets: ./prepare_environment.sh --render-units, then restart the units it names; Helm: a chart from then on). It derives the key from FXA_OAUTH_SECRET, nothing to add to .env: see Self-hosted accounts.
  • The accounts services stay (health: starting). They can take a few minutes after a start or an upgrade.
  • A new release runs on the old schema with Quadlets. podman auto-update replaced the accounts services without running the migrations: see Quadlets.
  • Firefox refuses to sign in to your accounts server. The release is too old for that Firefox (for example Firefox 154 needs a recent one, because it switched to Bearer tokens). Upgrade: see Self-hosted accounts.
  • Firefox signs in to the account, but Sync never starts; the admin UI's setup check says "Sign-in keys (FXA_OAUTH_SERVER_URL) ... connection refused". See Sign-in keys: connection refused.
  • docker compose up -d fails with bind: address already in use for a port of the host (127.0.0.1:9000, 127.0.0.1:5000...). Something else on the host has that port. Run ./prepare_environment.sh --upgrade: it moves the ports something else took to the next free ones and writes proxy/ again (Ports in use); then put the new proxy/ file into your proxy. By hand: change only that *_EXPORT_PORT in .env, then ./prepare_environment.sh --render-proxy.

Sign-in keys: connection refused

The Sync server checks every sign-in of Firefox with the keys at FXA_OAUTH_SERVER_URL. With the self-hosted accounts that is http://fxa-auth:9000/v1, container-to-container: inside the stack the accounts API always listens on fxa-auth:9000. FXA_AUTH_EXPORT_PORT is something else, the port of the host your reverse proxy reaches it on (127.0.0.1:9000 by default). Changing the host port, for example to 127.0.0.1:9010 because the host had 9000 in use already, must not change FXA_OAUTH_SERVER_URL: set to http://fxa-auth:9010/v1, nothing answers there, every sign-in to Sync fails, and the admin UI's setup check Sign-in keys says connection refused (with the explicit FXA_OAUTH_SERVER_URL is container-to-container message). prepare_environment.sh --upgrade and --render-proxy warn of it too.

Fix .env:

FXA_OAUTH_SERVER_URL=http://fxa-auth:9000/v1
FXA_AUTH_EXPORT_PORT=127.0.0.1:9010

then docker compose up -d (Quadlets: ./prepare_environment.sh --render-units, then restart the units it names). The setup itself picks a free host port for you when one is in use (Ports in use), and never changes the ports inside the stack.

Backups

  • The backup or backup-fxa container is unhealthy. Its loop stopped, its last run failed, or the last good run is more than 25 hours old. docker compose logs backup says which. See Check that it runs.
  • A run failed with "another backup run ... was still going on". A run waits for the one going on at most an hour, then fails. Check what holds the lock: a run that was killed has its lock taken over by the next run.
  • A restore loaded nothing, but did not fail. Without gunzip -t a missing run (mistyped, or pruned by BACKUP_KEEP) gives the client an empty input, which it takes as a success. Use the commands of Restore: they check the file first.
  • The restore of ffs fails on a DEFINER. Restore the accounts as the user that made the dumps (FXA_MYSQL_USER): see Restore.

Podman

  • A setting changed in .env has no effect. The installed units hold their own copy of the values: change it there, see Change a setting.
  • prepare_environment.sh stops with ERROR: <unit> needs <NAME>, which is not set in .env. A unit has a %NAME% placeholder the script has no value for: the units of config/quadlet/ and the script come from different releases. Use the files of one release.
  • A one-shot unit does not run again (the migrations, the init container, the sign-up policy). They stay "active" after their first run. Stop them along with the units that need them, as in Quadlets.
  • The Sync server's state, logs and health checks. See Check the units.
  • MySQL containers that cannot find /etc/my.cnf or their error messages, and that Podman cannot stop (given PID did not die within timeout), on an Ubuntu host with mysql-server installed. The host's AppArmor profile for /usr/sbin/mysqld confines the mysqld of the mysql image in rootless Podman's containers too, as in Kubernetes: disable that profile as said there, or use MariaDB (the default) or PostgreSQL.
  • An init container stops with Timeout waiting for database connection (postgres_db:5432): ... could not translate host name "postgres_db" (or Unknown server host 'tokenserver_db') right after docker compose up -d recreated the stack, with docker compose on Podman 4 (Ubuntu 24.04's Podman 4.9). The database container runs and is healthy, but Podman's DNS (aardvark-dns) does not find it: now and then Podman 4's network tools lose track of a container that compose recreated along with the others of its network. Run docker compose down (it keeps the volumes, unlike down -v), then docker compose up -d. We have seen it on Podman 4.9 only, not on Podman 5.
  • rootless netns: kill network process: permission denied on Ubuntu 26.04. Ubuntu's AppArmor profile of pasta refuses the signal Podman sends it when the last container of its bridge networks goes (Ubuntu bug 2154379): docker compose up on the Podman socket cannot remove the containers it recreates. Until Ubuntu fixes it, let pasta receive that signal, as CI does (pasta-apparmor.sh), and again after an update of the passt package puts the profile back:

    sudo sed -i 's|^\(\s*include <abstractions/pasta>\)$|\1\n  signal (receive) set=(term) peer=podman,|' /etc/apparmor.d/usr.bin.pasta
    sudo apparmor_parser -r /etc/apparmor.d/usr.bin.pasta
    

Kubernetes

  • The Sync server's pod stays in Init: (Init:Error, Init:CrashLoopBackOff). Read the logs of its init containers, in their order: kubectl logs deploy/ffs-firefox-sync-syncstorage -c wait-db (the database servers answer), -c db-bootstrap (your own server: its users and grants, a refused password stops it here), -c sync-role (PostgreSQL), -c migrate (a refused password of the bundled servers), -c db-init (maxUsers). A database server: kubectl logs sts/ffs-firefox-sync-tokenserver-db and sts/ffs-firefox-sync-storage-db (MariaDB, MySQL), sts/ffs-firefox-sync-postgres (PostgreSQL).
  • The accounts' auth pod stays in Init:. Read its init containers' logs in their order: kubectl logs deploy/ffs-firefox-sync-fxa-auth -c keys-copy (with fxa.keys.existingSecret), -c wait-db, -c db-bootstrap (your own server: its users and grants), -c migrations (a refused password, or the database image of MySQL 9), -c policy, -c wait-redis. The content and profile pods wait for it in wait-accounts and wait-auth.
  • CreateContainerConfigError. A key is missing from your existingSecret: kubectl describe pod names it. See existingSecret.
  • helm install fails with firefox-sync: ... before anything starts. The values are refused: a release name too long, database.bundled=false without database.external.host or existingSecret, the admin UI's Ingress on the Sync server's host... The message says what to set.
  • helm upgrade fails with StatefulSet.apps ... is invalid: spec: Forbidden: updates to statefulset spec. database.persistence.* changed, which Kubernetes never changes. The chart refuses that change first, unless it cannot read the cluster (helm template, a GitOps render). See Volumes.
  • 413 Request Entity Too Large in about:sync-log. The body size of the Ingress: see Ingress.
  • A node keeps an old latest image. With image.pullPolicy: IfNotPresent it does not pull a newer one: see Helm.
  • The bundled MySQL in CrashLoopBackOff on kind or k3s, on an Ubuntu host with mysql-server installed. The host's AppArmor profile for /usr/sbin/mysqld also confines the mysqld of the mysql image, in pods that run unconfined. It denies /etc/my.cnf, /usr/share/mysql-8.4 and the init file of the admin UI's users: the first start aborts half-way, and every next one finds no mysql.user. Disable that profile (the host's own MySQL then runs unconfined too), then delete the PVC of the half-made database:

    sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/ && sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
    

    Or use MariaDB (the default) or PostgreSQL. Docker's containers are not affected: they run in docker-default.

  • helm upgrade fails with firefox-sync: database.image.<backend> is ... but the bundled database ... runs .... The new image would move the data to another major: see Database majors.

Admin UI

  • A setup check fails. Each check says what is wrong and what to do: see Setup checks.
  • The admin UI does not start, and says ffs_admin_ui cannot create its tables. Its grant on ffs_admin is missing on your own server: run the new database-setup.sql there (The admin UI's own database).
  • With Quadlets on your own server, the admin UI does not start after an upgrade. Its unit has only ADMIN_DB_PASSWORD: create its users first (With Quadlets).
  • Max users edits, slot resets or account deletions fail with a missing procedure. An older init image runs with the newer admin UI: see Rollback.