Troubleshooting¶
Symptoms and their fixes, by area. Each entry starts with what you see: an error text, a state, a log line. For how the stack is set up, start from the docs index.
Sync server¶
docker compose up -dfails, the Sync server stays down, withERROR: SYNC_X and X are both set .... A setting has both its old and its new name, in.envor in the environment docker compose runs in. Run./prepare_environment.sh --upgradefor.env, orunsetthe old name in your shell. See When both names are set.- The init container refuses to start on an empty value, or on a
<REPLACE_WITH_...>placeholder. The public URL or a database password was never filled in. Set it in.env(Settings). - The init container logs a warning listing old setting names, and the setup check "Old setting names (.env, environment)" fails. Run
./prepare_environment.sh --upgrade: see Renamed settings. 413 Request Entity Too Largeinabout:sync-log. The reverse proxy's limit of the request body is lower than the Sync server's: raise it, see Request size. With the Helm chart, the Ingress annotation: see Ingress.- Anything else in
SYNC_MAX_USERS(empty, not a number) stops the init container, and with it the Sync server. The forms it takes: Limits. - The account limit does not change after editing
.env. Withinitial:Nthe value is set only once; after that the admin UI manages it. See Limits.
Databases¶
ERROR 1419 ... You do not have the SUPER privilege and binary logging is enabled(init container orfxa-policy). Setlog_bin_trust_function_creators=ONon the server: see Server requirements.ERROR 2026 ... self-signed certificate in certificate chain, then "Timeout waiting for database connection" in the init container. SetSYNC_DB_TLS_VERIFY=false(Helm:database.external.tlsVerify: false).- "Timeout waiting for database connection", or the migrations never finish. The containers cannot reach the server; the init container's message ends with the database client's last error (a name that does not resolve, a refused connection, a login). Check the host (not
localhost), the port, the server's listen address, the firewall and the users' host part. Test from the stack's network:docker compose run --rm --no-deps --entrypoint mariadb tokenserver_db_init --host=... --user=... -p --skip-ssl-verify-server-cert -e 'SELECT 1'. With the Helm chart:kubectl logs deploy/<release>-firefox-sync-syncstorage -c wait-db(the server answers) and-c db-bootstrap(the logins). the Sync databases on your server lack what this release needs(or the accounts' databases), indb-bootstraporfxa-db-bootstrap. A user, grant or database ofdatabase-setup.sqlis missing, or a login is refused: the error names the statement.setup migrations NNN not applied: this release needs a file your server has not run, its SQL follows in the log. See When a release needs new users or grants.setup migrations of ... were applied as another file than this release's, indb-bootstrap. A file ofdb/*/setup/differs from the one that ran on your server: an edited checkout or image. Use this release's files; see Setup migrations.Access deniedfrom PostgreSQL, orno pg_hba.conf entry. Add ahostline for the containers' addresses: see Server requirements.- A MariaDB or MySQL container does not start, with
keep-major.sh: this database is MySQL 8, the image is MySQL 9(or MariaDB). Its image setting (SYNC_MARIADB_IMAGE,SYNC_MYSQL_IMAGE,FXA_MARIADB_IMAGEorFXA_MYSQL_IMAGE) is not set, and itsltsdefault is another major than the data../prepare_environment.sh --upgradepins it (The database majors of an install from before). To move the data to the new major instead: MySQL 8 to 9. Plugin 'mysql_native_password' is not loaded(MySQL 9, also forroot). A user is still onmysql_native_password: for example an install from before moved tomysql:ltswithout step 2 of MySQL 8 to 9. Back onFXA_MYSQL_IMAGE=mysql:8the data no longer opens, because MySQL 9 upgraded it. Either restore the backup of that procedure (Restore), or start MySQL 9 once with--skip-grant-tables, runFLUSH PRIVILEGESand theALTER USERof step 2.- The PostgreSQL container does not start, with
keep-major.sh: this database is PostgreSQL 18 ... Not started.The image is another major than the data:SYNC_POSTGRES_IMAGEis unset andpostgres:latestmoved on, or the pin changed. Set it to the major the data has (the message names it): see PostgreSQL. - The setup check of the event log fails, and the admin UI's log says the event log is in the tokenserver database. An install from before
ADMIN_DB_PASSWORD: see The admin UI's own database.
Self-hosted accounts¶
ER_NOT_SUPPORTED_AUTH_MODEfromfxa-migrationsorfxa-auth. The accounts image has no mozilla/fxa#21709 (Mozilla's images, or ours from before). ThenFXA_MYSQL_USERmust usemysql_native_password, on MySQL 8.4 withmysql_native_password=ON(fxa-db:FXA_MYSQL_IMAGE=mysql:8withFXA_MYSQL_ARGS). Use our current images for MySQL 9: see MySQL or MariaDB.ER_CANT_DROP_FIELD_OR_KEY: Can't DROP FOREIGN KEY ..._ibfk_1fromfxa-migrations. The server is MariaDB 12 and the accounts image does not have mozilla/fxa#21494 (Mozilla's images, or ours from before). Use our current images, or MariaDB 11.4 to 11.8 on a new database: see MariaDB 12.fxa-migrationsstops at once withAccess denied for userandnot retried. The accounts database refusedFXA_MYSQL_USER: a wrongFXA_MYSQL_PASSWORD, or missing grants. Unlike a server that does not answer yet, which is retried for 5 minutes, this stops at the first try. Fix the password or the grants (Your own server), then start again. With the Helm chart the auth pod stays in its init containers until then.docker compose up -dstops with an error after an upgrade.fxa-migrationsfailed:docker compose logs fxa-migrationssays why. To go back to the older release, restore the backup: see Self-hosted accounts.fxa-authstops at once withConfig 'mfa.jwt.secretKey' must be set in production(ormust be at least 32 bytes), and staysunhealthy. The accounts image is v1.349.0 or later, the checkout older: since v1.349.0 the auth server needs its own key for the tokens that guard the password, email and 2-step changes. Update the checkout (git pull), thendocker compose up -d(Quadlets:./prepare_environment.sh --render-units, then restart the units it names; Helm: a chart from then on). It derives the key fromFXA_OAUTH_SECRET, nothing to add to.env: see Self-hosted accounts.- The accounts services stay
(health: starting). They can take a few minutes after a start or an upgrade. - A new release runs on the old schema with Quadlets.
podman auto-updatereplaced the accounts services without running the migrations: see Quadlets. - Firefox refuses to sign in to your accounts server. The release is too old for that Firefox (for example Firefox 154 needs a recent one, because it switched to Bearer tokens). Upgrade: see Self-hosted accounts.
- Firefox signs in to the account, but Sync never starts; the admin UI's setup check says "Sign-in keys (FXA_OAUTH_SERVER_URL) ... connection refused". See Sign-in keys: connection refused.
docker compose up -dfails withbind: address already in usefor a port of the host (127.0.0.1:9000,127.0.0.1:5000...). Something else on the host has that port. Run./prepare_environment.sh --upgrade: it moves the ports something else took to the next free ones and writesproxy/again (Ports in use); then put the newproxy/file into your proxy. By hand: change only that*_EXPORT_PORTin.env, then./prepare_environment.sh --render-proxy.
Sign-in keys: connection refused¶
The Sync server checks every sign-in of Firefox with the keys at FXA_OAUTH_SERVER_URL. With the self-hosted accounts that is http://fxa-auth:9000/v1, container-to-container: inside the stack the accounts API always listens on fxa-auth:9000. FXA_AUTH_EXPORT_PORT is something else, the port of the host your reverse proxy reaches it on (127.0.0.1:9000 by default). Changing the host port, for example to 127.0.0.1:9010 because the host had 9000 in use already, must not change FXA_OAUTH_SERVER_URL: set to http://fxa-auth:9010/v1, nothing answers there, every sign-in to Sync fails, and the admin UI's setup check Sign-in keys says connection refused (with the explicit FXA_OAUTH_SERVER_URL is container-to-container message). prepare_environment.sh --upgrade and --render-proxy warn of it too.
Fix .env:
FXA_OAUTH_SERVER_URL=http://fxa-auth:9000/v1
FXA_AUTH_EXPORT_PORT=127.0.0.1:9010
then docker compose up -d (Quadlets: ./prepare_environment.sh --render-units, then restart the units it names). The setup itself picks a free host port for you when one is in use (Ports in use), and never changes the ports inside the stack.
Backups¶
- The
backuporbackup-fxacontainer isunhealthy. Its loop stopped, its last run failed, or the last good run is more than 25 hours old.docker compose logs backupsays which. See Check that it runs. - A run failed with "another backup run ... was still going on". A run waits for the one going on at most an hour, then fails. Check what holds the lock: a run that was killed has its lock taken over by the next run.
- A restore loaded nothing, but did not fail. Without
gunzip -ta missing run (mistyped, or pruned byBACKUP_KEEP) gives the client an empty input, which it takes as a success. Use the commands of Restore: they check the file first. - The restore of
ffsfails on aDEFINER. Restore the accounts as the user that made the dumps (FXA_MYSQL_USER): see Restore.
Podman¶
- A setting changed in
.envhas no effect. The installed units hold their own copy of the values: change it there, see Change a setting. prepare_environment.shstops withERROR: <unit> needs <NAME>, which is not set in .env. A unit has a%NAME%placeholder the script has no value for: the units ofconfig/quadlet/and the script come from different releases. Use the files of one release.- A one-shot unit does not run again (the migrations, the init container, the sign-up policy). They stay "active" after their first run. Stop them along with the units that need them, as in Quadlets.
- The Sync server's state, logs and health checks. See Check the units.
- MySQL containers that cannot find
/etc/my.cnfor their error messages, and that Podman cannot stop (given PID did not die within timeout), on an Ubuntu host withmysql-serverinstalled. The host's AppArmor profile for/usr/sbin/mysqldconfines themysqldof themysqlimage in rootless Podman's containers too, as in Kubernetes: disable that profile as said there, or use MariaDB (the default) or PostgreSQL. - An init container stops with
Timeout waiting for database connection (postgres_db:5432): ... could not translate host name "postgres_db"(orUnknown server host 'tokenserver_db') right afterdocker compose up -drecreated the stack, with docker compose on Podman 4 (Ubuntu 24.04's Podman 4.9). The database container runs and is healthy, but Podman's DNS (aardvark-dns) does not find it: now and then Podman 4's network tools lose track of a container that compose recreated along with the others of its network. Rundocker compose down(it keeps the volumes, unlikedown -v), thendocker compose up -d. We have seen it on Podman 4.9 only, not on Podman 5. -
rootless netns: kill network process: permission deniedon Ubuntu 26.04. Ubuntu's AppArmor profile ofpastarefuses the signal Podman sends it when the last container of its bridge networks goes (Ubuntu bug 2154379):docker compose upon the Podman socket cannot remove the containers it recreates. Until Ubuntu fixes it, letpastareceive that signal, as CI does (pasta-apparmor.sh), and again after an update of thepasstpackage puts the profile back:sudo sed -i 's|^\(\s*include <abstractions/pasta>\)$|\1\n signal (receive) set=(term) peer=podman,|' /etc/apparmor.d/usr.bin.pasta sudo apparmor_parser -r /etc/apparmor.d/usr.bin.pasta
Kubernetes¶
- The Sync server's pod stays in
Init:(Init:Error,Init:CrashLoopBackOff). Read the logs of its init containers, in their order:kubectl logs deploy/ffs-firefox-sync-syncstorage -c wait-db(the database servers answer),-c db-bootstrap(your own server: its users and grants, a refused password stops it here),-c sync-role(PostgreSQL),-c migrate(a refused password of the bundled servers),-c db-init(maxUsers). A database server:kubectl logs sts/ffs-firefox-sync-tokenserver-dbandsts/ffs-firefox-sync-storage-db(MariaDB, MySQL),sts/ffs-firefox-sync-postgres(PostgreSQL). - The accounts' auth pod stays in
Init:. Read its init containers' logs in their order:kubectl logs deploy/ffs-firefox-sync-fxa-auth -c keys-copy(withfxa.keys.existingSecret),-c wait-db,-c db-bootstrap(your own server: its users and grants),-c migrations(a refused password, or the database image of MySQL 9),-c policy,-c wait-redis. The content and profile pods wait for it inwait-accountsandwait-auth. CreateContainerConfigError. A key is missing from yourexistingSecret:kubectl describe podnames it. See existingSecret.helm installfails withfirefox-sync: ...before anything starts. The values are refused: a release name too long,database.bundled=falsewithoutdatabase.external.hostorexistingSecret, the admin UI's Ingress on the Sync server's host... The message says what to set.helm upgradefails withStatefulSet.apps ... is invalid: spec: Forbidden: updates to statefulset spec.database.persistence.*changed, which Kubernetes never changes. The chart refuses that change first, unless it cannot read the cluster (helm template, a GitOps render). See Volumes.413 Request Entity Too Largeinabout:sync-log. The body size of the Ingress: see Ingress.- A node keeps an old
latestimage. Withimage.pullPolicy: IfNotPresentit does not pull a newer one: see Helm. -
The bundled MySQL in
CrashLoopBackOffon kind or k3s, on an Ubuntu host withmysql-serverinstalled. The host's AppArmor profile for/usr/sbin/mysqldalso confines themysqldof themysqlimage, in pods that run unconfined. It denies/etc/my.cnf,/usr/share/mysql-8.4and the init file of the admin UI's users: the first start aborts half-way, and every next one finds nomysql.user. Disable that profile (the host's own MySQL then runs unconfined too), then delete the PVC of the half-made database:sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/ && sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqldOr use MariaDB (the default) or PostgreSQL. Docker's containers are not affected: they run in
docker-default. -
helm upgradefails withfirefox-sync: database.image.<backend> is ... but the bundled database ... runs .... The new image would move the data to another major: see Database majors.
Admin UI¶
- A setup check fails. Each check says what is wrong and what to do: see Setup checks.
- The admin UI does not start, and says
ffs_admin_uicannot create its tables. Its grant onffs_adminis missing on your own server: run the newdatabase-setup.sqlthere (The admin UI's own database). - With Quadlets on your own server, the admin UI does not start after an upgrade. Its unit has only
ADMIN_DB_PASSWORD: create its users first (With Quadlets). - Max users edits, slot resets or account deletions fail with a missing procedure. An older init image runs with the newer admin UI: see Rollback.