Vaultwarden can't connect to MariaDB database since 1.30.0

[FlakyPi]

Subject of the issue

Vaultwarden can't connect to MariaDB database since updated to 1.30.0. It was working flawlesly on previous versions.
The database is accesible from command line using the same user and password.
Reverting to 1.29.2, Vaultwarden works well again.

Deployment environment

• OS version: Raspbian 12.1

• Vaultwarden version: 1.30.0

• Install method: systemd service, binary and web vault extracted from 1.30.0-alpine docker image

• Clients used: N/A

• Reverse proxy and version: N/A

• MySQL/MariaDB or PostgreSQL version: 10.11.3-MariaDB

• Other relevant details:

systemd service env:

LANG=es_ES.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/var/lib/vaultwarden
LOGNAME=vaultwarden
USER=vaultwarden
INVOCATION_ID=50612c529870406f9cccfafb664a01d9
JOURNAL_STREAM=8:42362
SYSTEMD_EXEC_PID=6990
DATABASE_URL=mysql://vaultwarden:mypassword@127.0.0.1/vaultwarden
TEMPLATES_FOLDER=/usr/share/vaultwarden/templates
WEB_VAULT_FOLDER=/usr/share/vaultwarden/web-vault/

Steps to reproduce

$ systemctl start vaultwarden

Expected behaviour

Vaultwarden starts normally and can access MariaDB database.

Actual behaviour

Vaultwarden fails to connect to MariaDB database and after several tries the service exits with an error result.

Troubleshooting data

Vaultwarden log:

/--------------------------------------------------------------------\
|                        Starting Vaultwarden                        |
|                           Version 1.30.0                           |
|--------------------------------------------------------------------|
| This is an *unofficial* Bitwarden implementation, DO NOT use the   |
| official channels to report bugs/features, regardless of client.   |
| Send usage/configuration questions or feature requests to:         |
|   https://github.com/dani-garcia/vaultwarden/discussions or        |
|   https://vaultwarden.discourse.group/                             |
| Report suspected bugs/issues in the software itself at:            |
|   https://github.com/dani-garcia/vaultwarden/issues/new            |
\--------------------------------------------------------------------/
[INFO] Using saved config from `data/config.json` for configuration.
[2023-11-07 22:47:22.918][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:23.932][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:24.945][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:25.960][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:26.974][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:27.988][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:29.003][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:30.015][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
RN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:32.043][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:33.055][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:34.070][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:35.083][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:36.096][vaultwarden::util][WARN] Can't connect to database, retrying: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)
[2023-11-07 22:47:37.111][vaultwarden][ERROR] Error creating database pool: DieselCon.
[CAUSE] BadConnection(
    "Bad handshake",
)

MySQL log:

2023-11-07 22:47:22 583 [Warning] Aborted connection 583 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:23 584 [Warning] Aborted connection 584 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:24 585 [Warning] Aborted connection 585 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:25 586 [Warning] Aborted connection 586 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:26 587 [Warning] Aborted connection 587 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:27 588 [Warning] Aborted connection 588 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:29 589 [Warning] Aborted connection 589 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:30 590 [Warning] Aborted connection 590 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:31 591 [Warning] Aborted connection 591 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:32 592 [Warning] Aborted connection 592 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:33 593 [Warning] Aborted connection 593 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:34 594 [Warning] Aborted connection 594 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:35 595 [Warning] Aborted connection 595 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:36 596 [Warning] Aborted connection 596 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
2023-11-07 22:47:37 597 [Warning] Aborted connection 597 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)

[david-samson]

Unable to update to server:latest (1.30.0) on docker installation via Portainer as well.
Logs aren't as close as being as explicit as the ones above.
It just won't start using the exact same config as with 1.29.2 which has been working flawlessly.

[BlackDex]

But what do your logs say then?
I Doubt it is the same issue actually

[david-samson]

Possible, it just doesn't say anything.

I keep the same config, only updating to the latest image, was on 1.29.2 and was updated to 1.30.0, and I get an error 400 in portainer and no logs at all in the console when I attempt to start the container.

Forcing the version back to vaultwarden/server:1.29.2 fixes the issue and, as soon as I change it back to either vaultwarden/server:latest of vaultwarden/server:1.30.0, the exact same behavior returns.

[BlackDex]

Which docker version are you running? And what kind of platform?

[BlackDex]

@FlakyPi both binaries are extracted from an alpine image?

[stefan0xC]

According to the bug report I've linked they have used a similar patch, which does not seem to work with 3.3.7 anymore. Not sure if their fix is correct though which is why I've linked the changes to the upstream repository.

[BlackDex]

Ah yes, i see it's still unresolved at MariaDB.
https://jira.mariadb.org/projects/CONC/issues/CONC-668?filter=allopenissues

I do see this: mariadb-corporation/mariadb-connector-c@8320f0d

[BlackDex]

I might revert back to 3.3.5, that is the same version currently used in 3.18 of Alpine. 3.3.7 is used in edge.

[BlackDex]

Reverted back to v3.3.5, images are being build: https://github.com/BlackDex/rust-musl/actions/runs/6804662255
And those then will be used the next time a Vaultwarden build is being triggered

[BlackDex]

MariaDB Connector-C v3.3.8 has been released which has a fix for this included.
I tested both armv6 and armv7 via Vaultwarden using this version and it works without issues.

The new Alpine build packages are currently building (my build images i mean)