Running Vaultwarden in a server cluster / High Availability

[ThomasLeister]

I've recently started using my own Vaultwarden based server. Now that part of my credentials reside on my server, I'm thinking about making it more resilient to outages. I'm aware that Bitwarden clients usually store a cached version of my password safe locally, but anyway I'd like to make Vaultwarden highly available somehow.

The most obvious choice would be to run Vaultwarden on at least three different servers and link its MySQL databases to a cluster, e.g. via Mariadb Galera Cluster. I'm just not sure if this is enough and if sessions can easily be handled by multiple servers at the same time. If there something like an internal state of Vaultwarden that is not stored in MySQL / database and needs to be synced separately? Is Vaultwarden even capable of being run in multiple instances for one password safe?

[BlackDex]

I suggest to read: https://vaultwarden.discourse.group/t/is-the-bitwarden-rs-service-stateless-if-its-using-postgres-backend/450

[BlackDex]

No. Any push messages is only sent on a change. That change is handled by one server and that server can send a message just fine too the push servers. Doesn't matter which server sends it.

That isn't the same for the websockets. Every server which has active websocket connections need to know about a change from any active server, doesn't matter if that server has websocket connections or not which receives the update from the client. That communication need to be handled somehow internally.

MQ could be a solution, but i do not really like the need to have an extra service running for this to work, like sn MQ service. That is why i mentored the gossip protocol. That should in theory be possible to do something like this.

[archef2000]

Many services use an extra service to make HA work (like redis). And what if the servers can't reach each other and only share the db and a redis connection. (If you have mulitiple locatioons that are only reachable localy for example)

Aren't push messages also used for "login with device"?

But with gossip it is like a ring token right? But would latency not be a problem with multiple servers in the chain and also they need to reach each other.

[BlackDex]

If all servers can reach the same Redis/MQ/DB server, then they probably also can talk to each other.

Push messages are indeed used for the Login with device. But the server in the end sends a WebSocket message with the right data to the right connection. And that connection in an HA setup can be on any server, not necessarily on the same server which receives the validation from the client device. So, that somehow needs to be communicated between all servers, hence why i say that all websocket messages need to be handled by all servers, not just one.

Gossip kinda is a ring yes. And latency isn't that huge i think. And a small amount of latency isn't that bad i think.

Also, there are different HA definitions.
HA In the sense of really High Available, if one server goes down, the other takes it over, that can be either hot or cold standby, doesn't matter that much.
Or
HA In the sense of load balancing, where each server is active and can handle all connections from any client no matter where they come from.

In my opinion, the challenges you are talking about are mostly tied to the Load Balancing example. For which i actually see no use case for Vaultwarden. It can handle the traffic well enough with just one instance. Maybe there are some differences in which type of Database you use which can help a bit. But that is about it i think.

So, from my point of view, a standby HA setup is more relevant for a tool like Vaultwarden, and that doesn't create these issues at all.
The only thing which needs to be checked is the file storage that it is shared. The websocket connections will try to reconnect and that will work just fine on the other instance, and once all websocket connections are over to the other instance they all receive the correct messages.

[archef2000]

I would like to setup Vaultwarden on two servers so i can also update my vault if one of them is down but I would probably set the DNS up in round robin so it would be kind of load balancing. And with the servers reaching each other maybe you have some authentication infront of vaultwarden so you would need to create a seperate channel and I think that, that an extra service that is made for these things in the back is not too much to ask from the user and you can cover more use cases as it would be universal.

As redis is easy to setup for testing and also make it HA with a cluster see the example of nextcloud.

As I don't know any MQ service that is low on resources and easy to implement and run.

What thinks would need to be "shared" besides the files if it would HA with LB?

[archef2000]

Would you be open to making Vautwarden HA (load balancing) if I would make a PR? With shared websocket and login requests?