Upgrade from 1.30.0 to 1.30.1, log shows a large number Upgraded websocket I/O handler failed: WebSocket protocol error: Sending after closing is not allowed

[keepalivesrc]

After upgrading to version 1.30.1, this type of log appeared and I have no idea what happened. Please help me

1. This is the log error message after upgrading to 1.30.1

2. This is my nginx configuration

map $http_upgrade $connection_upgrade

server {
        listen 8090 ssl http2;
        server_name vaultwarden.example.tld;
        error_page 400 497 =444 /;

        # SSL
        ssl_certificate /etc/nginx/ssl/fullchain.cer;
        ssl_certificate_key /etc/nginx/ssl/prive.key;

        # security headers
        add_header X-XSS-Protection "1; mode=block" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header Referrer-Policy "strict-origin" always;
        add_header Content-Security-Policy "default-src 'self' https: 'unsafe-inline'; frame-ancestors 'self';" always;
        add_header Permissions-Policy "interest-cohort=()" always;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

        # Block forbidden city
        if ($allowed_country = no) {
            return 444;
        }

        # restrict path
        location / {
            return '444';
        }

        # restrict methods
        if ($request_method !~ ^(GET|POST|PUT|DELETE)$) {
            return '444';
        }

        # reverse proxy
        location /vault/ {
            proxy_pass http://127.0.0.1:8080;
            proxy_set_header Host $host;
            proxy_http_version 1.1;
            proxy_cache_bypass $http_upgrade;

            # Proxy SSL
            proxy_ssl_server_name on;

            # Proxy headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Forwarded $proxy_add_forwarded;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Port $server_port;

            # Proxy timeouts
            proxy_connect_timeout 60s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;

            proxy_buffering on;
            proxy_buffer_size 512k;
            proxy_buffers 512 1024k;
            proxy_busy_buffers_size 2048k;
            client_max_body_size 100M;
            client_body_buffer_size 20M;
        }
    }

3.This is my vaultwarden configuration

ROCKET_PROFILE=release
DEBIAN_FRONTEND=noninteractive
SIGNUPS_ALLOWED=false
INVITATIONS_ALLOWED=false
WEB_VAULT_ENABLED=false
ROCKET_ADDRESS=127.0.0.1
ROCKET_PORT=8080
DOMAIN=https://vaultwarden.example.tld/vault/

[BlackDex]

Hello,

There is nothing changed between those versions which could cause this i think.

I would start by removing all the security headers except for the STS, Vaultwarden already sends these headers with all the correct values when needed.

And i think the buffering might cause some issues here. And maybe caching also has some influence, but about that I'm not sure.

[Ducky6944]

Just wanted to through this out there to keep possible solutions in a single place. Was having the same issue, the Specific Header authorization that I had listed in a forward auth setup for authentik in traefik was the culprit.

[rama31244]

How did you fix it typkrft? I have authelia forward auth set up in traefik too and am not sure how to check the authorization header

[Ducky6944]

@rama31244 in my case I was adding the authorization header in a forwardAuth middleware. This problem might have multiple etiologies, but for me it was specifically that head. If you suspect thats the issue just look at your middlewares, maybe authelia is adding it. You can look in your requests to see if it's there.

[rama31244]

I'm fairly new to networking so sorry if this a stupid question but how can I check requests to look for the header? I use Firefox browser with the bitwarden extension and vaultwarden runs as a docker container on my server

[Ducky6944]

@rama31244 the easiest way would be just opening the console in your browser, going to the network tab, reloading the page and looking through headers of the requests.

[mada199122]

this Upgraded websocket I/O handler failed: WebSocket protocol error: Sending after closing is not allowed error started to apper since the very last update 1.30.1. I've never seen it with the 1.30.

I use the Swag reverse proxy so it is NGINX base.
The config file that I'm using is the following

## Version 2023/11/12
# make sure that your vaultwarden container is named vaultwarden
# make sure that vaultwarden is set to work with the base url /vaultwarden/
# if you are using bitwarden (the official image), use the bitwarden conf
# if you are using vaultwarden (an unofficial implementation), use the vaultwarden conf
#
# vaultwarden defaults to port 80 and can be changed using the environment variable ROCKET_PORT on the vaultwarden container
#
# Environmental Variable DOMAIN=https://<DOMAIN>/vaultwarden must be set in vaultwarden container including subfolder.

location /vaultwarden {
    return 301 $scheme://$host/vaultwarden/;
}

location ^~ /vaultwarden/ {
    # enable the next two lines for http auth
    #auth_basic "Restricted";
    #auth_basic_user_file /config/nginx/.htpasswd;

    # enable for ldap auth (requires ldap-server.conf in the server block)
    #include /config/nginx/ldap-location.conf;

    # enable for Authelia (requires authelia-server.conf in the server block)
    #include /config/nginx/authelia-location.conf;

    # enable for Authentik (requires authentik-server.conf in the server block)
    #include /config/nginx/authentik-location.conf;

    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app vaultwarden;
    set $upstream_port 80;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;

}

location ~  ^(/vaultwarden)?/admin {
    # enable the next two lines for http auth
    #auth_basic "Restricted";
    #auth_basic_user_file /config/nginx/.htpasswd;

    # enable for ldap auth (requires ldap-server.conf in the server block)
    #include /config/nginx/ldap-location.conf;

    # enable for Authelia (requires authelia-server.conf in the server block)
    #include /config/nginx/authelia-location.conf;

    # enable for Authentik (requires authentik-server.conf in the server block)
    #include /config/nginx/authentik-location.conf;

    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app vaultwarden;
    set $upstream_port 80;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;

}

location ~ (/vaultwarden)?/api {
    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app vaultwarden;
    set $upstream_port 80;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;

}

location ~ (/vaultwarden)?/notifications/hub {
    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app vaultwarden;
    set $upstream_port 80;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;

}

[mada199122]

yes it is in the proxy.conf

this is my config, that is the version that SWAG is shipped with.

## Version 2023/02/09 - Changelog: https://github.com/linuxserver/docker-swag/commits/master/root/defaults/nginx/proxy.conf.sample

# Timeout if the real server is dead
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

# Proxy Connection Settings
proxy_buffers 32 4k;
proxy_connect_timeout 240;
proxy_headers_hash_bucket_size 128;
proxy_headers_hash_max_size 2048;
proxy_http_version 1.1;
proxy_read_timeout 240;
proxy_redirect http:// $scheme://;
proxy_send_timeout 240;

# Proxy Cache and Cookie Settings
proxy_cache_bypass $cookie_session;
#proxy_cookie_path / "/; Secure"; # enable at your own risk, may break certain apps
proxy_no_cache $cookie_session;

# Proxy Header Settings
proxy_set_header Connection $connection_upgrade;
proxy_set_header Early-Data $ssl_early_data;
proxy_set_header Host $host;
proxy_set_header Proxy "";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Method $request_method;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Uri $request_uri;
proxy_set_header X-Original-Method $request_method;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Real-IP $remote_addr;

the timeout is 240 so it should be fine, shouldn't it?

Now I removed the part related to /notifications/hub to the config. I don't use Authelia with Vaultwarden.

[BlackDex]

Ok, proxy_buffers might cause issues here.
The rest seems fine, though a lot of headers are set sending to Vaultwarden which we do nothing with.

Same btw for proxy_cache. Try to disable both for Vaultwarden.

[mada199122]

Is there an easy way to disable both specifically for vaultwarden?
Thanks

[BlackDex]

set this either within the server block or the individual location blocks

proxy_cache off;
proxy_buffering off;
proxy_request_buffering off;

[mada199122]

set the three parameter but I still have the error.
it seems less frequent then before but it is still present.

[BlackDex]

I'm looking at the logs again. I'm not sure if that is all, but i see a sigterm shutdown action but not really a fully shutdown message.
It might be that the Vaultwarden container didn't fully shutdown maybe? Could that be causing the issue?

Please check and verify if the container fully stopped before starting it again.

[BlackDex]

Also, if the reverse proxy sends a close afterwards, or closes the connection, then this could happen.
It looks to me the connection just doesn't stay open, and that is a configuration option of the reverse proxy.

It could also be something else in between, like firewall or CloudFront etc..

[mirolm]

This started to happen with the last update and i think after the last "update crates 4883650" commit. Easiest to reproduce is when you have the chrome extension and you lock it. I think this is where the ext closes the websocket and this message pops in the logs. Maybe server tries to do something with the websocket after the nginx already torn it down.

Nov 22 21:07:32 altair vaultwarden[3546]: [2023-11-22 21:07:32.046][vaultwarden::api::notifications][INFO] Closing WS connection from 192.168.88.1
Nov 22 21:07:32 altair vaultwarden[3546]: [2023-11-22 21:07:32.046][rocket::server][ERROR] Upgraded websocket I/O handler failed: WebSocket protocol error: Sending after closing is not allowed

[BlackDex]

Thx, I'll see if i can reproduce this using the same procedure

[FlakyPi]

Same here, those error messages never appeared before 1.30.1. I haven't changed anything at all in my proxy configuration since 1.29.0.

[BlackDex]

I checked my logs and i do also see them. Not that often though, but probably because i do not lock it that often.
I do think they can be ignored, but I will take a further look into them.

[ARVEDz]

Same on my end.

I can easily reproduce the error by locking my vault in the browser extension. If I deactivate "Websocket Support" in Nginx Proxy Manager though, the error doesn´t show up anymore, but granting other devices approval for log in also does not work anymore.

[BlackDex]

Which is expected since that uses websocket.
I also can reproduce it so I'll check further, but again, they do not harm the workings of Vaultwarden in any way.

[mada199122]

any news reguarding this error?
thanks

[BlackDex]

Nope. Did not had the time yet to look at this. Also, it's not a breaking error which makes Vaultwarden not work anymore.

[BlackDex]

I found the issue, and i also have fixed it.

[mada199122]

thank you!!!

[christofkac]

Can you please release a version with this fix? I'm sending all syslog error messages to my mobile via Telegram and this channel is now quite polluted ;-)
Thanks
Christof

[BlackDex]

You can also filter them.
There might be some other items we want/need to fix before we want to release a next version.

We also have no timeline for releases.

[christofkac]

OK, was worth a try ;-) Thanks for all your efforts, really cool tool!

[BlackDex]

It was indeed 😉

[rama31244]

And when you say reloading the page, do you mean my vaultwarden admin page?

…

On Tue, 2 Jan 2024, 2:52 pm Brandon, ***@***.***> wrote: @rama31244 <https://github.com/rama31244> the easiest way would be just opening the console in your browser, going to the network tab, reloading the page and looking through headers of the requests. — Reply to this email directly, view it on GitHub <#4090 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUKELZRWMH5IBP4WTEWVIOLYMOABNAVCNFSM6AAAAAA7T4XZUCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TSOJQGMZDA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>