Check the network speed with an SFTP connection directly to your server. Upload / download a file from your location to your server.
The speed check on our server performed by our hoster was also fine. No surprise - as the speed test software connects to the best available server, which might be connected on a different network than yours.
The direct test without VPN showed the same issue - so it was not related to VPN, but to the general connection between my office, homeoffice + other places and our server.