You can find that file in the systemd mount file for the Cloudron backup.

systemctl status mnt-cloudronbackup.mount
● mnt-cloudronbackup.mount - cloudronbackup
     Loaded: loaded (/etc/systemd/system/mnt-cloudronbackup.mount; enabled; preset: enabled)
     Active: active (mounted) since Sat 2025-05-17 07:04:02 UTC; 2 weeks 1 day ago
      Where: /mnt/cloudronbackup
       What: REDACTED@REDACTED.your-storagebox.de:/home
      Tasks: 11 (limit: 9212)
     Memory: 9.9M (peak: 18.1M swap: 2.5M swap peak: 11.1M)
        CPU: 25min 37.199s
     CGroup: /system.slice/mnt-cloudronbackup.mount
             ├─ 905 /sbin/mount.fuse.sshfs REDACTED@REDACTED.your-storagebox.de:/home /mnt/cloudronbackup -o rw,allow_oth>
             └─9487 ssh -x -a -oClearAllForwardings=yes -oport=23 -oIdentityFile=/home/yellowtent/platformdata/sshfs/id>

Notice: journal has been rotated since unit was started, output may be incomplete.

So the file in question would be:

/etc/systemd/system/mnt-cloudronbackup.mount

I wonder if it still does any buffering when writing to /dev/null ?

If it's going through FUSE, it certainly does. It likely also does going through the FS for /dev/null.

Yet Another Data Point - I did a lot more testing today, and I think I'm as far as I can go. The good news: I can consistently get 16 - 25 MB/s read speeds.

TL;DR: using this command gives me the best read performance (2x-3x improvement): nice -n -10 sshfs -s -o direct_io,compression=no

Why I'm using these options:

direct_io

direct_io disables caching, and had quite an interesting effect on reads.

Using the -f -d options I was able to watch the packets going through. I was wrong before about the writes being bigger than the reads; they're not. But the writes are being done more parallel than the reads.

Before direct_io:

[01315] READ
  [01308]           DATA    32781bytes (31ms)
  [01309]           DATA    32781bytes (31ms)
  [01310]           DATA    32781bytes (31ms)
  [01311]           DATA    32781bytes (31ms)
[01316] READ
[01317] READ
[01318] READ
[01319] READ
  [01312]           DATA    32781bytes (31ms)
  [01313]           DATA    32781bytes (31ms)
  [01314]           DATA    32781bytes (31ms)
  [01315]           DATA    32781bytes (31ms)

READ requests 4 chunks at a time, waits for them, and then requests 4 more.

[05895] WRITE
  [05827]         STATUS       28bytes (34ms)
  [05828]         STATUS       28bytes (34ms)
  [05829]         STATUS       28bytes (35ms)
  [05830]         STATUS       28bytes (35ms)
  [05831]         STATUS       28bytes (35ms)
  [05832]         STATUS       28bytes (34ms)
  [05833]         STATUS       28bytes (34ms)
  [05834]         STATUS       28bytes (34ms)
  [05835]         STATUS       28bytes (34ms)
[05896] WRITE
[05897] WRITE

WRITE requests at least 60 chunks at a time, and sometimes I saw over 100 chunks pending.

After turning on direct_io, the reads look more like the writes:

[06342] READ
[06343] READ
[06344] READ
  [06313]           DATA    32781bytes (31ms)
  [06314]           DATA    32781bytes (31ms)
  [06315]           DATA    32781bytes (31ms)
  [06316]           DATA    32781bytes (31ms)
  [06317]           DATA    32781bytes (32ms)
  [06318]           DATA    32781bytes (32ms)
  [06319]           DATA    32781bytes (32ms)
  [06320]           DATA    32781bytes (32ms)
  [06321]           DATA    32781bytes (33ms)
  [06322]           DATA    32781bytes (35ms)
  [06323]           DATA    32781bytes (35ms)
  [06324]           DATA    32781bytes (36ms)
  [06325]           DATA    32781bytes (36ms)
  [06326]           DATA    32781bytes (36ms)
  [06327]           DATA    32781bytes (37ms)

Note the difference in the chunk IDs and you can see it's allowing at most 31 chunks to be pending before requesting more.

I think this is the primary reason for the speed increase.

-s for single threading

I noticed that running it on a single thread caused the degredation of the repeated file reads to be less pronounced. Instead of dropping back to 8 MB/s after a few reads, it does 25 MB/s read at least 5-6 times (500-600 MB) before dropping down to 16 MB/s. Also, it recovers back to 25 MB/s over time, whereas with multi-threaded I needed to restart the SSHFS connection in order to get 25 MB/s speeds again.

nice

Since there seems to be an element of CPU bottleneck (as resolved by running in a single process) I also wanted to give this process priority. It seems to help the session get more 25 MB/s reads before slowing down

compression=no

Because we're now on one thread, and we're hogging lots of CPU time, I disabled compression. I didn't notice a difference in throughput with it on, but turning it off helps reduce CPU load

Next Steps:

I will run this test a few more times, and probably even adjust my mount for the volume manually to see if it helps performance.

There is definitely some element of throttling / filling up, because repeated reads in the same session can get slower, and starting a new session can help the speed go back up. I'm not sure if this is on the client side or the server side. Any insights would be greatly appreciated.

Even though I wish there was a clearer answer, I'll be happy if the 2x boost to read speed works.

P.S. - I even tried a "high performance SSH" binary hpnssh, and it did not make a noticeable difference in my tests.

You should see a difference with SCP file transfers being fast, since it just streams data of known size down the pipe.

The following links may provide more hints:

1. https://stackoverflow.com/questions/75574954/read-write-syscall-in-fuse
2. https://www.reddit.com/r/linuxquestions/comments/jmtn0k/can_you_help_me_understand_sshscpsshfs_better/
3. https://superuser.com/questions/344255/faster-way-to-mount-a-remote-file-system-than-sshfs

And here are some insights on FUSE from someone who built their own FS on FUSE:
https://blog.gistre.epita.fr/posts/angelina.kuntz-2024-09-22-how_to_implement_our_own_file_system/

I decided to do another test with my laptop <-> hetzner server. So now we're transferring between the SSD on the VPS and the SSD on my laptop. The speeds are the same:
writing from laptop to VPS: 50 MB/s
reading from VPS to laptop: 7 MB/s

I checked and both my laptop and Cloudron are using the same version of SSHFS and Fuse:

SSHFS version 3.7.3
FUSE library version 3.14.0
using FUSE kernel interface version 7.31
fusermount3 version: 3.14.0

To get the best picture possible of the traffic, I put a fast watch on the connection:
watch -n 0 ss -t state established dst <server_ip>/24

With this, I can see that the write sizes are ~10x bigger than the read sizes. I'm wondering if this is why the performance difference is only seen in WAN situations; because 10x the round trips hurts a lot more on the WAN than the LAN. And for those of us with storage boxes in Europe and servers outside of europe, 10X the round trips really hurts.

I finally stumbled across this article (https://www.admin-magazine.com/HPC/Articles/Sharing-Data-with-SSHFS) which does some pretty detailed performance testing and tuning with SSHFS.

The options they suggest for the sshfs mount didn't help much. I did notice that using direct_io can sometimes make the read speeds go up to 20 MB/s, but it's not reliable.

I'm wondering if we're now into the realm of TCP configuration (which is the article's option #2) to increase a TCP buffer sizes. This would be a server-wide change, and is out of my depth. What are your thoughts, @nebulon ?

Since SSHFS is barely maintained, are there other volume options coming? From what I've read, CIFS could work if we override the user ID at the mount configuration level.

nebulon@saturn:/mnt/cloudronbackup/test$ dd if=/dev/zero of=tempfile bs=1M count=100 conv=fdatasync
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 20.1171 s, 5.2 MB/s
nebulon@saturn:/mnt/cloudronbackup/test$ dd if=tempfile of=/dev/null bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 10.0484 s, 10.4 MB/s
....
scp -i /home/yellowtent/platformdata/sshfs/id_rsa_... -P 23 tempfile u193526@u193526.your-storagebox.de:./test/
tempfile                                                                                                  100%  100MB   4.9MB/s   00:20    

So I can't say I can reproduce those results.

For the cloudron backup. Once a snapshot is uploaded for backups, the rotation and uploading of only changed data is very fast. The backup rotation is special cased for sshfs to run the filesystem operations to create hardlinks directly on the storage server. This makes it fast. During a restore, now all the data has to be downloaded, which will of course take much much longer than just uploading the changed data every day.

If you find any sshfs/fuse mounting options which improve the performance, the current mountpoint systemd config looks like:

...
Options=allow_other,port=23,IdentityFile=/home/yellowtent/platformdata/sshfs/id_rsa...,StrictHostKeyChecking=no,reconnect
Type=fuse.sshfs

So basically Cloudron sets up the mountpoint with all defaults, maybe there is some room for improvement. I will run some tests in a bit on my sshfs volume.

The remote server I'm mounting through SSHFS is a server in my LAN and my cloudron server is with Hetzner. My home connection is gigabit fiber.

I did my testing by mounting via SSHFS and just seeing how quickly I could read/write with this:

dd if=/dev/zero of=tempfile bs=1M count=100 conv=fdatasync
dd if=tempfile of=/dev/null bs=1M count=100

For my SCP test, I just copied the same file both directions.

I tried different sizes and counts for dd; the writing (of=tempfile) is always 50MB/s, and the reading (if=tempfile) is always 7MB/s or less.

Still, there might be some parameters in Cloudron for SSHFS that might could get improved.
Could you disclose how you did your testing and also what provider for SSHFS you are using?