Storage management in Immich?

[shrey]

What's the optimum way to manage the data (app code + uploaded files) and backups (of the app code)?

Currently, i've set the app to use an external Volume as the primary 'Data Directory'.

But, as the Data Directory also includes the uploaded files (which could be hundreds of GBs), it does seem prudent to have the uploaded files stored in a non-regular-backup location.
Initially, I did try mounting a volume and configuring the immich.json, but that didn't seem to work.

immich.json ->

{
"UPLOAD_LOCATION":"/media/<volume name>/data/immich_files",
"THUMB_LOCATION":"/media/<volume name>/data/thumbs",
"ENCODED_VIDEO_LOCATION":"/media/<volume name>/data/encoded-video",
"PROFILE_LOCATION":"/media/<volume name>/data/profile"
}

Error log ->

WARN [Api:DuplicateService] Unknown keys found: { "UPLOAD_LOCATION": "/media/<volume name>/data/immich_files", "THUMB_LOCATION": "/media/<volume name>/data/thumbs", "ENCODED_VIDEO_LOCATION": "/media/<volume name>/data/encoded-video", "PROFILE_LOCATION": "/media/<volume name>/data/profile" }

---

Immich references ->
https://immich.app/docs/guides/custom-locations
https://immich.app/docs/install/docker-compose#step-2---populate-the-env-file-with-custom-values

[nebulon]

Instead of setting up immich for external volume as storage, I would instead move the app as such to the volume: https://docs.cloudron.io/apps/#data-directory

[shrey]

@nebulon
That would imply: backups would include everything, no?

Which isn't really feasible, multiple backups a day, of potentially hundreds of GB of data.

[shrey]

@shrey said in Storage management in Immich?:

Currently, i've set the app to use an external Volume as the primary 'Data Directory'.

But, as the Data Directory also includes the uploaded files (which could be hundreds of GBs), it does seem prudent to have the uploaded files stored in a non-regular-backup location.

@nebulon

[nebulon]

This is true with the backups, however I wonder in this case what the benefit of backups is if not for the pictures itself. I am not sure what the app will do in case the file storage and database gets out-of-sync.

[girish]

@shrey I think it might be better to just set the data directory and just disable backups for immich instead. As you know, it's still in active development and moving around files will cause things to go out of sync with the database. This will just cause unnecessary headaches if we make this configurable. It is similar to nextcloud in this aspect (whose data directory can also not be configured in Cloudron package)

[shrey]

Hi @girish @nebulon, is a proper backup system for immich still pending?

[nebulon]

What is the requirement here specific to immich? After rereading the thread, you want metadata/database to be backed up but not the actual pictures? If so we probably won't support such setups as having a rsync backup storage with hardlink support (like for example hetzner storage box with sshfs) is very efficient already and the metadata/database backup is anyways mostly useless without the actual data to restore also.

But maybe I am missing some use-case here.

[shrey]

@nebulon said in Storage management in Immich?:

After rereading the thread, you want metadata/database to be backed up but not the actual pictures?

Pretty much. The media files can be stored in a mounted volume.

But currently, if i store everything in the Data Directory (that gets backed up, when enabled globally), then i'm at a big loss, because, i can neither set a separate (more lenient than other apps) backup frequency, nor can i escape the pain of having to manage my backup storages efficiently (i realised today that immich's backups were enabled in my case, which is 4 times a day, for a month, and around 150GB every time!!).

[nebulon]

In such cases the rsync backup with hardlinks, will not duplicate any data even if many backups are created. So may be worth moving to such a storage in your case.

Further we have per-app backup schedules planned for the next releases.

[girish]

We are trying to add per-app backup schedule and format for coming release.

[True]

I have the same issue, so let me provide a use case why moving the app's data directory is not such a great usecase. Here is a story to illustrate:

• You are an event photographer. Let's assume you are publishing pictures en-masse using an image editing software like Lightroom
• Lightroom workflow allows pictures to be exported on a NFS/CIFS/SMB share
• You attach the NFS/CIFS/SMB share to the app as a mount
• Immich picks up the images just fine from the external directory (mount)
• The RAW images (masters) are stored separately on a PC or a NAS. They don't need to be on Immich. They are the backup or backed up elsewhere essentially
• You are collecting pictures from other participants from an event and since they are likely not advanced users they will use the upload button
• The upload button uploads images to the app's storage space rather than the mount
• You want import newly uploaded pictures from other people to your Lightroom database and the best method is via a separate NFS/CIFS/SMB share for uploads
• Moving the user uploaded pictures from the app's storage to the mount is not possible from within the app
• Moving the user uploaded pictures from the app's Cloudron file manager to the mount is not practical. Also it might break things.

My point:

• You can't publish pictures and import new pictures easily directly from&to lightroom if everything is in the app's native storage.
• To upload you need to use the upload button or API.
• If you were to choose the launch Immich's API from within Lightroom - after export you can launch a custom script under the "post processing" section, but good luck passing the correct parameters. It will be a nightmare to manage and update.
• Using nextcloud is cumbersome for most people. They need simplicity. A dedicated solution. Also Immich has better sharing features and presentation.

Complications aside, since the metadata could be baked into the pictures via lightroom or phone the Immich Metadata DB is not that precious. It's just an outlet. Exposing the .env could cause severe complications, but the risk is mitigated by other methods already. There should be an option for this with the neccessary disclaimers. Image Pros would appreciate.

[nebulon]

Not sure if immich is the right tool for this then. Anyways though, it only has one data root, so if this is some mounted (network) disk, the upload button would also upload there.

All in all though, Cloudron is designed to back up apps as whole, random things may happen if database and assets on disk go out of sync due to a restore. Not sure if we can realistically support such situations, as we have to find a line between the various use-cases and thus have to make some opinionated decisions.

[shrey]

What i've ended up implementing so far:

• move immich data to an external (mounted) volume
• disable automatic backups
• maintain automatic snapshotting of the volume, externally

Just need to figure out how to maintain backups of the db, externally.
And then, hope that it all works out if the need to restore everything arises.

[nebulon]

I would not recommend implementing a custom backup scheme, easy to miss things. If you use some sshfs rsync backup storage with hardlinks, the backups would be quite efficient. For extra safety the snapshotting of course helps, but anything custom will cause more trouble if things go wrong in the future.