Storage management in Immich?

[shrey]

@nebulon said in Storage management in Immich?:

After rereading the thread, you want metadata/database to be backed up but not the actual pictures?

Pretty much. The media files can be stored in a mounted volume.

But currently, if i store everything in the Data Directory (that gets backed up, when enabled globally), then i'm at a big loss, because, i can neither set a separate (more lenient than other apps) backup frequency, nor can i escape the pain of having to manage my backup storages efficiently (i realised today that immich's backups were enabled in my case, which is 4 times a day, for a month, and around 150GB every time!!).

[nebulon]

In such cases the rsync backup with hardlinks, will not duplicate any data even if many backups are created. So may be worth moving to such a storage in your case.

Further we have per-app backup schedules planned for the next releases.

[girish]

We are trying to add per-app backup schedule and format for coming release.

[True]

I have the same issue, so let me provide a use case why moving the app's data directory is not such a great usecase. Here is a story to illustrate:

• You are an event photographer. Let's assume you are publishing pictures en-masse using an image editing software like Lightroom
• Lightroom workflow allows pictures to be exported on a NFS/CIFS/SMB share
• You attach the NFS/CIFS/SMB share to the app as a mount
• Immich picks up the images just fine from the external directory (mount)
• The RAW images (masters) are stored separately on a PC or a NAS. They don't need to be on Immich. They are the backup or backed up elsewhere essentially
• You are collecting pictures from other participants from an event and since they are likely not advanced users they will use the upload button
• The upload button uploads images to the app's storage space rather than the mount
• You want import newly uploaded pictures from other people to your Lightroom database and the best method is via a separate NFS/CIFS/SMB share for uploads
• Moving the user uploaded pictures from the app's storage to the mount is not possible from within the app
• Moving the user uploaded pictures from the app's Cloudron file manager to the mount is not practical. Also it might break things.

My point:

• You can't publish pictures and import new pictures easily directly from&to lightroom if everything is in the app's native storage.
• To upload you need to use the upload button or API.
• If you were to choose the launch Immich's API from within Lightroom - after export you can launch a custom script under the "post processing" section, but good luck passing the correct parameters. It will be a nightmare to manage and update.
• Using nextcloud is cumbersome for most people. They need simplicity. A dedicated solution. Also Immich has better sharing features and presentation.

Complications aside, since the metadata could be baked into the pictures via lightroom or phone the Immich Metadata DB is not that precious. It's just an outlet. Exposing the .env could cause severe complications, but the risk is mitigated by other methods already. There should be an option for this with the neccessary disclaimers. Image Pros would appreciate.

[nebulon]

Not sure if immich is the right tool for this then. Anyways though, it only has one data root, so if this is some mounted (network) disk, the upload button would also upload there.

All in all though, Cloudron is designed to back up apps as whole, random things may happen if database and assets on disk go out of sync due to a restore. Not sure if we can realistically support such situations, as we have to find a line between the various use-cases and thus have to make some opinionated decisions.

[shrey]

What i've ended up implementing so far:

• move immich data to an external (mounted) volume
• disable automatic backups
• maintain automatic snapshotting of the volume, externally

Just need to figure out how to maintain backups of the db, externally.
And then, hope that it all works out if the need to restore everything arises.

[nebulon]

I would not recommend implementing a custom backup scheme, easy to miss things. If you use some sshfs rsync backup storage with hardlinks, the backups would be quite efficient. For extra safety the snapshotting of course helps, but anything custom will cause more trouble if things go wrong in the future.

[andreasdueren]

How are people managing this? I am interested in setting Immich up for something but that would require about 10TB of storage. Really only an option with Hetzner Storage Box. I don’t care too much about a backup of the application. And I can set up backups of the photo files from the storage box.

[jdaviescoates]

I don't use Immich but can you not just mount a Hetzner Storage Box as a Cloudron Volume and then mount that as app storage? Volumes don't get backed up. That's what I do with all my Music. The originals are on my laptop. They are synced to a Hetzner Storage Box setup as a Cloudron volume and then that volume is mounted into Navidrome and Jellyfin. Presumably something similar could be done with Immich?

[andreasdueren]

@jdaviescoates I'm not seeing how I can tell Immich to store the files on the external drive. For that I'd need modify the .env file.

[nebulon]

Maybe the external library feature would work with Immich as mentioned in https://forum.cloudron.io/topic/13587/mount-nextcloud-data-as-immich-external-library

[andreasdueren]

@nebulon I've looked into it but from what I understand these are read only from the APP. I'd like to use the app for uploading, not just viewing.

[perelin]

Same use case here: would like to manage my photos with Immich, but want/need to store them on a Hetzner Storage Box (or any other external storage). My library is just to big to put it on the actual Cloudron Server. How can this be achieved? Did not find any clear path to this. Storage Templates maybe?

[jdaviescoates]

Could we not just use the config settings shown here to point at mounted Cloudron Volume? https://immich.app/docs/guides/custom-locations/

[andreasdueren]

@jdaviescoates nope, since we don't have access to the dotenv file

[jdaviescoates]

@andreasdueren said in Storage management in Immich?:

@jdaviescoates nope, since we don't have access to the dotenv file

Sounds to my like @staff just need to add a symlinked .env file to the package that we can add config options too using the File Manager, just like many other packages already have?

[joseph]

I think the immich approach is to set thing in /app/data/immich.json

[perelin]

@joseph the immich.json does not expose a config setting for the file upload/storage path [1]. Looks like the way to go is the env var UPLOAD_LOCATION [2]

[1] https://immich.app/docs/install/config-file/
[2] https://immich.app/docs/install/environment-variables/#docker-compose

[nebulon]

So those env variables are for the immich docker compose environment, which Cloudron does not use. So they won't help us here. The config file does not have this, since it is not something which should be configured.

Instead like the docker compose setup, our package needs to be able to deal with paths accordingly, so we would have to essentially introduce our own "env" vars to define paths to be able to use Cloudron volume mounts. However given that those are not backed up in the same way main app storage is, a restore will inevitably cause issues with the files in storage getting out of sync with the app's database.

All in all I am really not sure if we should even lead users down that path with the risk involved for the benefit of cheaper storage

[perelin]

To understand the issue: when I restore a backup there might possibly be

a) new photos in the external store that are not in the restored db because they were added to Immich in inbetween backup and restore time

and/or

b) photos missing in the external store that are still in the restored db because they have been deleted from Immich inbetween backup and restore time

Both sound like states that Immich should address at some point in their ongoing development. Either re-index the storage path or clean db of file entries no longer available. But I have not checked if they have discussions ongoing about such features.

I generally would opt for letting users shoot themselves in the foot, given enough red, bold disclaimers