Calibre not able to write on mounted (cifs) volume

[odie]

@girish Tried NFS - mounted, and read+write permissions work well from file browser.

WHen selecting this mount as a volume in Calibre, I get the message "db not writeable" upon clicking "save", and when I go to the main library window, the entire app crashes with a "Calibre-Web 500 Internal Server Error" message.

In the logs, these messages appear:

Sep 11 08:39:41(Background on this error at: https://sqlalche.me/e/14/e3q8)
Sep 11 08:39:41[2023-09-11 06:39:41,825] WARN {cps.config_sql:385} invalidating configuration
Sep 11 08:39:41[2023-09-11 06:39:41,825] DEBUG {cps.config_sql:373} _ConfigSQL updating storage
Sep 11 08:39:41[2023-09-11 06:39:41,825] ERROR {cps.config_sql:384} (sqlite3.OperationalError) unable to open database: /media/CalibreLibNFS/metadata.db
Sep 11 08:39:41[2023-09-11 06:39:41,840] DEBUG {cps.config_sql:373} _ConfigSQL updating storage
Sep 11 08:39:41[SQL: attach database '/media/CalibreLibNFS/metadata.db' as calibre;]
Sep 11 08:39:45AttributeError: 'NoneType' object has no attribute 'query'
Sep 11 08:39:45File "/app/code/calibre-web/cps/db.py", line 743, in generate_linked_query
Sep 11 08:39:45File "/app/code/calibre-web/cps/db.py", line 782, in fill_indexpage
Sep 11 08:39:45File "/app/code/calibre-web/cps/db.py", line 789, in fill_indexpage_with_archived_books
Sep 11 08:39:45File "/app/code/calibre-web/cps/usermanagement.py", line 35, in decorated_view
Sep 11 08:39:45File "/app/code/calibre-web/cps/web.py", line 408, in render_books_list
Sep 11 08:39:45File "/app/code/calibre-web/cps/web.py", line 790, in index
Sep 11 08:39:45File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1503, in dispatch_request
Sep 11 08:39:45File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1517, in full_dispatch_request
Sep 11 08:39:45File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1519, in full_dispatch_request
Sep 11 08:39:45File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2073, in wsgi_app
Sep 11 08:39:45File "/usr/local/lib/python3.10/dist-packages/flask_login/utils.py", line 290, in decorated_view
Sep 11 08:39:45Traceback (most recent call last):
Sep 11 08:39:45[2023-09-11 06:39:45,879] ERROR {cps:1449} Exception on / [GET]
Sep 11 08:39:45entries, random, pagination = calibre_db.fill_indexpage(page, 0, db.Books, True, order[0],
Sep 11 08:39:45query = (self.session.query(database, ub.ArchivedBook.is_archived, ub.ReadBook.read_status)
Sep 11 08:39:45random_query = self.generate_linked_query(config_read_column, database)
Sep 11 08:39:45response = self.full_dispatch_request()
Sep 11 08:39:45return current_app.ensure_sync(func)(*args, **kwargs)
Sep 11 08:39:45return login_required(func)(*args, **kwargs)
Sep 11 08:39:45return render_books_list("newest", sort_param, 1, page)
Sep 11 08:39:45return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
Sep 11 08:39:45return self.fill_indexpage_with_archived_books(page, database, pagesize, db_filter, order, False,
Sep 11 08:39:45rv = self.dispatch_request()
Sep 11 08:39:45rv = self.handle_user_exception(e)

Also, I have these entries in the log - I am on the latest installable version from the app store:

[2023-09-11 06:39:29,428] DEBUG {cps.updater:329} Stable version: {'version': '0.6.20'}

[girish]

@odie said in New: Not able to write on mounted volume:

Sep 11 08:39:41[2023-09-11 06:39:41,825] ERROR {cps.config_sql:384} (sqlite3.OperationalError) unable to open database: /media/CalibreLibNFS/metadata.db

I think this error is different from the previous one. Does the above file exist / have right permissions ?

[odie]

@girish said in New: Not able to write on mounted volume:

@odie said in New: Not able to write on mounted volume:

Sep 11 08:39:41[2023-09-11 06:39:41,825] ERROR {cps.config_sql:384} (sqlite3.OperationalError) unable to open database: /media/CalibreLibNFS/metadata.db

I think this error is different from the previous one. Does the above file exist / have right permissions ?

It is the same file on the remote share (it's just an NFS mount). It has the same permissions. It has these permissions:

-rw-rw-rw-

Incidentally, the problem description suggestion you make in post #2 is perhaps accurate. I notice that the files are uploaded to the mounted folder when adding via calibre-web interface. However, I cannot change/edit metadata (when mounting as CIFS). Like I said in post #4, when the folder is mounted as an NFS share, I get the Error 500.

Also, like I said in my initial post - when the calibre app is restarted (via the cloudron dashboard), the database location is not stored persistent. This happens with both NFS and CIFS mounted volumes.

[girish]

@odie OK, I understand the situation a bit better now. There are 2 problems:

• Moving the library does not work. The package keeps resetting the library location. I am working on fix this.
• Database cannot be moved. Do you mean calibre's own database here (app.db) or do you mean the library database? If it's the latter, I am working on fixing that above. If it's the former, can you clarify why you want to move this? It's just a very small sqlite database.

[girish]

@odie Can you please check with the latest package? The library can now be moved wherever you want (of course, not to a cifs volume).

[girish]

I put some docs here - https://docs.cloudron.io/apps/calibre-web/#library

[odie]

@girish said in Calibre not able to write on mounted (cifs) volume:

@odie Can you please check with the latest package? The library can now be moved wherever you want (of course, not to a cifs volume).

Hello @girish,
I checked with the new image, and database location is now changed persistently. It remembers the location across restarts and across the upgrade.

I have a two questions which I would love to get your advice on (related to the fact that cifs volumes don’t handle the permissions). I really would like to keep the database on my cifs storage, so that the client computers with calibre desktop can still use the database.

1. Where on my linux server would you recommend me setting up a clean database (path)? My Cloudron runs on a single purpose pc (only Cloudron installed). Where could this folder be placed?

2. How can I setup an automatic synchronization between this location and my CIFS storage, ignoring permissions but keeping a two way sync going? It doesn’t have to be real time, but it needs to be automatic.

Thanks!

[girish]

@odie said in Calibre not able to write on mounted (cifs) volume:

I really would like to keep the database on my cifs storage, so that the client computers with calibre desktop can still use the database.

Is this because you are on Windows ?

The upstream project does not support having a library in CIFS. I think you will have ask upstream for a workaround and then we can try to see how we can set things up on Cloudron side.

I also don't know how things will work out if you just rsync a calibre library between computers (given that this won't be atomic?). I don't have that much experience with calibre database internals.

[odie]

@girish said in Calibre not able to write on mounted (cifs) volume:

@odie said in Calibre not able to write on mounted (cifs) volume:

I really would like to keep the database on my cifs storage, so that the client computers with calibre desktop can still use the database.

Is this because you are on Windows ?

The upstream project does not support having a library in CIFS. I think you will have ask upstream for a workaround and then we can try to see how we can set things up on Cloudron side.

I also don't know how things will work out if you just rsync a calibre library between computers (given that this won't be atomic?). I don't have that much experience with calibre database internals.

I have a couple of devices running Windows and Mac, that all use the Calibre Desktop (I work from different places, so it's easier that way). They all have a local Calibre Libray, which is then synced to this network share that I also access from the Cloudron Calibre Web app. This strategy enables me to keep all Calibre Libraries in sync - I can make changes on one computer, and these sync to different computers via the cifs share. I was hoping that this strategy would also work on Calibre Web as well. It would be useful to be able to do quick edits (e.g. metadata edits), or the occasional upload also on web.

[girish]

@odie how do you sync between the computers? rsync with a cron job?

[odie]

@girish said in Calibre not able to write on mounted (cifs) volume:

@odie how do you sync between the computers? rsync with a cron job?

I probably should have used rsync, but I've always found it easier with point&click.... On my Macs, I use FreeFileSync (https://freefilesync.org/) and on Windows, I use SyncBackFree (https://www.2brightsparks.com/). It is sufficient for me to syn every 4 hours, so I use scheduled syncs.

[girish]

@odie so, just to take a step back. If we ignore all this CIFS stuff ...

On Cloudron the library is at /app/data/library by default. This maps to the host path at /home/yellowtent/appsdata/<appid>/data/library (might want to double check this path). Is that path not enough to sync via ssh?

[odie]

@girish said in Calibre not able to write on mounted (cifs) volume:

@odie so, just to take a step back. If we ignore all this CIFS stuff ...

On Cloudron the library is at /app/data/library by default. This maps to the host path at /home/yellowtent/appsdata/<appid>/data/library (might want to double check this path). Is that path not enough to sync via ssh?

Thanks. I've been moderately successful with this.

I can sync using rsync and the -rz (and rzv, rzP) flags.

The syntax I use is:

rsync -rz /CIFSshare/ /home/yellowtent/appsdata/<appid>/data/library

This way, I manage to sync from the CIFS mount to the /home/yellowtent/appsdata/<appid>/data/library folder. Calibre-web now runs with a writeable database.

Two minor problems with this:

1. using the -r option, everything syncs on every subsequent execution of the command (ie all files are copied from the CIFSshare)
2. using the -a option instead of -r, the database would be writeable on first sync. But using the -a option resulted in a locked db on second, third etc. run.

The bigger problem is this:
When I use rsync, changes I make on the Cloudron/calibre-web app are not synced back to the NAS folder. So basically, this leaves me to lose edits made via Calibre-web on Cloudron every time I update the library from one of the clients (since local changes on Cloudron calibre-web are not synced to the CIFS share). Relying on a manual sync after every library update on Calibre-web is next to impossible (I often read on my mobile devices, and I like to edit metadata while reading).

So basically, it almost appears safer to keep the library on a read-only CIFSshare...