Celery stuck in restarting loop
-
Hey everyone,
I have had paperless running on Cloudron for several months without a Problem.
Now I realized that the E-Mail importer is stuck. When I tried to find out why I saw that celery is stuck in a restart loop.
I restarted the app and also the server.
When I do a clean reinstallation on my server the error does not show up. So I could recover from a backup. But if possible I would be able to solve the problem under the hood.Anyone an Idea what might be going on?
Thank you for your help!
2024-09-02T11:09:52.000Z __ - ... __ - _ 2024-09-02T11:09:52.000Z _dbm.error: cannot add item to database 2024-09-02T11:09:52.000Z celery beat v5.4.0 (opalescent) is starting. 2024-09-02T11:09:52.000Z f = BytesIO(self.dict[key.encode(self.keyencoding)]) 2024-09-02T11:09:52.000Z humanize_seconds(self.scheduler.max_interval)) 2024-09-02T11:09:52.000Z return self.get_scheduler() 2024-09-02T11:09:52.000Z return super().__get__(instance, owner) 2024-09-02T11:09:52.000Z return symbol_by_name(self.scheduler_cls, aliases=aliases)( 2024-09-02T11:09:52.000Z self._create_schedule() 2024-09-02T11:09:52.000Z self._store['entries'] 2024-09-02T11:09:52.000Z self._store['entries'] = {} 2024-09-02T11:09:52.000Z self.dict[key.encode(self.keyencoding)] = f.getvalue() 2024-09-02T11:09:52.000Z self.setup_schedule() 2024-09-02T11:09:52.000Z service.start() 2024-09-02T11:09:52.000Z super().__init__(*args, **kwargs) 2024-09-02T11:09:52.000Z val = self.func(instance) 2024-09-02T11:09:52.000Z value = self.cache[key] 2024-09-02T11:09:53.000Z 2024-09-02 11:09:53,996 INFO exited: paperless-scheduler.service (exit status 1; not expected) 2024-09-02T11:09:55.000Z 2024-09-02 11:09:54,999 INFO spawned: 'paperless-scheduler.service' with pid 620 2024-09-02T11:09:56.000Z 2024-09-02 11:09:56,001 INFO success: paperless-scheduler.service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2024-09-02T11:09:57.000Z 2024-09-02T11:09:57.000Z 2024-09-02T11:09:57.000Z 2024-09-02T11:09:57.000Z 2024-09-02T11:09:57.000Z . broker -> redis://redis-8c6815a8-fe2f-4f22-bcdf-fd9c324f5b7c:6379// 2024-09-02T11:09:57.000Z . db -> /app/data/data/celerybeat-schedule.db 2024-09-02T11:09:57.000Z . loader -> celery.loaders.app.AppLoader 2024-09-02T11:09:57.000Z . logfile -> [stderr]@%INFO 2024-09-02T11:09:57.000Z . maxinterval -> 5.00 minutes (300s) 2024-09-02T11:09:57.000Z . scheduler -> celery.beat.PersistentScheduler 2024-09-02T11:09:57.000Z Configuration -> 2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred: 2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred: 2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred: 2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred: 2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/functools.py", line 981, in __get__ 2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/shelve.py", line 111, in __getitem__ 2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/shelve.py", line 113, in __getitem__ 2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/shelve.py", line 125, in __setitem__ 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/apps/beat.py", line 113, in start_scheduler 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 264, in __init__ 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 513, in __init__ 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 541, in setup_schedule 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 570, in _create_schedule 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 574, in _create_schedule 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 634, in start 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 668, in get_scheduler 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 677, in scheduler 2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/kombu/utils/objects.py", line 40, in __get__ 2024-09-02T11:09:57.000Z KeyError: 'entries' 2024-09-02T11:09:57.000Z KeyError: b'entries' 2024-09-02T11:09:57.000Z LocalTime -> 2024-09-02 11:09:57 2024-09-02T11:09:57.000Z Traceback (most recent call last): 2024-09-02T11:09:57.000Z Traceback (most recent call last): 2024-09-02T11:09:57.000Z Traceback (most recent call last): 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,474] [INFO] [celery.beat] beat: Starting... 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,485] [CRITICAL] [celery.beat] beat raised exception <class '_dbm.error'>: error('cannot add item to database') 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,486] [WARNING] [celery.redirected] Traceback (most recent call last): 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 111, in __getitem__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] : 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] KeyError 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] value = self.cache[key] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 570, in _create_schedule 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] 'entries' 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] Traceback (most recent call last): 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] self._store['entries'] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 113, in __getitem__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected] KeyError 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected] f = BytesIO(self.dict[key.encode(self.keyencoding)]) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] File "/usr/local/bin/celery", line 8, in <module> 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] : 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] Traceback (most recent call last): 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] b'entries' 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/__main__.py", line 15, in main 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/bin/celery.py", line 236, in main 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] sys.exit(_main()) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] sys.exit(main()) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] return celery(auto_envvar_prefix="CELERY") 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] return self.main(*args, **kwargs) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,493] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,493] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,493] [WARNING] [celery.redirected] rv = self.invoke(ctx) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1688, in invoke 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected] return _process_result(sub_ctx.command.invoke(sub_ctx)) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] return __callback(*args, **kwargs) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] return ctx.invoke(self.callback, **ctx.params) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/bin/base.py", line 135, in caller 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] return f(ctx, *args, **kwargs) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] return f(get_current_context(), *args, **kwargs) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/apps/beat.py", line 84, in run 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/bin/beat.py", line 72, in beat 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] return beat().run() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] self.start_scheduler() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/apps/beat.py", line 113, in start_scheduler 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 634, in start 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected] service.start() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,499] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,499] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/kombu/utils/objects.py", line 40, in __get__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,499] [WARNING] [celery.redirected] humanize_seconds(self.scheduler.max_interval)) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected] File "/usr/lib/python3.10/functools.py", line 981, in __get__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected] return super().__get__(instance, owner) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected] val = self.func(instance) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 668, in get_scheduler 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 677, in scheduler 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected] return self.get_scheduler() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 513, in __init__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] return symbol_by_name(self.scheduler_cls, aliases=aliases)( 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] super().__init__(*args, **kwargs) 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 264, in __init__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 541, in setup_schedule 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] self._create_schedule() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] self.setup_schedule() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 125, in __setitem__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 574, in _create_schedule 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] self._store['entries'] = {} 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] self.dict[key.encode(self.keyencoding)] = f.getvalue() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] . 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] : 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] _dbm 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] cannot add item to database 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] error 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,684] [WARNING] [celery.redirected] Exception ignored in: 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 162, in __del__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected] <function Shelf.__del__ at 0x7f94faea8f70> 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected] Traceback (most recent call last): 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 144, in close 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 168, in sync 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] self.close() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] self.sync() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] File "/usr/lib/python3.10/shelve.py", line 125, in __setitem__ 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] _dbm 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] self.dict[key.encode(self.keyencoding)] = f.getvalue() 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] self[key] = entry 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] . 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] : 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] cannot add item to database 2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] error -
N nebulon locked this topic on
-
N nebulon referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login