Celery stuck in restarting loop

wheez

Hey everyone,

I have had paperless running on Cloudron for several months without a Problem.
Now I realized that the E-Mail importer is stuck. When I tried to find out why I saw that celery is stuck in a restart loop.
I restarted the app and also the server.
When I do a clean reinstallation on my server the error does not show up. So I could recover from a backup. But if possible I would be able to solve the problem under the hood.

Anyone an Idea what might be going on?

Thank you for your help!

2024-09-02T11:09:52.000Z __    -    ... __   -        _
2024-09-02T11:09:52.000Z _dbm.error: cannot add item to database
2024-09-02T11:09:52.000Z celery beat v5.4.0 (opalescent) is starting.
2024-09-02T11:09:52.000Z f = BytesIO(self.dict[key.encode(self.keyencoding)])
2024-09-02T11:09:52.000Z humanize_seconds(self.scheduler.max_interval))
2024-09-02T11:09:52.000Z return self.get_scheduler()
2024-09-02T11:09:52.000Z return super().__get__(instance, owner)
2024-09-02T11:09:52.000Z return symbol_by_name(self.scheduler_cls, aliases=aliases)(
2024-09-02T11:09:52.000Z self._create_schedule()
2024-09-02T11:09:52.000Z self._store['entries']
2024-09-02T11:09:52.000Z self._store['entries'] = {}
2024-09-02T11:09:52.000Z self.dict[key.encode(self.keyencoding)] = f.getvalue()
2024-09-02T11:09:52.000Z self.setup_schedule()
2024-09-02T11:09:52.000Z service.start()
2024-09-02T11:09:52.000Z super().__init__(*args, **kwargs)
2024-09-02T11:09:52.000Z val = self.func(instance)
2024-09-02T11:09:52.000Z value = self.cache[key]
2024-09-02T11:09:53.000Z 2024-09-02 11:09:53,996 INFO exited: paperless-scheduler.service (exit status 1; not expected)
2024-09-02T11:09:55.000Z 2024-09-02 11:09:54,999 INFO spawned: 'paperless-scheduler.service' with pid 620
2024-09-02T11:09:56.000Z 2024-09-02 11:09:56,001 INFO success: paperless-scheduler.service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-02T11:09:57.000Z 
2024-09-02T11:09:57.000Z 
2024-09-02T11:09:57.000Z 
2024-09-02T11:09:57.000Z 
2024-09-02T11:09:57.000Z . broker -> redis://redis-8c6815a8-fe2f-4f22-bcdf-fd9c324f5b7c:6379//
2024-09-02T11:09:57.000Z . db -> /app/data/data/celerybeat-schedule.db
2024-09-02T11:09:57.000Z . loader -> celery.loaders.app.AppLoader
2024-09-02T11:09:57.000Z . logfile -> [stderr]@%INFO
2024-09-02T11:09:57.000Z . maxinterval -> 5.00 minutes (300s)
2024-09-02T11:09:57.000Z . scheduler -> celery.beat.PersistentScheduler
2024-09-02T11:09:57.000Z Configuration ->
2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred:
2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred:
2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred:
2024-09-02T11:09:57.000Z During handling of the above exception, another exception occurred:
2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/functools.py", line 981, in __get__
2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/shelve.py", line 111, in __getitem__
2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/shelve.py", line 113, in __getitem__
2024-09-02T11:09:57.000Z File "/usr/lib/python3.10/shelve.py", line 125, in __setitem__
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/apps/beat.py", line 113, in start_scheduler
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 264, in __init__
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 513, in __init__
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 541, in setup_schedule
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 570, in _create_schedule
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 574, in _create_schedule
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 634, in start
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 668, in get_scheduler
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 677, in scheduler
2024-09-02T11:09:57.000Z File "/usr/local/lib/python3.10/dist-packages/kombu/utils/objects.py", line 40, in __get__
2024-09-02T11:09:57.000Z KeyError: 'entries'
2024-09-02T11:09:57.000Z KeyError: b'entries'
2024-09-02T11:09:57.000Z LocalTime -> 2024-09-02 11:09:57
2024-09-02T11:09:57.000Z Traceback (most recent call last):
2024-09-02T11:09:57.000Z Traceback (most recent call last):
2024-09-02T11:09:57.000Z Traceback (most recent call last):
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,474] [INFO] [celery.beat] beat: Starting...
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,485] [CRITICAL] [celery.beat] beat raised exception <class '_dbm.error'>: error('cannot add item to database')
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,486] [WARNING] [celery.redirected] Traceback (most recent call last):
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 111, in __getitem__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] :
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] KeyError
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,487] [WARNING] [celery.redirected] value = self.cache[key]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 570, in _create_schedule
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] 'entries'
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] Traceback (most recent call last):
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,488] [WARNING] [celery.redirected] self._store['entries']
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 113, in __getitem__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected] KeyError
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,489] [WARNING] [celery.redirected] f = BytesIO(self.dict[key.encode(self.keyencoding)])
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected]   File "/usr/local/bin/celery", line 8, in <module>
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] :
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] Traceback (most recent call last):
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,490] [WARNING] [celery.redirected] b'entries'
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/__main__.py", line 15, in main
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/bin/celery.py", line 236, in main
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] sys.exit(_main())
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,491] [WARNING] [celery.redirected] sys.exit(main())
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] return celery(auto_envvar_prefix="CELERY")
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,492] [WARNING] [celery.redirected] return self.main(*args, **kwargs)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,493] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,493] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,493] [WARNING] [celery.redirected] rv = self.invoke(ctx)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1688, in invoke
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,494] [WARNING] [celery.redirected] return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] return __callback(*args, **kwargs)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,495] [WARNING] [celery.redirected] return ctx.invoke(self.callback, **ctx.params)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/bin/base.py", line 135, in caller
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] return f(ctx, *args, **kwargs)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,496] [WARNING] [celery.redirected] return f(get_current_context(), *args, **kwargs)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/apps/beat.py", line 84, in run
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/bin/beat.py", line 72, in beat
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] return beat().run()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,497] [WARNING] [celery.redirected] self.start_scheduler()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/apps/beat.py", line 113, in start_scheduler
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 634, in start
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,498] [WARNING] [celery.redirected] service.start()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,499] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,499] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/kombu/utils/objects.py", line 40, in __get__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,499] [WARNING] [celery.redirected] humanize_seconds(self.scheduler.max_interval))
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/functools.py", line 981, in __get__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected] return super().__get__(instance, owner)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,500] [WARNING] [celery.redirected] val = self.func(instance)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 668, in get_scheduler
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 677, in scheduler
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,501] [WARNING] [celery.redirected] return self.get_scheduler()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 513, in __init__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] return symbol_by_name(self.scheduler_cls, aliases=aliases)(
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,502] [WARNING] [celery.redirected] super().__init__(*args, **kwargs)
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 264, in __init__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 541, in setup_schedule
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] self._create_schedule()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,503] [WARNING] [celery.redirected] self.setup_schedule()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 125, in __setitem__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected]   File "/usr/local/lib/python3.10/dist-packages/celery/beat.py", line 574, in _create_schedule
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] self._store['entries'] = {}
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,504] [WARNING] [celery.redirected] self.dict[key.encode(self.keyencoding)] = f.getvalue()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] .
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] :
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] _dbm
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] cannot add item to database
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,505] [WARNING] [celery.redirected] error
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,684] [WARNING] [celery.redirected] Exception ignored in:
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 162, in __del__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected] <function Shelf.__del__ at 0x7f94faea8f70>
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,685] [WARNING] [celery.redirected] Traceback (most recent call last):
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 144, in close
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 168, in sync
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] self.close()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,686] [WARNING] [celery.redirected] self.sync()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected]
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected]   File "/usr/lib/python3.10/shelve.py", line 125, in __setitem__
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] _dbm
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] self.dict[key.encode(self.keyencoding)] = f.getvalue()
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,687] [WARNING] [celery.redirected] self[key] = entry
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] .
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] :
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] cannot add item to database
2024-09-02T11:09:57.000Z [2024-09-02 11:09:57,688] [WARNING] [celery.redirected] error

nebulon

This is a duplicate of https://forum.cloudron.io/topic/12195/high-cpu-utilization-since-update-1-25-1/18 will close this one here

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Celery stuck in restarting loop