ModuleNotFoundError: No module named 'onnxruntime'

[lukas]

Just checked the LOG file and found this error(s), which are repeating very often:

Feb 02 22:01:44 ModuleNotFoundError: No module named 'onnxruntime'
Feb 02 22:01:44[02/02/24 21:01:44] INFO Worker exiting (pid: 668099)
Feb 02 22:01:44[02/02/24 21:01:44] ERROR Worker (pid:668099) exited with code 3
Feb 02 22:01:44[02/02/24 21:01:44] ERROR Shutting down: Master
Feb 02 22:01:44[02/02/24 21:01:44] ERROR Reason: Worker failed to boot.
Feb 02 22:01:452024-02-02 21:01:45,965 INFO exited: machine-learning (exit status 3; not expected)
Feb 02 22:01:462024-02-02 21:01:46,968 INFO spawned: 'machine-learning' with pid 668100
Feb 02 22:01:472024-02-02 21:01:47,970 INFO success: machine-learning entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 02 22:01:48[02/02/24 21:01:48] INFO Starting gunicorn 21.2.0
Feb 02 22:01:48[02/02/24 21:01:48] INFO Listening at: http://127.0.0.1:3003 (668100)
Feb 02 22:01:48[02/02/24 21:01:48] INFO Using worker: uvicorn.workers.UvicornWorker
Feb 02 22:01:48[02/02/24 21:01:48] INFO Booting worker with pid: 668104
Feb 02 22:01:48[02/02/24 21:01:48] ERROR Exception in worker process
Feb 02 22:01:48 ╭─────── Traceback (most recent call last) ───────╮
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/arbiter.py:609 in spawn_worker │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 606 │ │ │ util._setproctitle("worker [% │
Feb 02 22:01:48 │ 607 │ │ │ self.log.info("Booting worker │
Feb 02 22:01:48 │ 608 │ │ │ self.cfg.post_fork(self, work │
Feb 02 22:01:48 │ ❱ 609 │ │ │ worker.init_process() │
Feb 02 22:01:48 │ 610 │ │ │ sys.exit(0) │
Feb 02 22:01:48 │ 611 │ │ except SystemExit: │
Feb 02 22:01:48 │ 612 │ │ │ raise │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/workers/base.py:134 in init_process │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 131 │ │ │ self.reloader = reloader_cls( │
Feb 02 22:01:48 │ 132 │ │ │ │ │ │ │ │ │ │ │
Feb 02 22:01:48 │ 133 │ │ │
Feb 02 22:01:48 │ ❱ 134 │ │ self.load_wsgi() │
Feb 02 22:01:48 │ 135 │ │ if self.reloader: │
Feb 02 22:01:48 │ 136 │ │ │ self.reloader.start() │
Feb 02 22:01:48 │ 137 │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/workers/base.py:146 in load_wsgi │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 143 │ │
Feb 02 22:01:48 │ 144 │ def load_wsgi(self): │
Feb 02 22:01:48 │ 145 │ │ try: │
Feb 02 22:01:48 │ ❱ 146 │ │ │ self.wsgi = self.app.wsgi() │
Feb 02 22:01:48 │ 147 │ │ except SyntaxError as e: │
Feb 02 22:01:48 │ 148 │ │ │ if not self.cfg.reload: │
Feb 02 22:01:48 │ 149 │ │ │ │ raise │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/app/base.py:67 in wsgi │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 64 │ │
Feb 02 22:01:48 │ 65 │ def wsgi(self): │
Feb 02 22:01:48 │ 66 │ │ if self.callable is None: │
Feb 02 22:01:48 │ ❱ 67 │ │ │ self.callable = self.load() │
Feb 02 22:01:48 │ 68 │ │ return self.callable │
Feb 02 22:01:48 │ 69 │ │
Feb 02 22:01:48 │ 70 │ def run(self): │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/app/wsgiapp.py:58 in load │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 55 │ │ if self.cfg.paste is not None: │
Feb 02 22:01:48 │ 56 │ │ │ return self.load_pasteapp() │
Feb 02 22:01:48 │ 57 │ │ else: │
Feb 02 22:01:48 │ ❱ 58 │ │ │ return self.load_wsgiapp() │
Feb 02 22:01:48 │ 59 │
Feb 02 22:01:48 │ 60 │
Feb 02 22:01:48 │ 61 def run(): │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/app/wsgiapp.py:48 in load_wsgiapp │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 45 │ │ │ │ raise ConfigError("No appl │
Feb 02 22:01:48 │ 46 │ │
Feb 02 22:01:48 │ 47 │ def load_wsgiapp(self): │
Feb 02 22:01:48 │ ❱ 48 │ │ return util.import_app(self.app_ur │
Feb 02 22:01:48 │ 49 │ │
Feb 02 22:01:48 │ 50 │ def load_pasteapp(self): │
Feb 02 22:01:48 │ 51 │ │ from .pasterapp import get_wsgi_ap │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /app/code/venv/lib/python3.11/site-packages/gun │
Feb 02 22:01:48 │ icorn/util.py:371 in import_app │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 368 │ │ module, obj = parts[0], parts[1] │
Feb 02 22:01:48 │ 369 │ │
Feb 02 22:01:48 │ 370 │ try: │
Feb 02 22:01:48 │ ❱ 371 │ │ mod = importlib.import_module(mod │
Feb 02 22:01:48 │ 372 │ except ImportError: │
Feb 02 22:01:48 │ 373 │ │ if module.endswith(".py") and os. │
Feb 02 22:01:48 │ 374 │ │ │ msg = "Failed to find applica │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /usr/lib/python3.11/importlib/__init__.py:126 │
Feb 02 22:01:48 │ in import_module │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 123 │ │ │ if character != '.': │
Feb 02 22:01:48 │ 124 │ │ │ │ break │
Feb 02 22:01:48 │ 125 │ │ │ level += 1 │
Feb 02 22:01:48 │ ❱ 126 │ return _bootstrap._gcd_import(name[le │
Feb 02 22:01:48 │ 127 │
Feb 02 22:01:48 │ 128 │
Feb 02 22:01:48 │ 129 _RELOADING = {} │
Feb 02 22:01:48 │ in _gcd_import:1204 │
Feb 02 22:01:48 │ in _find_and_load:1176 │
Feb 02 22:01:48 │ in _find_and_load_unlocked:1147 │
Feb 02 22:01:48 │ in _load_unlocked:690 │
Feb 02 22:01:48 │ in exec_module:940 │
Feb 02 22:01:48 │ in _call_with_frames_removed:241 │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ /usr/src/app/machine-learning/app/main.py:15 in │
Feb 02 22:01:48 │ <module> │
Feb 02 22:01:48 │ │
Feb 02 22:01:48 │ 12 import orjson │
Feb 02 22:01:48 │ 13 from fastapi import Depends, FastAPI, For │
Feb 02 22:01:48 │ 14 from fastapi.responses import ORJSONRespo │
Feb 02 22:01:48 │ ❱ 15 from onnxruntime.capi.onnxruntime_pybind1 │
Feb 02 22:01:48 │ 16 from starlette.formparsers import MultiPa │
Feb 02 22:01:48 │ 17 │
Feb 02 22:01:48 │ 18 from app.models.base import InferenceMode │
Feb 02 22:01:48 ╰─────────────────────────────────────────────────╯

Regards
Lukas

[archos]

I have the same problem. Image thumbnails also display badly.

[msbt]

same here, restart loop which results in ~100% cpu usage

[nebulon]

I can reproduce the issue about ModuleNotFoundError: No module named 'onnxruntime' still investigating why that is not found/installed via poetry like all the other modules.

[nebulon]

This is fixed with latest package now.

[archos]

@nebulon I updated the app, but unfortunately the image preview is still broken.

Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [ExceptionsHandler] pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.
Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [ExceptionsHandler] pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.
Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [ExceptionsHandler] pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.
Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [ExceptionsHandler] pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.
Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [QueryFailedError: pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.
Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [QueryFailedError: pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.
Feb 04 20:42:44[Nest] 52 - 02/04/2024, 7:42:44 PM ERROR [QueryFailedError: pgvecto.rs: pgvecto.rs must be loaded via shared_preload_libraries.

at async AssetController.getAssetThumbnail (/usr/src/app/dist/immich/api-v1/asset/asset.controller.js:62:9)] Unable to send file: QueryFailedError
Feb 04 20:47:23at async AssetController.getAssetThumbnail (/usr/src/app/dist/immich/api-v1/asset/asset.controller.js:62:9)] Unable to send file: QueryFailedError
Feb 04 20:47:23at async AssetController.getAssetThumbnail (/usr/src/app/dist/immich/api-v1/asset/asset.controller.js:62:9)] Unable to send file: QueryFailedError
Feb 04 20:47:23at async AssetController.getAssetThumbnail (/usr/src/app/dist/immich/api-v1/asset/asset.controller.js:62:9)] Unable to send file: QueryFailedError

[girish]

@archos maybe related to https://docs.cloudron.io/apps/immich/#pgvectors ?

[archos]

@girish yes it's fine now, thank you.