Metabase erroring after recent update to package version 1.21.6

[ChristopherMag]

Last night 4/16/2024 our Metabase instance updated to package version 1.21.6 and is now inaccessible.

The app is in a running state but when you try to access we see the standard cloudron error page of "This app is currently not responding. Please try refreshing the page in a few minutes."

Restarting the app causes the app to stay in a starting state for a while and eventually enters a running state but with the same error page displayed.

Looking at the logs the processes inside the app continually restart over and over and the part that appears to be erroring is a migration that it tries to execute but then fails:

Apr 17 11:40:11 2024-04-17 15:40:11,608 INFO db.liquibase :: Checking if Database has unrun migrations...
Apr 17 11:40:11 2024-04-17 15:40:11,830 INFO db.liquibase :: Database has unrun migrations. Checking if migraton lock is taken...
Apr 17 11:40:11 2024-04-17 15:40:11,835 INFO db.liquibase :: No migration lock found.
Apr 17 11:40:11 2024-04-17 15:40:11,994 INFO db.liquibase :: Running 1 migrations ...
Apr 17 11:40:12 2024-04-17 15:40:12,189 INFO driver.impl :: Initializing driver :mongo...
Apr 17 11:40:12 2024-04-17 15:40:12,190 INFO plugins.classloader :: Added URL file:/app/code/plugins/mongo.metabase-driver.jar to classpath
Apr 17 11:40:12 2024-04-17 15:40:12,191 DEBUG plugins.init-steps :: Loading plugin namespace metabase.driver.mongo...
Apr 17 11:40:12 2024-04-17 15:40:12,312 INFO driver.impl :: Registered driver :mongo 🚚
Apr 17 11:40:12 2024-04-17 15:40:12,318 INFO metabase.util :: ⮦ Load lazy loading driver :mongo took 127.7 ms
Apr 17 11:40:12 2024-04-17 15:40:12,337 INFO driver.impl :: Initializing driver :sql...
Apr 17 11:40:12 2024-04-17 15:40:12,338 INFO driver.impl :: Initializing driver :sql-jdbc...
Apr 17 11:40:12 2024-04-17 15:40:12,338 INFO driver.impl :: Initializing driver :csv...
Apr 17 11:40:12 2024-04-17 15:40:12,338 INFO plugins.classloader :: Added URL file:/app/code/plugins/csv.metabase-driver.jar to classpath
Apr 17 11:40:12 2024-04-17 15:40:12,338 DEBUG plugins.init-steps :: Loading plugin namespace metabase.driver.csv...
Apr 17 11:40:12 2024-04-17 15:40:12,346 ERROR liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-04-09T10:00:03::qnkhuat encountered an exception.
Apr 17 11:40:12 <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - UPDATE SUMMARY <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - Run: 1 <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - Previously run: 268 <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - Filtered out: 17 <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - -------------------------------
Apr 17 11:40:12 Total change sets: 286 <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - FILTERED CHANGE SETS SUMMARY <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - DBMS mismatch: 17 <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 -
Apr 17 11:40:12 2024-04-17 15:40:12,386 ERROR metabase.core :: Metabase Initialization FAILED <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - liquibase.exception.CommandExecutionException: java.lang.ClassCastException: class java.lang.ExceptionInInitializerError cannot be cast to class java.lang.Exception (java.lang.ExceptionInInitializerError and java.lang.Exception are in module java.base of loader 'bootstrap') <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.command.CommandScope.execute(CommandScope.java:253) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Liquibase.lambda$update$0(Liquibase.java:245) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Scope.lambda$child$0(Scope.java:186) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Scope.child(Scope.java:195) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Scope.child(Scope.java:185)
Apr 17 11:40:12 at liquibase.Scope.child(Scope.java:164) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Liquibase.runInScope(Liquibase.java:1419) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Liquibase.update(Liquibase.java:234) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Liquibase.update(Liquibase.java:212) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.Liquibase.update(Liquibase.java:194) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.liquibase$migrate_up_if_needed_BANG_.invokeStatic(liquibase.clj:305) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.liquibase$migrate_up_if_needed_BANG_.invoke(liquibase.clj:287) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$migrate_BANG_$fn__51104.invoke(setup.clj:80) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.liquibase$do_with_liquibase$f_STAR___48780.invoke(liquibase.clj:139) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.liquibase$do_with_liquibase.invokeStatic(liquibase.clj:142) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.liquibase$do_with_liquibase.invoke(liquibase.clj:130) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$migrate_BANG_.invokeStatic(setup.clj:75) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$migrate_BANG_.doInvoke(setup.clj:56) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.RestFn.invoke(RestFn.java:445) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$run_schema_migrations_BANG_.invokeStatic(setup.clj:147) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$run_schema_migrations_BANG_.invoke(setup.clj:141) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$setup_db_BANG_$fn__51132$fn__51133.invoke(setup.clj:165) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.util.jvm$do_with_us_locale.invokeStatic(jvm.clj:239) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.util.jvm$do_with_us_locale.invoke(jvm.clj:225) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$setup_db_BANG_$fn__51132.invoke(setup.clj:160) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$setup_db_BANG_.invokeStatic(setup.clj:159) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db.setup$setup_db_BANG_.invoke(setup.clj:153) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db$setup_db_BANG_$fn__51152.invoke(db.clj:69) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db$setup_db_BANG_.invokeStatic(db.clj:64) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.db$setup_db_BANG_.invoke(db.clj:55) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$init_BANG__STAR_.invokeStatic(core.clj:116) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$init_BANG__STAR_.invoke(core.clj:101) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$init_BANG_.invokeStatic(core.clj:159) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$init_BANG_.invoke(core.clj:154) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$start_normally.invokeStatic(core.clj:171) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$start_normally.invoke(core.clj:165) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$entrypoint.invokeStatic(core.clj:204) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.core$entrypoint.doInvoke(core.clj:198) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.RestFn.invoke(RestFn.java:397) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.AFn.applyToHelper(AFn.java:152) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.RestFn.applyTo(RestFn.java:132) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.Var.applyTo(Var.java:705) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.core$apply.invokeStatic(core.clj:667) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.core$apply.invoke(core.clj:662) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.bootstrap$_main.invokeStatic(bootstrap.clj:31) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.bootstrap$_main.doInvoke(bootstrap.clj:28) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.RestFn.invoke(RestFn.java:397) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.AFn.applyToHelper(AFn.java:152) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at clojure.lang.RestFn.applyTo(RestFn.java:132) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at metabase.bootstrap.main(Unknown Source) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - Caused by: java.lang.ClassCastException: class java.lang.ExceptionInInitializerError cannot be cast to class java.lang.Exception (java.lang.ExceptionInInitializerError and java.lang.Exception are in module java.base of loader 'bootstrap') <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.command.CommandScope.logPrimaryExceptionToMdc(CommandScope.java:287) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - at liquibase.command.CommandScope.execute(CommandScope.java:245) <30>1 2024-04-17T15:40:12Z cloudron2 7541955b-ee8d-4327-a473-9fa434803fb3 1511 7541955b-ee8d-4327-a473-9fa434803fb3 - ... 49 more
Apr 17 11:40:12 2024-04-17 15:40:12,392 INFO metabase.core :: Metabase Shutting Down ...
Apr 17 11:40:12 2024-04-17 15:40:12,392 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
Apr 17 11:40:12 2024-04-17 15:40:12,396 WARN db.liquibase :: ()
Apr 17 11:40:12 2024-04-17 15:40:12,397 INFO metabase.core :: Metabase Shutdown COMPLETE

If I install a new instance of the Metabase app I get to the initialization screen and can start setting it up so I believe the issue is only with existing instances that are trying to execute the migration.

[nebulon]

Looks like the error is coming from a plugin at /app/code/plugins/csv.metabase-driver.jar which does not seem to exist in the default installation. Maybe this is incompatible with latest metabase?

[girish]

https://github.com/metabase/metabase/issues/40546 is probably the upstream issue. There is some fix at https://github.com/metabase/metabase/pull/40547 .

Our own metabase instance updated fine but maybe because it's quite an old instance. Is yours fairly recent? Looks like this bug was introduced in some recent version/migration with new installations (database).

[ChristopherMag]

@nebulon Thank you so much, I have deleted /app/code/plugins/csv.metabase-driver.jar, restarted metabase, and it is working as expected.

I did not correlate the migration failure with the csv driver being present as it didn't seem like the error came from that driver being initialized but instead came from liquidbase but removing it obviously worked, thank you!