Umami update 1.37.0 (package version 1.6.0) won't start, errors in logs

d19dotca

Hello,

Updating Umami to the latest version released a bit ago shows Umami constantly trying to start but fails, and the logs show these errors:

Aug 09 23:00:31 Gerror Command failed with exit code 1.
Aug 09 23:00:31 Ginfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Aug 09 23:00:38 ==> Changing ownership
Aug 09 23:00:38 => Building database
Aug 09 23:00:38 => Existing prisma managed, running migrations
Aug 09 23:00:39 Gyarn run v1.22.17
Aug 09 23:00:39 Gwarning Skipping preferred cache folder "/home/cloudron/.cache/yarn" because it is not writable.
Aug 09 23:00:39 Gwarning Selected the next writable cache folder in the list, will be "/tmp/.yarn-cache-1000".
Aug 09 23:00:39 G$ /app/code/node_modules/.bin/prisma migrate resolve --applied 01_init
Aug 09 23:00:39 Gwarning Cannot find a suitable global folder. Tried these: "/usr/local, /home/cloudron/.yarn"
Aug 09 23:00:39 Environment variables loaded from .env
Aug 09 23:00:39 Prisma schema loaded from prisma/schema.prisma
Aug 09 23:00:39 Datasource "db": PostgreSQL database "dbc437d27f129b45cca6a7b0ce4bcb6fa9", schema "public" at "postgresql:5432"
Aug 09 23:00:39 Error: P3008
Aug 09 23:00:39
Aug 09 23:00:39 The migration `01_init` is already recorded as applied in the database.
Aug 09 23:00:39
Aug 09 23:00:40 Gerror Command failed with exit code 1.
Aug 09 23:00:40 Ginfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Aug 09 23:00:40 Gyarn run v1.22.17
Aug 09 23:00:40 Gwarning Skipping preferred cache folder "/home/cloudron/.cache/yarn" because it is not writable.
Aug 09 23:00:40 Gwarning Selected the next writable cache folder in the list, will be "/tmp/.yarn-cache-1000".
Aug 09 23:00:40 G$ prisma migrate deploy
Aug 09 23:00:40 Gwarning Cannot find a suitable global folder. Tried these: "/usr/local, /home/cloudron/.yarn"
Aug 09 23:00:41 Environment variables loaded from .env
Aug 09 23:00:41 Prisma schema loaded from prisma/schema.prisma
Aug 09 23:00:41 Datasource "db": PostgreSQL database "dbc437d27f129b45cca6a7b0ce4bcb6fa9", schema "public" at "postgresql:5432"
Aug 09 23:00:41
Aug 09 23:00:41 2 migrations found in prisma/migrations
Aug 09 23:00:41
Aug 09 23:00:41 Error: P3009
Aug 09 23:00:41
Aug 09 23:00:41 migrate found failed migrations in the target database, new migrations will not be applied. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
Aug 09 23:00:41 The `02_add_event_data` migration started at 2022-08-10 06:00:10.447969 UTC failed
Aug 09 23:00:41
Aug 09 23:00:41
Aug 09 23:00:41 Gerror Command failed with exit code 1.
Aug 09 23:00:41 Ginfo Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Not sure if this is a packaging issue or a Umami issue. Rolled back for now.

girish

@d19dotca revoked for now. There are auto update tests, so I wonder what went wrong here. Testing manually seems to work though.

girish

Looks like maybe related to https://github.com/umami-software/umami/issues/1229

jdaviescoates

@girish yeah, see also the note here https://github.com/umami-software/umami/releases

Important

This release includes a new database migration. In this update we are making some changes to Events. Now instead of event_type and event_value columns, there is going to be a single renamed column, event_name. Having the two separate columns caused a lot of confusion regarding what to store in each field. There is now a new table called event_data which is used to store event related data. In the future you will be able to run reports against your events and event data.

Now to track events, you simply choose an event name, like button-click and optionally send some JSON data along with it.

umami.trackEvent('button-click', { userId: 123, buttonName: 'Newsletter signup' });

When you start the app for the first time, the database changes will be automatically applied. Your data however will not be automatically migrated because we can't determine what event_name you will want to use. Your old event table has been renamed to _event_old, which you can use to manually move data to the new table. If you've never used events, feel free to delete the _event_old table.

girish

The root cause is an upstream issue - https://github.com/umami-software/umami/issues/1393 and https://github.com/umami-software/umami/issues/1406 . Umami changed their db migration to prisma some releases ago. However, the index names created by prisma and old migration logic is different.

I don't know if this is easy to automation for migration, let's see what they come up with upstream.

jdaviescoates

@girish I know you revoked the previous update, but there is now another Umami 1.37.0 update available - is that one safe to update to?

It's a bit unclear, even though there is this warning

This update is a pre-release and not considered stable yet. Please update at your own risk.

girish

@jdaviescoates yeah, did that because we didn't want new installations to have the old schema. It seems people in old installs have to re-install - https://github.com/umami-software/umami/issues/1406#issuecomment-1213355769 . The schema migration between old umami installs and the new one is not easily automatable it seems if you follow that thread.

girish

OK, I have gone ahead and re-published as package version 2.0.0. This means that the app won't auto upgrade.

I put this in the changelog:

IMPORTANT: The database upgrade fails if you have been running umami for a while. There is no easy work around for this, you can manually try the database modifications in this upstream bug after you upgrade and place the app in Repair mode.
If possible, just start with a fresh installation of Umami. It can be quite a time sink to fix the database schema and you might still hit some issue later in future versions.

jdaviescoates

Thanks @girish but re:

The database upgrade fails if you have been running umami for a while.

How long is "a while"?

Seems a bit vague.

Also, it's hard for me to grok what difference how long one has been running the app would/ could make any difference?

Surely the db schema has changed from one version to the next even if I've only been running it for a millisecond? (I've actually been running it since May - is that "a while"?)

girish

@jdaviescoates Best I can make out, it all started with Umami release 1.33 - https://github.com/umami-software/umami/releases/tag/v1.33 . This release was in Cloudron package 1.2.0 (around Jun 27 2022). From upstream release notes, "This release will use Prisma migrations for the first time".

If you installed umami before that, I think upgrades will fail.

Best way to check really is to clone current app and try to upgrade the cloned instance and see if it upgrades or not

echokos

@girish This should be unacceptable to Cloudron IMO. If Umami can't provide reliable updates or at least release a repaired update after it's known to be broken, it shouldn't be in Cloudron. I use Cloudron so I don't have to deal with these kinds of things and certainly won't reinstall and start from scratch due to a bug in an update - that would include re-tagging 7 sites currently tracked in the software.

jdaviescoates

@echokos I understand your sentiment, but disagree.

It's impossible to know the future and any app could potentially change resulting in something like this happening (although it's obviously more likely with less mature apps).

But regardless of this existing issue I'd hate for Umami not to be on Cloudron because it's great! So easy to get up and running and works really nicely.

nebulon

I agree, such situations where upstream leaves us with no good path to migrate existing instances are pretty bad. We usually spend much time on finding ways to migrate, as it is indeed part of what we see as value we provide. In this case however @girish was next to me for quite some time trying to find a way, but after explaining the introduced mismatch of table foreign key constraint ids, it simply wasn't possible.

Depending on the case, we sometimes create a new appstore id for the app package to avoid existing installations to break on update, but it also means there won't be any more updates and users are still stuck

d19dotca

This is a difficult one. I feel like to some degree I've lost my use of Umami now, because I can't upgrade properly and it doesn't seem easy enough to deploy a copy of the latest version and migrate the data to it. That's not Cloudron's fault at all, I'm just very surprised this issue wasn't taken more seriously by the upstream developers. Guess I'll have to stick to Matomo for a while longer.

I had been using Umami (in addition to continuing with Matomo) for the past 2 or 3 months, hoping to cut over to it fully for my clients in another month or so and ditch Matomo after they had about 90 days of data, but that seems unlikely with the breaking changes they made to Umami and being unable to proceed without losing data. Unless I've misunderstood something about the situation (fingers crossed I may have and there's maybe a way I've overlooked to migrate / upgrade), but I'm guessing that's not the case and may need more time to review.

robi

@d19dotca Rough indeed.

You could think about using two instances and a cut-over by installing a new Umami, copying over the config file, then cutting over traffic to the new instance, keeping the old one around a bit, in case you need the data, which will become less useful over time and ready for cleanup in a month or two.

echokos

@nebulon Umami now has updated their software that seemingly solves a lot of people's former migration issues. Any word on when an updated Cloudron package will come containing their most recent build (1.39.5 from November '22)?

girish

@echokos Umami is at the latest since nov - https://forum.cloudron.io/topic/6936/umami-package-updates/19 . It has upstream version 1.39.5

echokos

@girish Thanks. The pop-up when I click upgrade shows:

girish

@echokos Which version of Cloudron are you running ? It's in the Settings page.

On the demo cloudron, it shows:

echokos

@girish Platform version: v7.3.4 (Ubuntu 20.04.4 LTS)

echokos

@girish I see the issue - it's not available for my upgrade path for some reason. I had to lock my current version at package version 1.51 due to the upgrade issues. However, the current version of Umami should fix the upgrade issues - it's just not avaialble as an upgrade for me in Cloudron... it's forcing the new version to be a new install only.

girish

@echokos Ah, I see what you are saying. Maybe you can try update via the CLI (please take a backup of current umami before you do this):

cloudron update --appstore-id is.umami.cloudronapp@2.3.0 --app umami.domain.com

This will allow you to jump versions.

echokos

@girish That worked! And the migration worked properly. Thank you!

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Umami update 1.37.0 (package version 1.6.0) won't start, errors in logs