Gitlab - How to restore / migrate a non Cloudron Gitlab

[BrutalBirdie]

People switching to Cloudron might have Gitlab running somewhere else already and want to migrate their Gitlab to the Cloudron app.

Problem is trying to follow the official Gitlab documentation for restoring from a backup.

https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab

A guide with Cloudron would be great!

Did anyone ever did this?

[girish]

@brutalbirdie I don't have an installation outside Cloudron but I tried a test backup/restore of between Cloudron instances. In the process, I found a bug in the package, so please update to latest package first (1.55.3).

Backup

This is for backup for an app installed in Cloudron. If GitLab is outside Cloudron, skip this step and somehow make a backup following https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab

• Installed GitLab. Creates repos, uploaded ssh keys and made some configuration changes.

• In a Web Terminal, I typed:

sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production

This creates a backup .tar file under /app/data/backups. Mine was named 1632462433_2021_09_24_14.2.3_gitlab_backup.tar

Restore

• Install GitLab on Cloudron. Make sure the GitLab version matches between what you have outside Cloudron with the Cloudron package. The current GitLab version on Cloudron is 14.2.4.

• Fix up the db file - This is a slightly awkward step. The dump inside the backup contains a reference to the previous installation's postgresql user. First, make a note of the new postgresql user. You can do this by running env | grep POSTGRESQL_USERNAME in the web temrinal (let's call this useryyyy). Now to update the dump, on linux atleast, this is easy using the Gnome archive manager. You can just open the tar file. Inside there is db/database.sql.gz. Open that again. Then in text editor, just replace all the userxxx in the file to be this new useryyyy. When you save and close the editor, it will ask to update the database.sql.gz. And then when you update the database.sql.gz, it will ask to update the backup .tar file. (there are two updates/saves).

• Now upload the .tar file in /app/data/backups of the new installation. Change owner of the backup file to git user. You can do this in the File manager or using chown

• In Web terminal,

sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production

I got output like:

...
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring uploads ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring builds ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring artifacts ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring pages ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring lfs objects ... 
2021-09-24 06:22:36 +0000 -- done
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.

Deleting backups/tmp ... done
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need to restore these files manually.
Restore task is done.

• Finally, copy over the secrets.yml from old installation into new installation's /app/data/secrets.yml. You might also have to copy over other gitlab.yml settings into /app/data/gitlab.yml.

[BrutalBirdie]

@girish Awesome! Will test this out!

[BrutalBirdie]

@girish said in Gitlab - How to restore / migrate a non Cloudron Gitlab:

Now to update the dump, on linux atleast, this is easy using the Gnome archive manager. You can just open the tar file. Inside there is db/database.sql.gz. Open that again. Then in text editor, just replace all the userxxx in the file to be this new useryyyy.

I am not quite sure which userxxx you are referring to which is to be replaced.
A more explicit search string would be really helpful.

[BrutalBirdie]

@brutalbirdie Do you mean all OWNER TO gitlab;?

So in RegEx OWNER TO.*;$

[girish]

@brutalbirdie If you open that database.sql file, there will be a LOT of userxxxx references (referring to the old database user). You can to replace all with the new database username.

[girish]

@brutalbirdie So, in the database.sql.gz, there are a lots of lines like ALTER FUNCTION public.trigger_21e7a2602957() OWNER TO user57c829faa1694c05980bbc6113b30ae2; (and many other OWNER stuff). The user57c829faa1694c05980bbc6113b30ae2 is the postgres username of the old installation. It has to be changed to the new postgres username from Cloudron. Mine was user970fb4e4a4654d1087af2cb720a159e1

Something like: sed -e 's/user57c829faa1694c05980bbc6113b30ae2/user970fb4e4a4654d1087af2cb720a159e1/' -i database.sql.

[BrutalBirdie]

@girish Problem is the DB I try to fix had the user gitlab and replacing just the string gitlab would destroy the DB.
If only the OWNER TO $DB_USER needs to be changed, then all good.

But from your answer I am still not sure if this is the only part where the user needs to be changed.

[girish]

@brutalbirdie ah, I understand your predicament now. That's correct only the OWNER TO needs to be replaced.

[BrutalBirdie]

@girish TOP! Will try it out on Monday.

[BrutalBirdie]

@brutalbirdie

2021-09-27 11:16:52 +0000 -- Cleaning the database ... 
2021-09-27 11:16:54 +0000 -- done
2021-09-27 11:16:54 +0000 -- Restoring database ... 
Restoring PostgreSQL database dbd5499e3cf81b43d093724d69fa223688 ... ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension pg_trgm
. . .
ALTER TABLE
ALTER TABLE
[DONE]
2021-09-27 11:17:24 +0000 -- done
2021-09-27 11:17:24 +0000 -- Restoring repositories ...
rake aborted!
TypeError: no implicit conversion of nil into String
/home/git/gitlab/lib/backup/gitaly_backup.rb:75:in `absolute_path'
/home/git/gitlab/lib/backup/gitaly_backup.rb:75:in `bin_path'
/home/git/gitlab/lib/backup/gitaly_backup.rb:28:in `start'
/home/git/gitlab/lib/backup/repositories.rb:47:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:124:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:80:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:restore

hmm

[BrutalBirdie]

I added a full --trace log here https://paste.cloudron.dev/mocukugoho.md

[girish]

@brutalbirdie great success! that exact error is what I fixed in package v1.55.3 . Are you using that package version?

I think you can also make it work in older packages with adding the line in /app/data/gitlab.yml:

backup:
    gitaly_backup_path: /home/git/gitaly/gitaly-backup

[BrutalBirdie]

@girish

App Title and Version
GitLab 14.2.3
App ID
d5499e3c-f81b-43d0-9372-4d69fa223688
Package Version
com.gitlab.cloudronapp@1.55.3
Last Updated
3 days ago

Btw. The Gitlab Version in this information is wrong, the version should be 14.2.4 since this is the version running in this release tag.
So I am running the v1.55.3
I will add gitaly_backup_path: /home/git/gitaly/gitaly-backup to backup: and try again, after a app restart.

[BrutalBirdie]

@girish
The last error

2021-09-28 06:36:02 +0000 -- Restoring container registry images ... 
rake aborted!
Errno::ENOENT: No such file or directory - /app/data/backups/registry.tar.gz
/home/git/gitlab/lib/backup/files.rb:95:in `run_pipeline!'
/home/git/gitlab/lib/backup/files.rb:60:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:277:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:86:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:registry:restore
(See full trace by running task with --trace)

But I think I can ignore this one since the old gitlab simply did not backup the registry.
Now the gitlab goes thru testing by his Owner.

[BrutalBirdie]

@girish The https://DOMAIN.TLD//admin/runners return error code 500.
This should be a problem with the migrated secrets.
The secrets where a gitlab-secrets.json I converted the json to yaml.

After that I noticed that gitlab_rails should be production.
Some more testing...

[BrutalBirdie]

Ahhh finally.
I deleted the registry: part in the secrets.yaml and now runners are accessible again!

After a final testing by the owner, I will write up a doc for this type of migration.

[BrutalBirdie]

Looks like we are all good now.
I will submit a PR for the docs on gitlab for a migration step.

[BrutalBirdie]

@girish

https://git.cloudron.io/cloudron/docs/-/merge_requests/28

I created a PR for this topic as a guide.
Have fun reading and fixing my mistakes (if there is something to change hit me up in a dm/chat/PR)

[girish]

@brutalbirdie thanks! I have merged it but I will move it to a separate Guide doc.