People switching to Cloudron might have Gitlab running somewhere else already and want to migrate their Gitlab to the Cloudron app. Problem is trying to follow the official Gitlab documentation for restoring from a backup. https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab A guide with Cloudron would be great! Did anyone ever did this?

@ brutalbirdie I don't have an installation outside Cloudron but I tried a test backup/restore of between Cloudron instances. In the process, I found a bug in the package, so please update to latest package first (1.55.3). Backup This is for backup for an app installed in Cloudron. If GitLab is outside Cloudron, skip this step and somehow make a backup following https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab Installed GitLab. Creates repos, uploaded ssh keys and made some configuration changes. In a Web Terminal, I typed: sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production This creates a backup .tar file under /app/data/backups . Mine was named 1632462433_2021_09_24_14.2.3_gitlab_backup.tar Restore Install GitLab on Cloudron. Make sure the GitLab version matches between what you have outside Cloudron with the Cloudron package. The current GitLab version on Cloudron is 14.2.4. Fix up the db file - This is a slightly awkward step. The dump inside the backup contains a reference to the previous installation's postgresql user. First, make a note of the new postgresql user. You can do this by running env | grep POSTGRESQL_USERNAME in the web temrinal (let's call this useryyyy). Now to update the dump, on linux atleast, this is easy using the Gnome archive manager. You can just open the tar file. Inside there is db/database.sql.gz . Open that again. Then in text editor, just replace all the userxxx in the file to be this new useryyyy. When you save and close the editor, it will ask to update the database.sql.gz. And then when you update the database.sql.gz, it will ask to update the backup .tar file. (there are two updates/saves). Now upload the .tar file in /app/data/backups of the new installation. Change owner of the backup file to git user. You can do this in the File manager or using chown In Web terminal, sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production I got output like: ... 2021-09-24 06:22:36 +0000 -- done 2021-09-24 06:22:36 +0000 -- Restoring uploads ... 2021-09-24 06:22:36 +0000 -- done 2021-09-24 06:22:36 +0000 -- Restoring builds ... 2021-09-24 06:22:36 +0000 -- done 2021-09-24 06:22:36 +0000 -- Restoring artifacts ... 2021-09-24 06:22:36 +0000 -- done 2021-09-24 06:22:36 +0000 -- Restoring pages ... 2021-09-24 06:22:36 +0000 -- done 2021-09-24 06:22:36 +0000 -- Restoring lfs objects ... 2021-09-24 06:22:36 +0000 -- done This task will now rebuild the authorized_keys file. You will lose any data stored in the authorized_keys file. Deleting backups/tmp ... done Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data and are not included in this backup. You will need to restore these files manually. Restore task is done. Finally, copy over the secrets.yml from old installation into new installation's /app/data/secrets.yml . You might also have to copy over other gitlab.yml settings into /app/data/gitlab.yml .

Looks like we are all good now. I will submit a PR for the docs on gitlab for a migration step.

Cloudron Forum

Apps | Demo | Docs | Install

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Gitlab - How to restore / migrate a non Cloudron Gitlab

Solved GitLab

20 Posts 2 Posters 2.8k Views 2 Watching

G girish
@brutalbirdie I don't have an installation outside Cloudron but I tried a test backup/restore of between Cloudron instances. In the process, I found a bug in the package, so please update to latest package first (1.55.3).

Backup

This is for backup for an app installed in Cloudron. If GitLab is outside Cloudron, skip this step and somehow make a backup following https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab
- Installed GitLab. Creates repos, uploaded ssh keys and made some configuration changes.
- In a Web Terminal, I typed:
```
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
```
This creates a backup .tar file under /app/data/backups. Mine was named 1632462433_2021_09_24_14.2.3_gitlab_backup.tar

Restore
- Install GitLab on Cloudron. Make sure the GitLab version matches between what you have outside Cloudron with the Cloudron package. The current GitLab version on Cloudron is 14.2.4.
- Fix up the db file - This is a slightly awkward step. The dump inside the backup contains a reference to the previous installation's postgresql user. First, make a note of the new postgresql user. You can do this by running env | grep POSTGRESQL_USERNAME in the web temrinal (let's call this useryyyy). Now to update the dump, on linux atleast, this is easy using the Gnome archive manager. You can just open the tar file. Inside there is db/database.sql.gz. Open that again. Then in text editor, just replace all the userxxx in the file to be this new useryyyy. When you save and close the editor, it will ask to update the database.sql.gz. And then when you update the database.sql.gz, it will ask to update the backup .tar file. (there are two updates/saves).
- Now upload the .tar file in /app/data/backups of the new installation. Change owner of the backup file to git user. You can do this in the File manager or using chown
- In Web terminal,
```
sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production
```
I got output like:
```
...
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring uploads ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring builds ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring artifacts ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring pages ... 
2021-09-24 06:22:36 +0000 -- done
2021-09-24 06:22:36 +0000 -- Restoring lfs objects ... 
2021-09-24 06:22:36 +0000 -- done
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.

Deleting backups/tmp ... done
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need to restore these files manually.
Restore task is done.
```
- Finally, copy over the secrets.yml from old installation into new installation's /app/data/secrets.yml. You might also have to copy over other gitlab.yml settings into /app/data/gitlab.yml.
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#4

@girish said in Gitlab - How to restore / migrate a non Cloudron Gitlab:

Now to update the dump, on linux atleast, this is easy using the Gnome archive manager. You can just open the tar file. Inside there is db/database.sql.gz. Open that again. Then in text editor, just replace all the userxxx in the file to be this new useryyyy.

I am not quite sure which userxxx you are referring to which is to be replaced.
A more explicit search string would be really helpful.
Like my work? Consider donating a drink. Cheers!
B 1 Reply Last reply

0
B BrutalBirdie

@girish said in Gitlab - How to restore / migrate a non Cloudron Gitlab:

Now to update the dump, on linux atleast, this is easy using the Gnome archive manager. You can just open the tar file. Inside there is db/database.sql.gz. Open that again. Then in text editor, just replace all the userxxx in the file to be this new useryyyy.

I am not quite sure which userxxx you are referring to which is to be replaced.
A more explicit search string would be really helpful.
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#5

@brutalbirdie Do you mean all OWNER TO gitlab;?

So in RegEx OWNER TO.*;$
Like my work? Consider donating a drink. Cheers!
G 2 Replies Last reply

0
B BrutalBirdie

@brutalbirdie Do you mean all OWNER TO gitlab;?

So in RegEx OWNER TO.*;$
G Offline
G Offline
girish
Staff
wrote on last edited by

#6

@brutalbirdie If you open that database.sql file, there will be a LOT of userxxxx references (referring to the old database user). You can to replace all with the new database username.
1 Reply Last reply

0
B BrutalBirdie

@brutalbirdie Do you mean all OWNER TO gitlab;?

So in RegEx OWNER TO.*;$
G Offline
G Offline
girish
Staff
wrote on last edited by girish

#7

@brutalbirdie So, in the database.sql.gz, there are a lots of lines like ALTER FUNCTION public.trigger_21e7a2602957() OWNER TO user57c829faa1694c05980bbc6113b30ae2; (and many other OWNER stuff). The user57c829faa1694c05980bbc6113b30ae2 is the postgres username of the old installation. It has to be changed to the new postgres username from Cloudron. Mine was user970fb4e4a4654d1087af2cb720a159e1

Something like: sed -e 's/user57c829faa1694c05980bbc6113b30ae2/user970fb4e4a4654d1087af2cb720a159e1/' -i database.sql.
B 1 Reply Last reply

0
G girish

@brutalbirdie So, in the database.sql.gz, there are a lots of lines like ALTER FUNCTION public.trigger_21e7a2602957() OWNER TO user57c829faa1694c05980bbc6113b30ae2; (and many other OWNER stuff). The user57c829faa1694c05980bbc6113b30ae2 is the postgres username of the old installation. It has to be changed to the new postgres username from Cloudron. Mine was user970fb4e4a4654d1087af2cb720a159e1

Something like: sed -e 's/user57c829faa1694c05980bbc6113b30ae2/user970fb4e4a4654d1087af2cb720a159e1/' -i database.sql.
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#8

@girish Problem is the DB I try to fix had the user gitlab and replacing just the string gitlab would destroy the DB.
If only the OWNER TO $DB_USER needs to be changed, then all good.

But from your answer I am still not sure if this is the only part where the user needs to be changed.
Like my work? Consider donating a drink. Cheers!
G 1 Reply Last reply

0
B BrutalBirdie

@girish Problem is the DB I try to fix had the user gitlab and replacing just the string gitlab would destroy the DB.
If only the OWNER TO $DB_USER needs to be changed, then all good.

But from your answer I am still not sure if this is the only part where the user needs to be changed.
G Offline
G Offline
girish
Staff
wrote on last edited by

#9

@brutalbirdie ah, I understand your predicament now. That's correct only the OWNER TO needs to be replaced.
B 1 Reply Last reply

2
G girish

@brutalbirdie ah, I understand your predicament now. That's correct only the OWNER TO needs to be replaced.
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#10

@girish TOP! Will try it out on Monday.
Like my work? Consider donating a drink. Cheers!
B 1 Reply Last reply

1

B BrutalBirdie

@girish TOP! Will try it out on Monday.

BrutalBirdie

Partner

wrote on

#11

@brutalbirdie

2021-09-27 11:16:52 +0000 -- Cleaning the database ... 
2021-09-27 11:16:54 +0000 -- done
2021-09-27 11:16:54 +0000 -- Restoring database ... 
Restoring PostgreSQL database dbd5499e3cf81b43d093724d69fa223688 ... ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension pg_trgm
. . .
ALTER TABLE
ALTER TABLE
[DONE]
2021-09-27 11:17:24 +0000 -- done
2021-09-27 11:17:24 +0000 -- Restoring repositories ...
rake aborted!
TypeError: no implicit conversion of nil into String
/home/git/gitlab/lib/backup/gitaly_backup.rb:75:in `absolute_path'
/home/git/gitlab/lib/backup/gitaly_backup.rb:75:in `bin_path'
/home/git/gitlab/lib/backup/gitaly_backup.rb:28:in `start'
/home/git/gitlab/lib/backup/repositories.rb:47:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:124:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:80:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:restore

hmm

B BrutalBirdie

@brutalbirdie

2021-09-27 11:16:52 +0000 -- Cleaning the database ... 
2021-09-27 11:16:54 +0000 -- done
2021-09-27 11:16:54 +0000 -- Restoring database ... 
Restoring PostgreSQL database dbd5499e3cf81b43d093724d69fa223688 ... ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension pg_trgm
. . .
ALTER TABLE
ALTER TABLE
[DONE]
2021-09-27 11:17:24 +0000 -- done
2021-09-27 11:17:24 +0000 -- Restoring repositories ...
rake aborted!
TypeError: no implicit conversion of nil into String
/home/git/gitlab/lib/backup/gitaly_backup.rb:75:in `absolute_path'
/home/git/gitlab/lib/backup/gitaly_backup.rb:75:in `bin_path'
/home/git/gitlab/lib/backup/gitaly_backup.rb:28:in `start'
/home/git/gitlab/lib/backup/repositories.rb:47:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:124:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:80:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:restore

hmm

BrutalBirdie

Partner

wrote on

#12

I added a full --trace log here https://paste.cloudron.dev/mocukugoho.md

B BrutalBirdie

I added a full --trace log here https://paste.cloudron.dev/mocukugoho.md
G Offline
G Offline
girish
Staff
wrote on last edited by girish

#13
@brutalbirdie great success! that exact error is what I fixed in package v1.55.3 . Are you using that package version?

I think you can also make it work in older packages with adding the line in /app/data/gitlab.yml:
```
backup:
    gitaly_backup_path: /home/git/gitaly/gitaly-backup
```
B 4 Replies Last reply

1
G girish
@brutalbirdie great success! that exact error is what I fixed in package v1.55.3 . Are you using that package version?

I think you can also make it work in older packages with adding the line in /app/data/gitlab.yml:
```
backup:
    gitaly_backup_path: /home/git/gitaly/gitaly-backup
```
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#14
@girish
```
App Title and Version
GitLab 14.2.3
App ID
d5499e3c-f81b-43d0-9372-4d69fa223688
Package Version
com.gitlab.cloudronapp@1.55.3
Last Updated
3 days ago
```
Btw. The Gitlab Version in this information is wrong, the version should be 14.2.4 since this is the version running in this release tag.
So I am running the v1.55.3
I will add gitaly_backup_path: /home/git/gitaly/gitaly-backup to backup: and try again, after a app restart.
Like my work? Consider donating a drink. Cheers!
1 Reply Last reply

0

G girish

@brutalbirdie great success! that exact error is what I fixed in package v1.55.3 . Are you using that package version?

I think you can also make it work in older packages with adding the line in /app/data/gitlab.yml:

backup:
    gitaly_backup_path: /home/git/gitaly/gitaly-backup

BrutalBirdie

Partner

wrote on

#15

@girish
The last error

2021-09-28 06:36:02 +0000 -- Restoring container registry images ... 
rake aborted!
Errno::ENOENT: No such file or directory - /app/data/backups/registry.tar.gz
/home/git/gitlab/lib/backup/files.rb:95:in `run_pipeline!'
/home/git/gitlab/lib/backup/files.rb:60:in `restore'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:277:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:86:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:registry:restore
(See full trace by running task with --trace)

But I think I can ignore this one since the old gitlab simply did not backup the registry.
Now the gitlab goes thru testing by his Owner.

G girish
@brutalbirdie great success! that exact error is what I fixed in package v1.55.3 . Are you using that package version?

I think you can also make it work in older packages with adding the line in /app/data/gitlab.yml:
```
backup:
    gitaly_backup_path: /home/git/gitaly/gitaly-backup
```
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#16

@girish The https://DOMAIN.TLD//admin/runners return error code 500.
This should be a problem with the migrated secrets.
The secrets where a gitlab-secrets.json I converted the json to yaml.

After that I noticed that gitlab_rails should be production.
Some more testing...
Like my work? Consider donating a drink. Cheers!
B 1 Reply Last reply

0
B BrutalBirdie

@girish The https://DOMAIN.TLD//admin/runners return error code 500.
This should be a problem with the migrated secrets.
The secrets where a gitlab-secrets.json I converted the json to yaml.

After that I noticed that gitlab_rails should be production.
Some more testing...
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by BrutalBirdie

#17

Ahhh finally.
I deleted the registry: part in the secrets.yaml and now runners are accessible again!

After a final testing by the owner, I will write up a doc for this type of migration.
Like my work? Consider donating a drink. Cheers!
B 1 Reply Last reply

1
B BrutalBirdie

Ahhh finally.
I deleted the registry: part in the secrets.yaml and now runners are accessible again!

After a final testing by the owner, I will write up a doc for this type of migration.
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#18

Looks like we are all good now.
I will submit a PR for the docs on gitlab for a migration step.
Like my work? Consider donating a drink. Cheers!
1 Reply Last reply

4
G girish
@brutalbirdie great success! that exact error is what I fixed in package v1.55.3 . Are you using that package version?

I think you can also make it work in older packages with adding the line in /app/data/gitlab.yml:
```
backup:
    gitaly_backup_path: /home/git/gitaly/gitaly-backup
```
B Offline
B Offline
BrutalBirdie
Partner
wrote on last edited by

#19

@girish

https://git.cloudron.io/cloudron/docs/-/merge_requests/28

I created a PR for this topic as a guide.
Have fun reading and fixing my mistakes (if there is something to change hit me up in a dm/chat/PR)
Like my work? Consider donating a drink. Cheers!
G 1 Reply Last reply

2
B BrutalBirdie

@girish

https://git.cloudron.io/cloudron/docs/-/merge_requests/28

I created a PR for this topic as a guide.
Have fun reading and fixing my mistakes (if there is something to change hit me up in a dm/chat/PR)
G Offline
G Offline
girish
Staff
wrote on last edited by

#20

@brutalbirdie thanks! I have merged it but I will move it to a separate Guide doc.
1 Reply Last reply

2
B BrutalBirdie referenced this topic on
B BrutalBirdie referenced this topic on

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Gitlab - How to restore / migrate a non Cloudron Gitlab

Backup

Restore