Cloudron Forum

CarbonBee

Hi,

I have a major issue on both my GitLab servers.I just updated them to 1.43.0 and both are not responding.
Here is the full log :

Dec 02 08:07:22 ==> Enabling LDAP integration
Dec 02 08:07:22 ==> Reusing existing host keys
Dec 02 08:07:22 ==> Fixing secrets
Dec 02 08:07:22 ==> Initializing gitlab shell
Dec 02 08:07:22 ==> Copying schema file
Dec 02 08:07:22 ==> Setting up tmp
Dec 02 08:07:23 ==> Fixing permissions
Dec 02 08:07:23 ==> Creating pg_trgm extension
Dec 02 08:07:23 NOTICE: extension "pg_trgm" already exists, skipping
Dec 02 08:07:23 ==> Upgrading existing db
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:29: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:30: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:23 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:24 `/home/git` is not writable.
Dec 02 08:07:24 Bundler will use `/tmp/bundler20201202-44-14wdz8y44' as your home directory temporarily.
Dec 02 08:07:24 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:24 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:24 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:28 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:32 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:34 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:36 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/acts-as-taggable-on-6.5.0/lib/acts_as_taggable_on/tagging.rb:9: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.9.0/lib/marginalia.rb:94: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Dec 02 08:07:37 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql_adapter.rb:648: warning: The called method `execute_and_clear_without_marginalia' is defined here
Dec 02 08:07:43 == 20201008013434 GenerateCiJwtSigningKey: migrating ==========================
Dec 02 08:07:43 rake aborted!
Dec 02 08:07:43 StandardError: An error has occurred, this and all later migrations canceled:
Dec 02 08:07:43
Dec 02 08:07:43 key must be 32 bytes or longer
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:60:in `crypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:36:in `encrypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:266:in `encrypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:350:in `encrypt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:165:in `block (2 levels) in attr_encrypted'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:77:in `block in attr_encrypted'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:50:in `public_send'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:50:in `_assign_attribute'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:42:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activemodel-6.0.3.3/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:29:in `perform_attribute_assignment'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/persistence.rb:620:in `block in update'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:212:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/persistence.rb:619:in `update'
Dec 02 08:07:43 /home/git/gitlab/db/migrate/20201008013434_generate_ci_jwt_signing_key.rb:21:in `block in up'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:70:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:70:in `block in find_each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:238:in `block in in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:222:in `loop'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:222:in `in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:135:in `find_in_batches'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/relation/batches.rb:69:in `find_each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/querying.rb:21:in `find_each'
Dec 02 08:07:43 /home/git/gitlab/db/migrate/20201008013434_generate_ci_jwt_signing_key.rb:20:in `up'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:831:in `exec_migration'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:811:in `block in migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:810:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1002:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1361:in `block in ddl_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/transactions.rb:212:in `transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1361:in `ddl_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1280:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1280:in `migrate_without_lock'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1229:in `block in migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1382:in `with_advisory_lock'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1229:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1061:in `up'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/migration.rb:1036:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <top (required)>'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/railties/databases.rake:84:in `each'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <top (required)>'
Dec 02 08:07:43 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'

I'm filling a support ticket right now, but if anyone has an idea on that issue...

CarbonBee

Hi,

I just updated my NextCloud to 20.0.2, and it's not responding.

Logs said :

Nov 30 08:17:29 Check primary keys.
Nov 30 08:17:29 Adding primary key to the filecache_extended table, this can take some time...
Nov 30 08:17:29
Nov 30 08:17:29 In AbstractPostgreSQLDriver.php line 51:
Nov 30 08:17:29
Nov 30 08:17:29 An exception occurred while executing 'ALTER TABLE oc_filecache_extended AD
Nov 30 08:17:29 D PRIMARY KEY (fileid)':
Nov 30 08:17:29
Nov 30 08:17:29 SQLSTATE[23505]: Unique violation: 7 ERROR: could not create unique index
Nov 30 08:17:29 "oc_filecache_extended_pkey"
Nov 30 08:17:29 DETAIL: Key (fileid)=(169141) is duplicated.
Nov 30 08:17:29
Nov 30 08:17:29
Nov 30 08:17:29 In PDOConnection.php line 83:
Nov 30 08:17:29
Nov 30 08:17:29 SQLSTATE[23505]: Unique violation: 7 ERROR: could not create unique index
Nov 30 08:17:29 "oc_filecache_extended_pkey"
Nov 30 08:17:29 DETAIL: Key (fileid)=(169141) is duplicated.
Nov 30 08:17:29
Nov 30 08:17:29
Nov 30 08:17:29 In PDOConnection.php line 78:
Nov 30 08:17:29
Nov 30 08:17:29 SQLSTATE[23505]: Unique violation: 7 ERROR: could not create unique index
Nov 30 08:17:29 "oc_filecache_extended_pkey"
Nov 30 08:17:29 DETAIL: Key (fileid)=(169141) is duplicated.
Nov 30 08:17:29
Nov 30 08:17:29
Nov 30 08:17:29 db:add-missing-primary-keys

And it's restarting every minutes.
Could someone help me on that?

Thanks,

CarbonBee

@msbt Well, any call to apt-get tells me that

You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 linux-image-generic

Which of course I can't do because of the lack of space in /boot.
I followed parts of this link :
dpkg --list 'linux-image*'|awk '{ if($1=="ii") print $2}'|grep -vuname -r``
rm -rf /boot/*-4.4.0-{161,164,178,179,184,185,186,187}-*
apt-get -f install
apt-get autoremove

And it worked fine!

CarbonBee

Hi,

Cloudron update are failing for some day, and the message is update exited with code 1 signal null

I went to journalctl -u cloudron-updater and found:

Nov 15 21:23:43 my.domain.com installer.sh[354]: dpkg: dependency problems prevent configuration of linux-image-generic:
Nov 15 21:23:43 my.domain.com installer.sh[354]:  linux-image-generic depends on linux-image-4.4.0-190-generic | linux-image-unsigned-4.4.0-190-generic; however:
Nov 15 21:23:43 my.domain.com installer.sh[354]:   Package linux-image-4.4.0-190-generic is not installed.
Nov 15 21:23:43 my.domain.com installer.sh[354]:   Package linux-image-unsigned-4.4.0-190-generic is not installed.
Nov 15 21:23:43 my.domain.com installer.sh[354]:  linux-image-generic depends on linux-modules-extra-4.4.0-190-generic; however:
Nov 15 21:23:43 my.domain.com installer.sh[354]:   Package linux-modules-extra-4.4.0-190-generic is not configured yet.
Nov 15 21:23:43 my.domain.com installer.sh[354]: dpkg: error processing package linux-image-generic (--configure):
Nov 15 21:23:43 my.domain.com installer.sh[354]:  dependency problems - leaving unconfigured
Nov 15 21:23:43 my.domain.com installer.sh[354]: Processing triggers for linux-image-4.4.0-189-generic (4.4.0-189.219) ...
Nov 15 21:23:44 my.domain.com installer.sh[354]: /etc/kernel/postinst.d/initramfs-tools:
Nov 15 21:23:44 my.domain.com installer.sh[354]: update-initramfs: Generating /boot/initrd.img-4.4.0-189-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]: cat: write error: No space left on device
Nov 15 21:23:49 my.domain.com installer.sh[354]: update-initramfs: failed for /boot/initrd.img-4.4.0-189-generic with 1.
Nov 15 21:23:49 my.domain.com installer.sh[354]: run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Nov 15 21:23:49 my.domain.com installer.sh[354]: dpkg: error processing package linux-image-4.4.0-189-generic (--configure):
Nov 15 21:23:49 my.domain.com installer.sh[354]:  subprocess installed post-installation script returned error exit status 1
Nov 15 21:23:49 my.domain.com installer.sh[354]: Errors were encountered while processing:
Nov 15 21:23:49 my.domain.com installer.sh[354]:  linux-modules-extra-4.4.0-190-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]:  linux-image-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]:  linux-image-4.4.0-189-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]: ==> installer: Failed to fix packages. Retry

I saw the line No space left on device and indeed df -h said /dev/md2 487M 482M 0 100% /boot
What is weird is that uname -nar says 4.4.0-154-generic but in /boot I have

initrd.img-4.4.0-104-generic
initrd.img-4.4.0-112-generic
initrd.img-4.4.0-154-generic
initrd.img-4.4.0-161-generic
initrd.img-4.4.0-164-generic
initrd.img-4.4.0-178-generic
initrd.img-4.4.0-179-generic
initrd.img-4.4.0-184-generic
initrd.img-4.4.0-185-generic
initrd.img-4.4.0-186-generic
initrd.img-4.4.0-187-generic
initrd.img-4.4.0-189-generic.new

Finally, I tired dpkg --confiure -a and my dpkg is locked : dpkg: error: dpkg frontend is locked by another process . Using lsof and ps, I found it is used by root 1213 0.0 0.0 65512 3000 ? Ss 2019 37:23 /usr/sbin/sshd -D

So, what should I do? Remove every files that are not 4.4.0-154-generic? Can I kill this sshd process that is locking dpkg?

CarbonBee

@girish Well, this issue still happens and for both our servers. I just went to my AWS console to remove a backup made the 5th which were no more existing in the Cloudron dashboard (I have a retention of 1 week).

CarbonBee

@nebulon Well, as this is production servers, I'd rather not remove our daily backups.

CarbonBee

@nebulon It's tarball backup.

CarbonBee

Hi,

I just went to my aws dashboard and I was surprised by the size taken by my Cloudron backups. I told Cloudron to make a backup every day and have a retention of a week.
After further investigations, all my backups still appears in aws, although only the last seven days appears in Cloudron.
It seems that Cloudron just "forget" about backups that passed retention, but does not remove it from AWS.

CarbonBee

Well, a technician from Kimsufi just told me that there are no upload limit on my server. I don't really understand what's going on, because on more or less the same setup on a different server, it works fine.
Is there a way to track it down and find out where the issue come from?

CarbonBee

@girish I tried to backup only the nextcloud, which generated the same error.
I have no idea on any upload limitation. My server is hosted by Kimsufi, an OVH discounter.

CarbonBee

It's always doing that. I haven't been able to make a full backup for days. This begins to be critical.

CarbonBee

Hi!

I recently setup my Cloudron server to backup with aws s3 everyday. The last few days, the backup ended when trying to upload my nextcloud backup (which weights more than 100GB).
The logs report the following (for security purpose, I removed the domain name, the application ID and the aws id) :

Oct 20 03:29:26 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 392M@1MBps (cloud.mydomain.fr)"}
Oct 20 03:29:37 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 104M@0MBps (cloud.mydomain.fr)"}
Oct 20 03:30:41 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 392M@1MBps (cloud.mydomain.fr)"}
Oct 20 03:30:55 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 1404M@2MBps (cloud.mydomain.fr)"}
Oct 20 03:34:06 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 104M@0MBps (cloud.mydomain.fr)"}
Oct 20 03:34:15 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 877M@1MBps (cloud.mydomain.fr)"}
Oct 20 03:34:37 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 104M@0MBps (cloud.mydomain.fr)"}
Oct 20 03:37:31 box:shell backup-snapshot/app_<appId> (stdout): 2020-10-20T01:37:18.907Z box:storage/s3 Error uploading [snapshot/app_<appId>.tar.gz.enc]: s3 upload error. { Error: write EPIPE
at WriteWrap.afterWrite [as oncomplete] (net.js:789:14)
message: 'write EPIPE',
errno: 'EPIPE',
code: 'NetworkingError',
syscall: 'write',
region: 'eu-west-3',
hostname: '<s3Id>.s3.eu-west-3.amazonaws.com',
retryable: true,
time: 2020-10-20T01:36:48.195Z }
Oct 20 03:37:19 box:backupupload upload completed. error: { BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError
at ManagedUpload.callback (/home/yellowtent/box/src/storage/s3.js:130:33)
at ManagedUpload.cleanup (/home/yellowtent/box/node_modules/aws-sdk/lib/s3/managed_upload.js:629:10)
at Response.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/s3/managed_upload.js:566:28)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:364:18)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
name: 'BoxError',
reason: 'External Error',
details: {},
message:
'Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError' }
Oct 20 03:37:41 box:backups runBackupUpload: result - {"result":"Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError"}
Oct 20 03:39:26 box:shell backup-snapshot/app_<appId> code: 50, signal: null
Oct 20 03:39:30 box:backups cloud.mydomain.fr Unable to backup { BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError
at /home/yellowtent/box/src/backups.js:863:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
at ChildProcess.emit (events.js:198:13)
at ChildProcess.EventEmitter.emit (domain.js:448:20)
at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
name: 'BoxError',
reason: 'External Error',
details: {},
message:
'Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError' }
Oct 20 03:39:31 box:taskworker Task took 16766.54 seconds
Oct 20 03:39:31 box:tasks setCompleted - 5308: {"result":null,"error":{"stack":"BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at ChildProcess.EventEmitter.emit (domain.js:448:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError"}}
Oct 20 03:39:32 box:tasks 5308: {"percent":100,"result":null,"error":{"stack":"BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at ChildProcess.EventEmitter.emit (domain.js:448:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError"}}

I tried with 10MB and 100MB upload part size, but this does not seems to change anything.
Do you have any idea on what's going on?

Thanks!

CarbonBee

I patch the my.domainname.fr.conf with the missing part (taken from my other Cloudron server).
I then uninstalled and re-installed jingo, and everything seems to work fine now.

CarbonBee

Ok, it seems there is some issues in nginx conf files.
The application/my.domainname.fr.conf ends brutally on

# only serve up the status page if we get proxy gateway errors
root /home/yellowtent/box/das

And nginx tells me that

invalid port in upstream "127.0.0.1:"

for the jingo nginx conf.

Any idea on what's going on?

CarbonBee

Hi!

I'm trying to install jingo to my Cloudron server, but I've got the following error :
"Nginx Error: Error reloading nginx: reload exited with code 1 signal null"
I checked but everything seems right with my nginx.

Do you have any idea on why it's happening ?

Thanks

CarbonBee

Hi!

On the subscription invoices, could it be possible to add the cloudron ID or the domain name or anything that can identify a server from another? We have 3 servers, thus 3 invoices every month, and we can not which invoice goes with which server.

Thanks!

CarbonBee

@robi That would be interesting, but that does not solve my current problem, which is: "How to backup Cloudron on s3 glacier deep archive".

One solution would be to backup on a regular s3 and then use aws lifecycle rule to put them in glacier. But then, it would not be accessible through Glacier API, only s3's, and it would be costly to do the transition from s3 to Glacier.

In a near future, could it be possible to integrate Glacier API?

CarbonBee

@marcusquinn said in Best aws s3 backup storage class:

How much are you storing?

Relevent question indeed, I missed it sorry.
Well, my servers backups wieght 145Go and 190Go respectively. I backup them daily and I would like to have at least one week of backup history.
That means a bit more than 2To.

CarbonBee

@marcusquinn These are the basic s3 plan prices. The Glacier Deep Archive is 0.0018$/Go : https://aws.amazon.com/fr/s3/pricing/.
Moreover we will use this solution for other needs (which require aws) and I would like to use the same service for all our applications, which includes cloudron backups.

CarbonBee

@luckow I understand your point, which I find really interesting. However, my point of view is slightly different: I've been running Cloudron on 2 servers for more than 4 years now and I have never nedded backup restoration. The event is rare enough to cost half a day or more if it happens. The second point is the price. With Glacier Deep Archive, a To cost ~1$/month.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Posts made by CarbonBee