Feedback: Cloudron Update 6.2.4

luckow

One of a few Cloudron instances has a problem with mysql after the update. I lost some data and everytime I restore the app from a backup I ended up with "Error : Addons Error - Error restoring mysql: read ECONNRESET". Just saying

girish

@luckow Were you updating from 6.1.2 ? Did I understand correctly that one app out of all your Cloudron instances had an update issue? What app was this (built-in or custom?) ? Also, can you provide the last few lines of the logs of mysql service when you trigger a restore ?

luckow

@girish custom app (staging environment. here: drupal 9.x) It hits 2 from 9 stage apps. No clue why only 2

Mar 12 19:19:46 box:tasks 6843: {"percent":65,"message":"Download LALA.jpg finished"}
Mar 12 19:19:46 box:backups Recreating empty directories in {"localRoot":"/home/yellowtent/appsdata/7ef8d8c0-3c51-45c5-b0af-a98bcb43da98","layout":[]}
Mar 12 19:19:46 box:backups downloadApp: time: 11.72
Mar 12 19:19:46 box:services URL restoreAddons
Mar 12 19:19:46 box:services URL restoreAddons: restoring ["localstorage","mysql","sendmail"]
Mar 12 19:19:46 box:services URL restoreMySql
Mar 12 19:19:53 box:apptask URL error installing app: BoxError: Error restoring mysql: read ECONNRESET
at Request._callback (/home/yellowtent/box/src/services.js:1354:40)
at self.callback (/home/yellowtent/box/node_modules/request/request.js:185:22)
at Request.emit (events.js:327:22)
at Request.EventEmitter.emit (domain.js:467:12)
at Request.onRequestError (/home/yellowtent/box/node_modules/request/request.js:877:8)
at ClientRequest.emit (events.js:315:20)
at ClientRequest.EventEmitter.emit (domain.js:467:12)
at TLSSocket.socketErrorListener (_http_client.js:469:9)
at TLSSocket.emit (events.js:315:20)
at TLSSocket.EventEmitter.emit (domain.js:467:12) {
reason: 'Addons Error',
details: {}

6.1.2 -> 6.2.4

girish

@luckow I think this is just MySQL wanting more memory. If you go to services -> mysql -> logs, do you see it restarting? Sometimes importing dumps can take a lot lot of memory.

luckow

@girish nope. Same error with doubled ram. Problem is a Duplicate entry in my app backup.

Mar 12 22:03:57 restore: restoring database 9e21f4c46c93e8f9
Mar 12 22:04:05 Stderr from db import: ERROR 1062 (23000) at line 4144: Duplicate entry 'masse' for key 'search_total.PRIMARY'
Mar 12 22:04:05 events.js:292
Mar 12 22:04:05 throw er; // Unhandled 'error' event
Mar 12 22:04:05 ^
Mar 12 22:04:05 Error: write EPIPE
Mar 12 22:04:05 at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)
Mar 12 22:04:05 Emitted 'error' event on Socket instance at:
Mar 12 22:04:05 at Socket.onerror (internal/streams/readable.js:760:14)
Mar 12 22:04:05 at Socket.emit (events.js:315:20)
Mar 12 22:04:05 at emitErrorNT (internal/streams/destroy.js:106:8)
Mar 12 22:04:05 at emitErrorCloseNT (internal/streams/destroy.js:74:3)
Mar 12 22:04:05 at processTicksAndRejections (internal/process/task_queues.js:80:21) {
Mar 12 22:04:05 errno: -32,
Mar 12 22:04:05 code: 'EPIPE',
Mar 12 22:04:05 syscall: 'write'
Mar 12 22:04:05 }
Mar 12 22:04:05 2021-03-12 21:04:05,178 INFO exited: mysql-service (exit status 1; not expected)

girish

@luckow ah, i see. I guess this is a result of some mysql 8.0 change?

drpaneas

After updating my redis is complaining ...

any ideas? what I should do? This looks like a regression.

drpaneas

Also looking at the graphite logs:

Mar 12 23:07:05 12/03/2021 22:07:05 :: [tagdb] Error tagging collectd.localhost.table-0e95e7bb-2ffb-4b5c-af67-64484fe51e33-memory.gauge-cache: Error requesting http://127.0.0.1:80/tags/tagMultiSeries: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /tags/tagMultiSeries (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdc0f856af0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Mar 12 23:07:26 12/03/2021 22:07:26 :: [tagdb] Tagging collectd.localhost.table-e1ca318a-a940-477b-bb4f-43722b127b92-memory.gauge-hierarchical_memory_limit, collectd.localhost.table-d03dd73e-4045-42f7-8e6d-7b6e36b2caba-memory.gauge-total_active_file, collectd.localhost.table-9cea909a-e860-4a20-8d38-bd7820088c93-memory.gauge-inactive_anon, collectd.localhost.table-4d6310dd-104d-4eae-bdd2-025ad353bc45-memory.gauge-total_pgmajfault, collectd.localhost.table-42c605b3-e603-43a2-bfa7-f24fc238c899-memory.gauge-pgpgin, collectd.localhost.table-33db8df1-0687-4415-b5a2-d1ffc5a1339c-memory.gauge-total_swap
Mar 12 23:07:26 12/03/2021 22:07:26 :: [tagdb] Error tagging collectd.localhost.table-e1ca318a-a940-477b-bb4f-43722b127b92-memory.gauge-hierarchical_memory_limit, collectd.localhost.table-d03dd73e-4045-42f7-8e6d-7b6e36b2caba-memory.gauge-total_active_file, collectd.localhost.table-9cea909a-e860-4a20-8d38-bd7820088c93-memory.gauge-inactive_anon, collectd.localhost.table-4d6310dd-104d-4eae-bdd2-025ad353bc45-memory.gauge-total_pgmajfault, collectd.localhost.table-42c605b3-e603-43a2-bfa7-f24fc238c899-memory.gauge-pgpgin, collectd.localhost.table-33db8df1-0687-4415-b5a2-d1ffc5a1339c-memory.gauge-total_swap: Error requesting http://127.0.0.1:80/tags/tagMultiSeries: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /tags/tagMultiSeries (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdc0e772700>: Failed to establish a new connection: [Errno 111] Connection refused'))
Mar 12 23:07:26 12/03/2021 22:07:26 :: [tagdb] Tagging collectd.localhost.table-0ea62e64-2569-4095-942e-81f50c4c93a3-memory.gauge-shmem
Mar 12 23:07:26 12/03/2021 22:07:26 :: [tagdb] Error tagging collectd.localhost.table-0ea62e64-2569-4095-942e-81f50c4c93a3-memory.gauge-shmem: Error requesting http://127.0.0.1:80/tags/tagMultiSeries: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /tags/tagMultiSeries (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdc0e7b18e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Mar 12 23:07:26 12/03/2021 22:07:26 :: [tagdb] Tagging collectd.localhost.aggregation-cpu-max.cpu-system
Mar 12 23:07:26 12/03/2021 22:07:26 :: [tagdb] Error tagging collectd.localhost.aggregation-cpu-max.cpu-system: Error requesting http://127.0.0.1:80/tags/tagMultiSeries: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /tags/tagMultiSeries (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdc0e789b20>: Failed to establish a new connection: [Errno 111] Connection refused'))

and redis gets oom:

Mar 12 22:54:34 2021-03-12 21:54:34,993 INFO exited: redis (terminated by SIGKILL; not expected)
Mar 12 22:54:35 2021-03-12 21:54:35,003 INFO spawned: 'redis' with pid 577
Mar 12 22:54:35 2021-03-12 21:54:35,003 INFO reaped unknown pid 576
Mar 12 22:54:35 577:C 12 Mar 2021 21:54:35.037 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Mar 12 22:54:35 577:C 12 Mar 2021 21:54:35.037 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=577, just started
Mar 12 22:54:35 577:C 12 Mar 2021 21:54:35.037 # Configuration loaded
Mar 12 22:54:35 577:M 12 Mar 2021 21:54:35.039 * Running mode=standalone, port=6379.
Mar 12 22:54:35 577:M 12 Mar 2021 21:54:35.040 # Server initialized
Mar 12 22:54:35 577:M 12 Mar 2021 21:54:35.040 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Mar 12 22:54:35 577:M 12 Mar 2021 21:54:35.611 * DB loaded from disk: 0.571 seconds

girish

@drpaneas Just give that redis more memory. The "e1ca318a" is the app id. If you put in the search in the apps view, you will know which app's redis needs a bit more memory. It was reported earlier that pixelfed's redis needs more memory after the update. I guess this is because of update to Redis 5.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Feedback: Cloudron Update 6.2.4