ECONNREFUSED

MarchinBunny

Yes, I am sure I am looking at the right log. As for my exact issue, the app will no longer start. It just says "Not Responding" and in the logs it just keep repeating that same healthcheck error.

I have been running the website for years and never had a problem with it up till this point. As I said, I noticed things felt a bit sluggish and so I looked and noticed some out of memory messages that seems to have only started occurring about a week and a half ago. 7/12/2023 is the first time it appears under Notifications. So, I tried increasing the memory limit. And as soon as I did that, I could no longer start it up anymore and the error keeps repeating in the logs "Healtheck error: Error: connect ECONNREFUSED 172.xx.xx.xx:80" (I put the x's there to hide the full address)

Also, I have not made any changes outside of that memory limit. Which I tried to put back to what it was prior and didn't help. I also don't see any other errors. And again, I tried backups as well... and even that's not working.

nebulon

If you follow the app logs (not system logs) in the logviewer and then restart the app, it should reveal more error details as the app is starting up.

girish

@MarchinBunny https://docs.cloudron.io/apps/#log-viewer is the app log viewer. If you restart the app (in the repair section), the log viewer will show the latest logs as to why it's not starting.

MarchinBunny

@nebulon said in ECONNREFUSED:

If you follow the app logs (not system logs) in the logviewer and then restart the app, it should reveal more error details as the app is starting up.

Ya, that's where I have been looking. Here is what the complete logs look like as it's starting. I see no other error. That error toward the bottom just repeats infinitely till I turn the app off. Also, I changed out some information to "x"s. Not sure what to keep private so I just did it with anything that seemed important. Though might have been unnecessary as it look encrypted but hey, better safe than sorry.

Anyway, let me know if you see something I don't. Cause that's the only error I see.

Jul 24 12:48:17 box:settings initCache: pre-load settings
Jul 24 12:48:17 box:taskworker Starting task 9461. Logs are at /home/yellowtent/platformdata/logs/35c7b265-e93e-4fb1-bb62-54ee4de0269d/apptask.log
Jul 24 12:48:17 box:apptask run: startTask installationState: pending_start runState: running
Jul 24 12:48:17 box:tasks update 9461: {"percent":10,"message":"Starting app services"}
Jul 24 12:48:17 Starting supervisor
Jul 24 12:48:17 box:tasks update 9461: {"percent":35,"message":"Starting container"}
Jul 24 12:48:18 2023-07-24 16:48:18,220 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
Jul 24 12:48:18 2023-07-24 16:48:18,220 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
Jul 24 12:48:18 2023-07-24 16:48:18,220 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
Jul 24 12:48:18 2023-07-24 16:48:18,232 INFO RPC interface 'supervisor' initialized
Jul 24 12:48:18 2023-07-24 16:48:18,232 CRIT Server 'inet_http_server' running without any HTTP authentication checking
Jul 24 12:48:18 2023-07-24 16:48:18,232 INFO RPC interface 'supervisor' initialized
Jul 24 12:48:18 2023-07-24 16:48:18,232 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Jul 24 12:48:18 2023-07-24 16:48:18,232 INFO supervisord started with pid 1
Jul 24 12:48:18 box:tasks update 9461: {"percent":80,"message":"Configuring reverse proxy"}
Jul 24 12:48:18 => Changing permissions
Jul 24 12:48:18 box:reverseproxy providerMatchesSync: subject=CN = thatmarchingbunny.com domain=thatmarchingbunny.com issuer=C = US, O = Let's Encrypt, CN = R3 wildcard=false/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Jul 24 12:48:18 box:reverseproxy expiryDate: subject=CN = thatmarchingbunny.com notBefore=May 28 21:10:28 2023 GMT notAfter=Aug 26 21:10:27 2023 GMT daysLeft=33.18204351851852
Jul 24 12:48:18 box:reverseproxy needsRenewal: false. force: false
Jul 24 12:48:18 box:reverseproxy ensureCertificate: thatmarchingbunny.com acme cert exists and is up to date
Jul 24 12:48:18 box:reverseproxy writeAppLocationNginxConfig: writing config for "thatmarchingbunny.com" to /home/yellowtent/platformdata/nginx/applications/35c7b265-e93e-4fb1-bb62-54ee4de0269d/thatmarchingbunny.com.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"thatmarchingbunny.com","hasIPv6":true,"ip":"172.xx.xx.xx","port":80,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/thatmarchingbunny.com.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/thatmarchingbunny.com.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"35c7b265-e93e-4fb1-bb62-54ee4de0269d","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Jul 24 12:48:18 box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Jul 24 12:48:18 box:tasks update 9461: {"percent":100,"message":"Done"}
Jul 24 12:48:18 box:taskworker Task took 1.78 seconds
Jul 24 12:48:18 box:tasks setCompleted - 9461: {"result":null,"error":null}
Jul 24 12:48:18 box:tasks update 9461: {"percent":100,"result":null,"error":null}
Jul 24 12:48:19 => Updating db settings
Jul 24 12:48:19 2023-07-24 16:48:19,235 INFO spawned: 'redis' with pid 11
Jul 24 12:48:19 2023-07-24 16:48:19,241 INFO spawned: 'redis-service' with pid 12
Jul 24 12:48:19 11:C 24 Jul 2023 16:48:19.270 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Jul 24 12:48:19 11:C 24 Jul 2023 16:48:19.271 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=11, just started
Jul 24 12:48:19 11:C 24 Jul 2023 16:48:19.272 # Configuration loaded
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.277 * Running mode=standalone, port=6379.
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.278 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.280 # Server initialized
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.281 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.283 * Loading RDB produced by version 6.0.16
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.284 * RDB age 67328 seconds
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.285 * RDB memory usage when created 0.78 Mb
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.286 * DB loaded from disk: 0.003 seconds
Jul 24 12:48:19 11:M 24 Jul 2023 16:48:19.287 * Ready to accept connections
Jul 24 12:48:19 Redis service endpoint listening on http://:::3000
Jul 24 12:48:19 Success: Updated the constant 'DB_HOST' in the 'wp-config.php' file with the value 'mysql:3306'.
Jul 24 12:48:19 Success: Updated the constant 'DB_NAME' in the 'wp-config.php' file with the value 'xxxxxxxxxxxxxxxx'.
Jul 24 12:48:20 => Healtheck error: Error: connect ECONNREFUSED 172.xx.xx.xx:80
Jul 24 12:48:20 Success: Updated the constant 'DB_USER' in the 'wp-config.php' file with the value 'xxxxxxxxxxxxxxxx'.
Jul 24 12:48:20 Success: Updated the constant 'DB_PASSWORD' in the 'wp-config.php' file with the value 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'.
Jul 24 12:48:20 => Updating redis settings
Jul 24 12:48:20 Success: Updated the constant 'WP_REDIS_HOST' in the 'wp-config.php' file with the value 'redis-35c7b265-e93e-4fb1-bb62-54ee4de0269d'.
Jul 24 12:48:20 Success: Updated the constant 'WP_REDIS_PORT' in the 'wp-config.php' file with the value '6379'.
Jul 24 12:48:20 Success: Updated the constant 'WP_REDIS_PASSWORD' in the 'wp-config.php' file with the value 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'.
Jul 24 12:48:20 2023-07-24 16:48:20,726 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 24 12:48:20 2023-07-24 16:48:20,728 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Jul 24 12:48:30 => Healtheck error: Error: connect ECONNREFUSED 172.xx.xx.xx:80
Jul 24 12:48:40 => Healtheck error: Error: connect ECONNREFUSED 172.xx.xx.xx:80
Jul 24 12:48:50 => Healtheck error: Error: connect ECONNREFUSED 172.xx.xx.xx:80

girish

@MarchinBunny it seems the app is getting stuck on start up. Can you please try this:

Put the app in recovery mode. You do this by Repair -> Enable Recovery Mode
Then, open a web terminal. Run /app/pkg/start.sh . Looks like it is getting stuck after updating redis configuration.
If above gets stuck ^C it and then can you run sudo -u www-data -i -- /app/pkg/wp --path=/app/data/public/ --skip-themes --skip-plugins core is-installed ? Does it hang?

MarchinBunny

@girish said in ECONNREFUSED:

@MarchinBunny it seems the app is getting stuck on start up. Can you please try this:

Put the app in recovery mode. You do this by Repair -> Enable Recovery Mode

Then, open a web terminal. Run /app/pkg/start.sh . Looks like it is getting stuck after updating redis configuration.

If above gets stuck ^C it and then can you run sudo -u www-data -i -- /app/pkg/wp --path=/app/data/public/ --skip-themes --skip-plugins core is-installed ? Does it hang?

It still hangs, though I am not sure if sudo -u www-data -i -- /app/pkg/wp --path=/app/data/public/ --skip-themes --skip-plugins core is-installed is doing anything.

micmc

@girish seems to me it is the redis server that is NOT loading correctly and it seems to be because of a lack of memory. At least two warning and ECONNREFUSED errors appear after the

Running mode=standalone, port=6379

line and when you look at the wp-config.php lines which contain the db connect credentials you see those lines are all reported as Success:

girish

@MarchinBunny I think you will have to contact us at support@cloudron.io , so we can debug further. It's not clear why it hangs. I think something WordPress related.

MarchinBunny

@girish said in ECONNREFUSED:

@MarchinBunny I think you will have to contact us at support@cloudron.io , so we can debug further. It's not clear why it hangs. I think something WordPress related.

I sent a support message through the cloudron interface, will that work?

girish

@MarchinBunny yes, that should work. But I have to say we haven't gotten any WP related ticket today atleast.

MarchinBunny

@girish said in ECONNREFUSED:

@MarchinBunny yes, that should work. But I have to say we haven't gotten any WP related ticket today atleast.

I will try again.

girish

@MarchinBunny there was a malware in WordPress. Something like https://stackoverflow.com/questions/76268370/wordpress-error-because-mo-files-inside-wp-include .

I see you are using the elementor plugin. A few months back there was a hack - https://www.bleepingcomputer.com/news/security/hackers-exploit-bug-in-elementor-pro-wordpress-plugin/ and https://www.bleepingcomputer.com/news/security/hackers-target-vulnerable-wordpress-elementor-plugin-after-poc-released/ . Maybe that one.. But there are lots of plugins in your site, it could be anything

girish

I recommend exporting all data (I managed to get the site running) and reimporting it into a fresh WP install. Do not import any code from your old installation. Only import data.

micmc

@girish said in ECONNREFUSED:

@MarchinBunny there was a malware in WordPress. Something like https://stackoverflow.com/questions/76268370/wordpress-error-because-mo-files-inside-wp-include .

Exactly, what I expected right from the start of the thread.

But there are lots of plugins in your site, it could be anything.

And, yes some malware could be running in the background for years before one find out.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

ECONNREFUSED