Baserow running but not responding ?

[girish]

This means the healthcheck of baserow is responding but the app itself is not working. Mmm... anyone else hit this? Otherwise, we have to debug further.

[nebulon]

A fresh install here works. Are there any other suspicious logs when the app restarts?

[timconsidine]

Yes, the logs are quite vocal, so I cherry-picked what I thought was the issue.
Let me re-examine and maybe post more.
However may be a while as leaving on another medical aid trip for Ukraine.
Bear with me please.

[timconsidine]

Problem continuing
Restarted redis, increased memory for it
Still Baserow doesn't come up.

App Title and VersionBaserow 1.21.2
App IDa60be1d6-3f37-48ed-acee-ee3ad989b66f
Package Versionio.baserow.cloudronapp@1.15.2
Last Updated26/10/2023

Last update appears roughly time it started failing, but not sure.
I can't understand the logs (very vocal, following section repeats constantly):

Nov 23 23:38:45ℹ Listening on: http://localhost:3000/
Nov 23 23:38:47
Nov 23 23:38:47-------------- celery@a60be1d6-3f37-48ed-acee-ee3ad989b66f v5.2.7 (dawn-chorus)
Nov 23 23:38:47--- ***** -----
Nov 23 23:38:47-- ******* ---- Linux-5.15.0-89-generic-x86_64-with-glibc2.35 2023-11-23 23:38:47
Nov 23 23:38:47- *** --- * ---
Nov 23 23:38:47- ** ---------- [config]
Nov 23 23:38:47- ** ---------- .> app: baserow:0x7f0614b16710
Nov 23 23:38:47- ** ---------- .> transport: redis://:**@redis-a60be1d6-3f37-48ed-acee-ee3ad989b66f:6379//
Nov 23 23:38:47- ** ---------- .> results: disabled://
Nov 23 23:38:47- *** --- * --- .> concurrency: 16 (prefork)
Nov 23 23:38:47-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
Nov 23 23:38:47--- ***** -----
Nov 23 23:38:47-------------- [queues]
Nov 23 23:38:47.> celery exchange=celery(direct) key=celery
Nov 23 23:38:47
Nov 23 23:38:47
Nov 23 23:38:47[tasks]
Nov 23 23:38:47. baserow.contrib.database.export.tasks.clean_up_old_jobs
Nov 23 23:38:47. baserow.contrib.database.export.tasks.run_export_job
Nov 23 23:38:47. baserow.contrib.database.fields.tasks.run_periodic_fields_updates
Nov 23 23:38:47. baserow.contrib.database.rows.tasks.clean_up_row_history_entries
Nov 23 23:38:47. baserow.contrib.database.search.tasks.async_update_tsvector_columns
Nov 23 23:38:47. baserow.contrib.database.table.tasks.run_row_count_job
Nov 23 23:38:47. baserow.contrib.database.table.tasks.setup_new_background_update_and_search_columns
Nov 23 23:38:47. baserow.contrib.database.table.tasks.unsubscribe_user_from_tables_when_removed_from_workspace
Nov 23 23:38:47. baserow.contrib.database.views.tasks._check_for_pending_view_index_updates
Nov 23 23:38:47. baserow.contrib.database.views.tasks.update_view_index
Nov 23 23:38:47. baserow.contrib.database.webhooks.tasks.call_webhook
Nov 23 23:38:47. baserow.core.action.tasks.cleanup_old_actions
Nov 23 23:38:47. baserow.core.jobs.tasks.clean_up_jobs
Nov 23 23:38:47. baserow.core.jobs.tasks.run_async_job
Nov 23 23:38:47. baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email
Nov 23 23:38:47. baserow.core.notifications.tasks.send_daily_and_weekly_notifications_summary_by_email
Nov 23 23:38:47. baserow.core.notifications.tasks.send_queued_notifications_to_users
Nov 23 23:38:47. baserow.core.notifications.tasks.singleton_send_instant_notifications_summary_by_email
Nov 23 23:38:47. baserow.core.snapshots.tasks.delete_application_snapshot
Nov 23 23:38:47. baserow.core.snapshots.tasks.delete_expired_snapshots
Nov 23 23:38:47. baserow.core.tasks.sync_templates_task
Nov 23 23:38:47. baserow.core.trash.tasks.mark_old_trash_for_permanent_deletion
Nov 23 23:38:47. baserow.core.trash.tasks.permanently_delete_marked_trash
Nov 23 23:38:47. baserow.core.usage.tasks.run_calculate_storage
Nov 23 23:38:47. baserow.core.user.tasks.check_pending_account_deletion
Nov 23 23:38:47. baserow.core.user.tasks.flush_expired_tokens
Nov 23 23:38:47. baserow.ws.tasks.broadcast_application_created
Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_channel_group
Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_group
Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_groups
Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_permitted_users
Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_users
Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_users_individual_payloads
Nov 23 23:38:47. baserow.ws.tasks.force_disconnect_users
Nov 23 23:38:47. baserow_enterprise.audit_log.tasks.clean_up_audit_log_entries
Nov 23 23:38:47. baserow_enterprise.tasks.unsubscribe_subject_from_tables_currently_subscribed_to_task
Nov 23 23:38:47. baserow_premium.license.tasks.license_check
Nov 23 23:38:47. baserow_premium.usage.tasks.run_calculate_seats
Nov 23 23:38:47. djcelery_email_send_multiple
Nov 23 23:38:47
Nov 23 23:38:47[2023-11-23 23:38:47,727: WARNING/ForkPoolWorker-1] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:47[2023-11-23 23:38:47,732: WARNING/ForkPoolWorker-1] 285743|2023-11-23 23:38:47.732|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:47[2023-11-23 23:38:47,826: WARNING/ForkPoolWorker-2] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:47[2023-11-23 23:38:47,831: WARNING/ForkPoolWorker-2] 285744|2023-11-23 23:38:47.831|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:47[2023-11-23 23:38:47,906: WARNING/ForkPoolWorker-3] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:47[2023-11-23 23:38:47,911: WARNING/ForkPoolWorker-3] 285745|2023-11-23 23:38:47.911|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,029: WARNING/ForkPoolWorker-4] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,035: WARNING/ForkPoolWorker-4] 285746|2023-11-23 23:38:48.035|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,123: WARNING/ForkPoolWorker-5] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,128: WARNING/ForkPoolWorker-5] 285747|2023-11-23 23:38:48.128|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,225: WARNING/ForkPoolWorker-6] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,231: WARNING/ForkPoolWorker-6] 285748|2023-11-23 23:38:48.231|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,323: WARNING/ForkPoolWorker-7] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,328: WARNING/ForkPoolWorker-7] 285749|2023-11-23 23:38:48.328|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,431: WARNING/ForkPoolWorker-8] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,438: WARNING/ForkPoolWorker-8] 285750|2023-11-23 23:38:48.438|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,569: WARNING/ForkPoolWorker-9] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:482023-11-23 23:38:48,569 INFO waiting for beatworker, exportworker, gunicorn, nginx, nuxt, worker to die
Nov 23 23:38:48[2023-11-23 23:38:48,575: WARNING/ForkPoolWorker-9] 285751|2023-11-23 23:38:48.574|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,741: WARNING/ForkPoolWorker-10] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,747: WARNING/ForkPoolWorker-10] 285752|2023-11-23 23:38:48.747|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:48[2023-11-23 23:38:48,912: WARNING/ForkPoolWorker-11] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:48[2023-11-23 23:38:48,917: WARNING/ForkPoolWorker-11] 285753|2023-11-23 23:38:48.917|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:49[2023-11-23 23:38:49,079: WARNING/ForkPoolWorker-12] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:49[2023-11-23 23:38:49,085: WARNING/ForkPoolWorker-12] 285754|2023-11-23 23:38:49.085|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:49[2023-11-23 23:38:49,243: WARNING/ForkPoolWorker-13] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:49[2023-11-23 23:38:49,249: WARNING/ForkPoolWorker-13] 285755|2023-11-23 23:38:49.249|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:49[2023-11-23 23:38:49,424: WARNING/ForkPoolWorker-14] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:49[2023-11-23 23:38:49,429: WARNING/ForkPoolWorker-14] 285756|2023-11-23 23:38:49.429|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:492023-11-23 23:38:49,474 INFO exited: nuxt (terminated by SIGKILL; not expected)
Nov 23 23:38:49[2023-11-23 23:38:49,602: WARNING/ForkPoolWorker-15] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:49[2023-11-23 23:38:49,608: WARNING/ForkPoolWorker-15] 285757|2023-11-23 23:38:49.608|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:49[2023-11-23 23:38:49,662: INFO/MainProcess] Connected to redis://:**@redis-a60be1d6-3f37-48ed-acee-ee3ad989b66f:6379//
Nov 23 23:38:49[2023-11-23 23:38:49,664: INFO/MainProcess] mingle: searching for neighbors
Nov 23 23:38:49[2023-11-23 23:38:49,703: WARNING/ForkPoolWorker-16] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set.
Nov 23 23:38:49[2023-11-23 23:38:49,708: WARNING/ForkPoolWorker-16] 285758|2023-11-23 23:38:49.708|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup.
Nov 23 23:38:50[2023-11-23 23:38:50,677: INFO/MainProcess] mingle: all alone
Nov 23 23:38:50[2023-11-23 23:38:50,693: INFO/MainProcess] celery@a60be1d6-3f37-48ed-acee-ee3ad989b66f ready.

[timconsidine]

Just tried a fresh install as a dummy instance - same problem.

[nebulon]

Tried to reproduce this on two Cloudrons but it seems to work here. Can you put the app in debug/recovery mode and open a webterminal into it and then run /app/pkg/start.sh manually?

[timconsidine]

Put into recovery and tried to start manually.
Same results.
Tried to disable recovery mode
Got Error : Inactive - Error getting IP of redis-c97adab3-f132-462b-a597-ad570120a92d service
Restarted redis and repeated.
Redis staying up for now, but still get app not responding

[timconsidine]

Might be an unbound issue :

root@c97adab3-f132-462b-a597-ad570120a92d:/app/code# systemctl status unbound
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

Rebooting ...
Rebooted
No change - same behaviour - all services showing green dot including redis
Same unbound response in problem app
Same unbound response in active running apps

[timconsidine]

I'm still on Cloudron 7.5.2
Haven't been brave enough to try 7.6.x
Will it make a difference ?

[nebulon]

I don't think this is unbound related in this case. The "Error getting IP of" error might indicate that Cloudron can't get the IP address of the docker container. That is obtained by querying docker. Can you check if that redis docker container is actually up? Try this with docker ps -a look for the redis-container and then a docker inspect <container>

[timconsidine]

Yes, all services running, including all redis.
Mysteriously suddenly got 'out of memory' notices for these Baserow instances.
Not happened before - seems to be triggered by going into recovery / restarting.
Have bumped memory from 3Gb to 5Gb
All instances now working : 2 live and 1 test (to be removed)
Wheee

I didn't think to increase memory as there were no reports about it, and 3GB seems ample.
Lesson for the future : bump up memory before doing anything else (even if not alerted) !

Thanks for your help @nebulon !

[makemrproper]

I was getting this. The default memory setting was not enough. Increased to 6gb, and it runs like a BMW.