Baserow running but not responding ?
Solved
Baserow
-
This means the healthcheck of baserow is responding but the app itself is not working. Mmm... anyone else hit this? Otherwise, we have to debug further.
-
Yes, the logs are quite vocal, so I cherry-picked what I thought was the issue.
Let me re-examine and maybe post more.
However may be a while as leaving on another medical aid trip for Ukraine.
Bear with me please. -
Problem continuing
Restarted redis, increased memory for it
Still Baserow doesn't come up.App Title and VersionBaserow 1.21.2 App IDa60be1d6-3f37-48ed-acee-ee3ad989b66f Package Versionio.baserow.cloudronapp@1.15.2 Last Updated26/10/2023Last update appears roughly time it started failing, but not sure.
I can't understand the logs (very vocal, following section repeats constantly):Nov 23 23:38:45ℹ Listening on: http://localhost:3000/ Nov 23 23:38:47 Nov 23 23:38:47-------------- celery@a60be1d6-3f37-48ed-acee-ee3ad989b66f v5.2.7 (dawn-chorus) Nov 23 23:38:47--- ***** ----- Nov 23 23:38:47-- ******* ---- Linux-5.15.0-89-generic-x86_64-with-glibc2.35 2023-11-23 23:38:47 Nov 23 23:38:47- *** --- * --- Nov 23 23:38:47- ** ---------- [config] Nov 23 23:38:47- ** ---------- .> app: baserow:0x7f0614b16710 Nov 23 23:38:47- ** ---------- .> transport: redis://:**@redis-a60be1d6-3f37-48ed-acee-ee3ad989b66f:6379// Nov 23 23:38:47- ** ---------- .> results: disabled:// Nov 23 23:38:47- *** --- * --- .> concurrency: 16 (prefork) Nov 23 23:38:47-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker) Nov 23 23:38:47--- ***** ----- Nov 23 23:38:47-------------- [queues] Nov 23 23:38:47.> celery exchange=celery(direct) key=celery Nov 23 23:38:47 Nov 23 23:38:47 Nov 23 23:38:47[tasks] Nov 23 23:38:47. baserow.contrib.database.export.tasks.clean_up_old_jobs Nov 23 23:38:47. baserow.contrib.database.export.tasks.run_export_job Nov 23 23:38:47. baserow.contrib.database.fields.tasks.run_periodic_fields_updates Nov 23 23:38:47. baserow.contrib.database.rows.tasks.clean_up_row_history_entries Nov 23 23:38:47. baserow.contrib.database.search.tasks.async_update_tsvector_columns Nov 23 23:38:47. baserow.contrib.database.table.tasks.run_row_count_job Nov 23 23:38:47. baserow.contrib.database.table.tasks.setup_new_background_update_and_search_columns Nov 23 23:38:47. baserow.contrib.database.table.tasks.unsubscribe_user_from_tables_when_removed_from_workspace Nov 23 23:38:47. baserow.contrib.database.views.tasks._check_for_pending_view_index_updates Nov 23 23:38:47. baserow.contrib.database.views.tasks.update_view_index Nov 23 23:38:47. baserow.contrib.database.webhooks.tasks.call_webhook Nov 23 23:38:47. baserow.core.action.tasks.cleanup_old_actions Nov 23 23:38:47. baserow.core.jobs.tasks.clean_up_jobs Nov 23 23:38:47. baserow.core.jobs.tasks.run_async_job Nov 23 23:38:47. baserow.core.notifications.tasks.beat_send_instant_notifications_summary_by_email Nov 23 23:38:47. baserow.core.notifications.tasks.send_daily_and_weekly_notifications_summary_by_email Nov 23 23:38:47. baserow.core.notifications.tasks.send_queued_notifications_to_users Nov 23 23:38:47. baserow.core.notifications.tasks.singleton_send_instant_notifications_summary_by_email Nov 23 23:38:47. baserow.core.snapshots.tasks.delete_application_snapshot Nov 23 23:38:47. baserow.core.snapshots.tasks.delete_expired_snapshots Nov 23 23:38:47. baserow.core.tasks.sync_templates_task Nov 23 23:38:47. baserow.core.trash.tasks.mark_old_trash_for_permanent_deletion Nov 23 23:38:47. baserow.core.trash.tasks.permanently_delete_marked_trash Nov 23 23:38:47. baserow.core.usage.tasks.run_calculate_storage Nov 23 23:38:47. baserow.core.user.tasks.check_pending_account_deletion Nov 23 23:38:47. baserow.core.user.tasks.flush_expired_tokens Nov 23 23:38:47. baserow.ws.tasks.broadcast_application_created Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_channel_group Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_group Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_groups Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_permitted_users Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_users Nov 23 23:38:47. baserow.ws.tasks.broadcast_to_users_individual_payloads Nov 23 23:38:47. baserow.ws.tasks.force_disconnect_users Nov 23 23:38:47. baserow_enterprise.audit_log.tasks.clean_up_audit_log_entries Nov 23 23:38:47. baserow_enterprise.tasks.unsubscribe_subject_from_tables_currently_subscribed_to_task Nov 23 23:38:47. baserow_premium.license.tasks.license_check Nov 23 23:38:47. baserow_premium.usage.tasks.run_calculate_seats Nov 23 23:38:47. djcelery_email_send_multiple Nov 23 23:38:47 Nov 23 23:38:47[2023-11-23 23:38:47,727: WARNING/ForkPoolWorker-1] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:47[2023-11-23 23:38:47,732: WARNING/ForkPoolWorker-1] 285743|2023-11-23 23:38:47.732|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:47[2023-11-23 23:38:47,826: WARNING/ForkPoolWorker-2] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:47[2023-11-23 23:38:47,831: WARNING/ForkPoolWorker-2] 285744|2023-11-23 23:38:47.831|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:47[2023-11-23 23:38:47,906: WARNING/ForkPoolWorker-3] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:47[2023-11-23 23:38:47,911: WARNING/ForkPoolWorker-3] 285745|2023-11-23 23:38:47.911|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,029: WARNING/ForkPoolWorker-4] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,035: WARNING/ForkPoolWorker-4] 285746|2023-11-23 23:38:48.035|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,123: WARNING/ForkPoolWorker-5] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,128: WARNING/ForkPoolWorker-5] 285747|2023-11-23 23:38:48.128|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,225: WARNING/ForkPoolWorker-6] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,231: WARNING/ForkPoolWorker-6] 285748|2023-11-23 23:38:48.231|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,323: WARNING/ForkPoolWorker-7] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,328: WARNING/ForkPoolWorker-7] 285749|2023-11-23 23:38:48.328|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,431: WARNING/ForkPoolWorker-8] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,438: WARNING/ForkPoolWorker-8] 285750|2023-11-23 23:38:48.438|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,569: WARNING/ForkPoolWorker-9] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:482023-11-23 23:38:48,569 INFO waiting for beatworker, exportworker, gunicorn, nginx, nuxt, worker to die Nov 23 23:38:48[2023-11-23 23:38:48,575: WARNING/ForkPoolWorker-9] 285751|2023-11-23 23:38:48.574|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,741: WARNING/ForkPoolWorker-10] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,747: WARNING/ForkPoolWorker-10] 285752|2023-11-23 23:38:48.747|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:48[2023-11-23 23:38:48,912: WARNING/ForkPoolWorker-11] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:48[2023-11-23 23:38:48,917: WARNING/ForkPoolWorker-11] 285753|2023-11-23 23:38:48.917|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:49[2023-11-23 23:38:49,079: WARNING/ForkPoolWorker-12] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:49[2023-11-23 23:38:49,085: WARNING/ForkPoolWorker-12] 285754|2023-11-23 23:38:49.085|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:49[2023-11-23 23:38:49,243: WARNING/ForkPoolWorker-13] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:49[2023-11-23 23:38:49,249: WARNING/ForkPoolWorker-13] 285755|2023-11-23 23:38:49.249|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:49[2023-11-23 23:38:49,424: WARNING/ForkPoolWorker-14] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:49[2023-11-23 23:38:49,429: WARNING/ForkPoolWorker-14] 285756|2023-11-23 23:38:49.429|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:492023-11-23 23:38:49,474 INFO exited: nuxt (terminated by SIGKILL; not expected) Nov 23 23:38:49[2023-11-23 23:38:49,602: WARNING/ForkPoolWorker-15] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:49[2023-11-23 23:38:49,608: WARNING/ForkPoolWorker-15] 285757|2023-11-23 23:38:49.608|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:49[2023-11-23 23:38:49,662: INFO/MainProcess] Connected to redis://:**@redis-a60be1d6-3f37-48ed-acee-ee3ad989b66f:6379// Nov 23 23:38:49[2023-11-23 23:38:49,664: INFO/MainProcess] mingle: searching for neighbors Nov 23 23:38:49[2023-11-23 23:38:49,703: WARNING/ForkPoolWorker-16] Not configuring telemetry due to BASEROW_ENABLE_OTEL not being set. Nov 23 23:38:49[2023-11-23 23:38:49,708: WARNING/ForkPoolWorker-16] 285758|2023-11-23 23:38:49.708|INFO|baserow.core.telemetry.telemetry:setup_logging:72 - Logger setup. Nov 23 23:38:50[2023-11-23 23:38:50,677: INFO/MainProcess] mingle: all alone Nov 23 23:38:50[2023-11-23 23:38:50,693: INFO/MainProcess] celery@a60be1d6-3f37-48ed-acee-ee3ad989b66f ready. -
Just tried a fresh install as a dummy instance - same problem.
-
N nebulon marked this topic as a question on
-
Put into recovery and tried to start manually.
Same results.
Tried to disable recovery mode
GotError : Inactive - Error getting IP of redis-c97adab3-f132-462b-a597-ad570120a92d service
Restarted redis and repeated.
Redis staying up for now, but still getapp not responding -
Might be an unbound issue :
root@c97adab3-f132-462b-a597-ad570120a92d:/app/code# systemctl status unbound System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is downRebooting ...
Rebooted
No change - same behaviour - all services showing green dot including redis
Same unbound response in problem app
Same unbound response in active running apps -
I'm still on Cloudron 7.5.2
Haven't been brave enough to try 7.6.x
Will it make a difference ? -
I don't think this is unbound related in this case. The "Error getting IP of" error might indicate that Cloudron can't get the IP address of the docker container. That is obtained by querying docker. Can you check if that redis docker container is actually up? Try this with
docker ps -alook for the redis-container and then adocker inspect <container> -
Yes, all services running, including all redis.
Mysteriously suddenly got 'out of memory' notices for these Baserow instances.
Not happened before - seems to be triggered by going into recovery / restarting.
Have bumped memory from 3Gb to 5Gb
All instances now working : 2 live and 1 test (to be removed)
Wheee
I didn't think to increase memory as there were no reports about it, and 3GB seems ample.
Lesson for the future : bump up memory before doing anything else (even if not alerted) !Thanks for your help @nebulon !
-
T timconsidine has marked this topic as solved on
-
I was getting this. The default memory setting was not enough. Increased to 6gb, and it runs like a BMW.
