Issues with Guacamole and Open Project

[gnulab]

Hi,

my cloudron hosts only 2 apps, guacamole and Open Project.

Their usage is very light, maximum concurrent users only 1-2 at any point of time.

Now all of a sudden my guacamole can't work.

Nov 16 17:38:26 guacd[16]: INFO: Creating new client for protocol "ssh"
Nov 16 17:38:26 guacd[16]: INFO: Connection ID is "$81311092-340e-41c0-896a-70355c781e98"
Nov 16 17:38:26 guacd[454]: DEBUG: Processing instruction: size
Nov 16 17:38:26 guacd[454]: DEBUG: Processing instruction: audio
Nov 16 17:38:26 guacd[454]: DEBUG: Processing instruction: video
Nov 16 17:38:26 guacd[454]: DEBUG: Processing instruction: image
Nov 16 17:38:26 guacd[454]: DEBUG: Processing instruction: timezone
Nov 16 17:38:26 guacd[454]: DEBUG: Processing instruction: name
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "scrollback" omitted. Using default value of 1000.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "font-name" omitted. Using default value of "monospace".
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "enable-sftp" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "sftp-root-directory" omitted. Using default value of "/".
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "sftp-disable-download" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "sftp-disable-upload" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "read-only" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "typescript-name" omitted. Using default value of "typescript".
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "create-typescript-path" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "recording-name" omitted. Using default value of "recording".
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "recording-exclude-output" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "recording-exclude-mouse" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "recording-include-keys" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "create-recording-path" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "server-alive-interval" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "backspace" omitted. Using default value of 127.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "terminal-type" omitted. Using default value of "linux".
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "timezone" omitted. Using default value of "Asia/Bangkok".
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "disable-copy" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "disable-paste" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: DEBUG: Parameter "wol-send-packet" omitted. Using default value of 0.
Nov 16 17:38:26 guacd[454]: INFO: User "@4e64e1aa-c52f-4b18-8e5b-6d55e2c7253a" joined connection "$81311092-340e-41c0-896a-70355c781e98" (1 users now present)
Nov 16 17:38:26 guacd[454]: DEBUG: Client is using protocol version "VERSION_1_5_0"
Nov 16 17:38:26 10:38:26.890 [http-nio-8080-exec-5] INFO o.a.g.tunnel.TunnelRequestService - User "gnulab" connected to connection "7".
Nov 16 17:38:36 guacd[454]: ERROR: Error parsing given address or port: Temporary failure in name resolution
Nov 16 17:38:36 guacd[454]: INFO: User "@4e64e1aa-c52f-4b18-8e5b-6d55e2c7253a" disconnected (0 users remain)
Nov 16 17:38:36 guacd[454]: INFO: Last user of connection "$81311092-340e-41c0-896a-70355c781e98" disconnected
Nov 16 17:38:36 guacd[454]: DEBUG: Requesting termination of client...
Nov 16 17:38:36 Exception in thread "Thread-44" java.lang.IllegalStateException: Message will not be sent because the WebSocket session has been closed
Nov 16 17:38:36 at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:442)
Nov 16 17:38:36 at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:314)
Nov 16 17:38:36 at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:254)
Nov 16 17:38:36 at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendString(WsRemoteEndpointImplBase.java:195)
Nov 16 17:38:36 at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:37)
Nov 16 17:38:36 at org.apache.guacamole.websocket.GuacamoleWebSocketTunnelEndpoint.sendInstruction(GuacamoleWebSocketTunnelEndpoint.java:152)
Nov 16 17:38:36 at org.apache.guacamole.websocket.GuacamoleWebSocketTunnelEndpoint.access$200(GuacamoleWebSocketTunnelEndpoint.java:53)
Nov 16 17:38:36 at org.apache.guacamole.websocket.GuacamoleWebSocketTunnelEndpoint$2.run(GuacamoleWebSocketTunnelEndpoint.java:253)
Nov 16 17:38:37 10:38:36.997 [http-nio-8080-exec-9] INFO o.a.g.tunnel.TunnelRequestService - User "gnulab" disconnected from connection "7". Duration: 10106 milliseconds
Nov 16 17:38:37 guacd[454]: DEBUG: Client terminated successfully.
Nov 16 17:38:37 guacd[16]: INFO: Connection "$81311092-340e-41c0-896a-70355c781e98" removed.

Open Project couldn't be launched too.

Nov 14 18:23:52 => Seed database if needed
basic data for standard edition
Nov 14 18:24:12 *** Seeding for locale: 'en'
Nov 14 18:24:15 *** Loading action_cable_engine seed data
Nov 14 18:24:15 *** Loading action_mailbox seed data
Nov 14 18:24:15 *** Loading action_text seed data
Nov 14 18:24:15 *** Loading action_view_railtie seed data
Nov 14 18:24:15 *** Loading active_storage seed data
Nov 14 18:24:15 *** Loading budgets_engine seed data
Nov 14 18:24:15 *** Loading costs seed data
Nov 14 18:24:15 *** Loading dashboards seed data
Nov 14 18:24:15 *** Loading date_validator_engine seed data
Nov 14 18:24:15 *** Loading doorkeeper_engine seed data
Nov 14 18:24:15 *** Loading grids_engine seed data
Nov 14 18:24:15 *** Loading lookbook seed data
Nov 14 18:24:15 *** Loading my_page_engine seed data
Nov 14 18:24:15 *** Loading ok_computer seed data
Nov 14 18:24:15 *** Loading openproject_auth_plugins seed data
Nov 14 18:24:15 *** Loading openproject_auth_saml seed data
Nov 14 18:24:15 *** Loading openproject_avatars seed data
Nov 14 18:24:15 *** Loading openproject_backlogs seed data
Nov 14 18:24:15 *** Loading openproject_bim seed data
Nov 14 18:24:15 *** Loading openproject_boards seed data
Nov 14 18:24:15 *** Loading openproject_calendar seed data
Nov 14 18:24:15 *** Loading openproject_documents seed data
Nov 14 18:24:15 *** Loading openproject_github_integration seed data
Nov 14 18:24:15 *** Loading openproject_job_status seed data
Nov 14 18:24:15 *** Loading openproject_ldap_groups seed data
Nov 14 18:24:15 *** Loading openproject_meeting seed data
Nov 14 18:24:15 *** Loading openproject_openid_connect seed data
Nov 14 18:24:15 *** Loading openproject_recaptcha seed data
Nov 14 18:24:15 *** Loading openproject_reporting seed data
Nov 14 18:24:15 *** Loading openproject_storages seed data
Nov 14 18:24:15 *** Loading openproject_team_planner seed data
Nov 14 18:24:15 *** Loading openproject_two_factor_authentication seed data
Nov 14 18:24:15 *** Loading openproject_webhooks seed data
Nov 14 18:24:15 *** Loading openproject_xls_export seed data
Nov 14 18:24:15 *** Loading overviews seed data
Nov 14 18:24:15 *** Loading primer_view_components seed data
Nov 14 18:24:15 *** Loading turbo seed data
Nov 14 18:24:15 *** Loading view_component_engine seed data
Nov 14 18:24:15 *** No need to seed an admin as there already is one.
Nov 14 18:24:15 *** Seeding admin user
Nov 14 18:24:15 *** Seeding data from environment variables
Nov 14 18:24:15 *** Seeding demo data
Nov 14 18:24:15 *** Skipping BasicData::Backlogs::SettingSeeder
Nov 14 18:24:15 *** Skipping DemoData::OverviewSeeder
Nov 14 18:24:15 *** Skipping DemoData::ProjectsSeeder
Nov 14 18:24:15 *** Skipping EnvData::LdapSeeder
Nov 14 18:24:15 *** Skipping colors as there are already some configured
Nov 14 18:24:15 *** Skipping document categories as there are already some configured
Nov 14 18:24:15 *** Skipping open color palette as there are already some configured
Nov 14 18:24:15 *** Skipping priorities as there are already some configured
Nov 14 18:24:15 *** Skipping roles as there are already some configured
Nov 14 18:24:15 *** Skipping settings as all settings already exist in the db
Nov 14 18:24:15 *** Skipping time entry activities as there are already some configured
Nov 14 18:24:15 *** Skipping types, statuses and workflows as there are already some configured
Nov 14 18:24:15 => Update LDAP config
Nov 14 18:24:15 => Update general config
Nov 14 18:24:15 INSERT 0 1
Nov 14 18:24:15 Loading discovered seeders: Backlogs::SettingSeeder, Documents::EnumerationSeeder
Nov 14 18:24:15 UPDATE 0
Nov 14 18:24:15 UPDATE 1
Nov 14 18:24:15 UPDATE 1
Nov 14 18:24:15 ↳ Backlogs::SettingSeeder
Nov 14 18:24:15 ↳ BuiltinUsersSeeder
Nov 14 18:24:15 ↳ ColorSchemeSeeder
Nov 14 18:24:15 ↳ ColorSeeder
Nov 14 18:24:15 ↳ Creating global queries
Nov 14 18:24:15 ↳ Creating groups
Nov 14 18:24:15 ↳ Documents::EnumerationSeeder
Nov 14 18:24:15 ↳ GlobalQuerySeeder
Nov 14 18:24:15 ↳ GroupSeeder
Nov 14 18:24:15 ↳ LdapSeeder
Nov 14 18:24:15 ↳ OverviewSeeder
Nov 14 18:24:15 ↳ PrioritySeeder
Nov 14 18:24:15 ↳ ProjectsSeeder
Nov 14 18:24:15 ↳ RoleSeeder
Nov 14 18:24:15 ↳ SettingSeeder
Nov 14 18:24:15 ↳ TimeEntryActivitySeeder
Nov 14 18:24:15 ↳ WorkflowSeeder
Nov 14 18:24:16 => Clear previous cache to reflect db changes
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 0
Nov 14 18:24:16 UPDATE 1
Nov 14 18:24:16 UPDATE 1
Nov 14 18:24:16 UPDATE 1
Nov 14 18:24:16 UPDATE 1
Nov 14 18:24:24 => Fixup the directory permissions
Nov 14 18:24:24 => Start apache
Nov 14 18:24:25
Nov 14 18:24:25 [Tue Nov 14 11:24:25.751437 2023] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.52 (Ubuntu) Phusion_Passenger/6.0.18 mod_perl/2.0.12 Perl/v5.34.0 configured -- resuming normal operations
Nov 14 18:24:25 [Tue Nov 14 11:24:25.752753 2023] [core:notice] [pid 1] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'!
Could not create nginx config directory: ENOSPC: no space left on device,2023-11-14T11:23:29.000Z => Migrate database

Looking at the log. Guacamole complains of dns resolution. Open Project complains of no disk space left.

So, how much disk space is really required by cloudron? And I doubt resolving dns isn't the problem faced by gaucamole.

How do I go about solving this instead of starting from scratch again?

Thank you

[nebulon]

The disk stats are only updated once per day, so they are not a live state. Maybe your server by now has run out of disk space. If so please follow https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full as since you were seeing dns issues, likely unbound is down.

[gnulab]

Hi,

guacamole is solved. Redownloading trust key and restarting unbound solves it.

As for Open Project, here is the disk usage from terminal:

root@g:~# df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs             202340     1976    200364   1% /run
/dev/vda1       53973208 31476756  19596548  62% /
tmpfs            1011688        0   1011688   0% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/2adb4e3aa02ae42d4f42f64b338abcc2dff9c441e278ed8c9c8b857449bd7b0e/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/893d943a6da7efd9c421d8b382545bf1087c6e321d7f579e37491463193568c6/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/def7ca2edc00af55f2e87d3d12a9b72ba544aa474daac0ce031f1e664649d767/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/e0fd7dcf64ccd46431170c11a9ccaabc0e4ae29ac5a461a1c037e82cd78b4575/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/56f8a71c02109cc65c8b7334d57f77b79825b997d2b3d035f73a16034766a058/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/dd08c4baa92862a501aadcad5256fcb91b25a9269b71190ac9af852597182e0a/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/dc062cba3af3f4f27c48e8759363d39567ebce34682da58479f749b9d3745af2/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/14af837367abe59eb18c5f5e6094390865e0272d72458cb340a9699bb16ea378/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/8069d9b97acc391211b4845ee94ea6e030070a90479bab02c392b96af47f4155/merged
tmpfs             202336        4    202332   1% /run/user/0
root@g:~#root@g:~# df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs             202340     1976    200364   1% /run
/dev/vda1       53973208 31476756  19596548  62% /
tmpfs            1011688        0   1011688   0% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/2adb4e3aa02ae42d4f42f64b338abcc2dff9c441e278ed8c9c8b857449bd7b0e/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/893d943a6da7efd9c421d8b382545bf1087c6e321d7f579e37491463193568c6/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/def7ca2edc00af55f2e87d3d12a9b72ba544aa474daac0ce031f1e664649d767/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/e0fd7dcf64ccd46431170c11a9ccaabc0e4ae29ac5a461a1c037e82cd78b4575/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/56f8a71c02109cc65c8b7334d57f77b79825b997d2b3d035f73a16034766a058/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/dd08c4baa92862a501aadcad5256fcb91b25a9269b71190ac9af852597182e0a/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/dc062cba3af3f4f27c48e8759363d39567ebce34682da58479f749b9d3745af2/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/14af837367abe59eb18c5f5e6094390865e0272d72458cb340a9699bb16ea378/merged
overlay         53973208 31476756  19596548  62% /var/lib/docker/overlay2/8069d9b97acc391211b4845ee94ea6e030070a90479bab02c392b96af47f4155/merged
tmpfs             202336        4    202332   1% /run/user/0
root@g:~#

/dev/vda1 53973208 31476756 19596548 62% /
Clearly 62% of the disk is still unused.

So, how do I make cloudron 'sees' and uses that disk space?

Thank you.

[girish]

@gnulab it will start using any free disk space automatically. Maybe try, going to the Repair section of the app. Click Enable Recovery Mode and then Click Disable Recovery Mode. This will recreate the container and maybe it works now. Can you try?