Update app : Read-only file system

[stephan]

Hello,

I’m encountering an issue when updating my custom app: I get a Read-only file system error, but I’m unsure why it only occurs during updates. The deployment to a new URL works fine, but when updating, I see the following error:

Jul 28 17:00:02 mkdir: can't create directory '/certs/ca': Read-only file system

My app is a Java application with the following Docker configuration:

FROM cloudron/base:4.2.0@sha256:46da2fffb36353ef714f97ae8e962bd2c212ca091108d768ba473078319a47f4

RUN mkdir -p /app/code
WORKDIR /app/code

RUN apt-get update && apt-get install -y openjdk-21-jdk-headless && rm -rf /var/cache/apt /var/lib/apt/lists

# copy code
ADD ./cicd/start.sh /app/code/

COPY ./target/back-0.0.1-SNAPSHOT.jar /app/code/back.jar
COPY ./src/main/resources/application-prod.properties /app/code/application-prod.properties

CMD [ "/app/code/start.sh" ]

The start.sh script is as follows:

#!/bin/bash

set -eu
chown -R cloudron:cloudron /app/data

echo "Starting app server"

exec gosu cloudron:cloudron java -jar myApp.jar --spring.profiles.active=prod

During the first deployment, everything works correctly, and the app starts fine. However, when I update the JAR file, I encounter this error:

Jul 28 17:03:03 mkdir: can't create directory '/certs/ca': Read-only file system
Jul 28 17:03:47 => Healtheck error: Error: Timeout of 7000ms exceeded
Jul 28 17:03:56 => Healtheck error: Error: connect EHOSTUNREACH 172.18.18.128:8080

Here is the manifest used for deploying my app:

{
  "version": "0.0.2",
  "healthCheckPath": "/",
  "httpPort": 8080,
  "addons": {
    "localstorage": {},
    "postgresql": {},
    "redis": {}
  },
  "icon": "file://logo.png",
 "manifestVersion": 2
}

I have checked various forums and the web but couldn’t find any solutions specific to my issue. Could anyone help me with this?

Thanks and have a nice day!

[joseph]

@stephan is that the entirety of start.sh ? If so, myApp.jar is calling mkdir command at some point and trying to create a directory named /certs/ca . It's not allowed to create directories arbitrarily inside the container, that is why the command fails with EROFS.

[stephan]

@joseph The start.sh file is entirety.
The java app is a spring app with redis and postgres.
In that case, why is it that on a first deployment I don't get the error and for the same version of the docker image I could get this error?

I'll have another look, but even locally I don't create files, I expose an api and store data in the database.

Thank for your answer.

[joseph]

@stephan I just noticed that you had a log line Starting app server . Does the mkdir error appear before or after log line ?

[stephan]

The mkdir error appear before Starting app server :

Jul 29 19:36:19 2024-07-29T17:36:19.712Z INFO 1 --- [API] [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
Jul 29 19:36:20 box:services teardownAddons: Tearing down []
Jul 29 19:36:20 box:tasks update 446: {"percent":45,"message":"Downloading icon"}
Jul 29 19:36:20 box:tasks update 446: {"percent":60,"message":"Updating addons"}
Jul 29 19:36:20 box:services setupAddons: Setting up ["localstorage","postgresql","redis"]
Jul 29 19:36:20 box:services setupAddons: setting up addon localstorage with options {}
Jul 29 19:36:20 box:services setupLocalStorage
Jul 29 19:36:20 box:shell createVolume /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/ca96d396-a77a-40a5-9ff8-707f40525bfa/data
Jul 29 19:36:20 box:services setupAddons: setting up addon postgresql with options {}
Jul 29 19:36:20 box:services Setting up postgresql
Jul 29 19:36:20 box:services Setting postgresql addon config to ...
Jul 29 19:36:20 box:services setupAddons: setting up addon redis with options {}
Jul 29 19:36:20 box:services Re-using existing redis container with state: {"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":380849,"ExitCode":0,"Error":"","StartedAt":"2024-07-29T17:32:35.671184747Z","FinishedAt":"0001-01-01T00:00:00Z"}
Jul 29 19:36:20 box:services Waiting for redis-ca96d396-a77a-40a5-9ff8-707f40525bfa
Jul 29 19:36:20 [GET] /healthcheck
Jul 29 19:36:20 box:tasks update 446: {"percent":70,"message":"Creating container"}
Jul 29 19:36:20 box:apptask createContainer: creating container
Jul 29 19:36:20 box:shell addLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add ca96d396-a77a-40a5-9ff8-707f40525bfa /tmp/ca96d396-a77a-40a5-9ff8-707f40525bfa.logrotate
Jul 29 19:36:20 box:apptask startApp: starting container
Jul 29 19:36:21 mkdir: can't create directory '/certs/ca': Read-only file system
Jul 29 19:36:21 box:tasks update 446: {"percent":90,"message":"Configuring reverse proxy"}
Jul 29 19:36:21 box:tasks update 446: {"percent":100,"message":"Done"}
Jul 29 19:36:21 box:taskworker Task took 10.293 seconds
Jul 29 19:36:21 box:tasks setCompleted - 446: {"result":null,"error":null}
Jul 29 19:36:21 box:tasks update 446: {"percent":100,"result":null,"error":null}
Jul 29 19:36:21 mkdir: can't create directory '/certs/ca': Read-only file system

In this exemple, i use the same docker image. Just execute and update with :

cloudron update --image docker pull myapp.domain.com/myImageDocker:1.0 --app test-java

[joseph]

@stephan mysterious. Something somewhere is calling mkdir. Could it be an obsolete docker image? (maybe you didn't build?) Do you know anything about that line? Cloudron is just running the docker image... Is your code/image public by any chance to have a look? I guess it is the code that you posted and nothing else? If so, there is nothing here that calls mkdir.

You can try this:

• cloudron debug . This puts the app in debug mode, meaning it won't start.
• cloudron exec . You now get a shell into the container
• Run start.sh manually . Try to figure if mkdir is coming from here

[stephan]

@joseph
I found my problem. It was a typo (user error). When I was running the following command: cloudron update --image [image_name] --app [name_app], I accidentally added an unexpected parameter, resulting in cloudron update --image docker pull [image_name] --app [name_app]. Naturally, this didn’t work, but I must admit I can’t explain why the error message was something other than a fail to load...

Sorry again for the inconvenience.

[joseph]

Glad you got it sorted out!