Cloudron Forum

I just thought I'd mention a fun use of Tailscale, which I'm unreasonably pleased with, even though it was minimal effort to do.

To start, I have Cloudron backing up to a local SSD. It's an old 2.5" that has enough space for the host backups (which are... rsync format, I think).

This weekend, I took a an old, small machine (a NUC I accumulated from years ago), installed Ubuntu 22.04 on it, plugged in an old 1TB USB HDD, and took it to an undisclosed remote location. (Read: a family member's house.) I set up my fstab to mount the USB drive by ID, so it should always come up on boot. (I also remembered to set the machine's bios so it would come on after power failures.)

I then installed Tailscale on both my Cloudron host and this aging NUC. Finally, I set up my crontab on the Cloudron host to run rclone every now and then. It clones my backup (from the SSD) to the remote, undisclosed location over the Tailscale network.

This saved me a ton of time in terms of setting up a hole in the remote router (for a secure SSH connection), as well as worrying about whether or not I have secured SSH adequately. Granted, I'm trusting Tailscale to do the right thing here, but I figure it has a better chance of being secure than me quickly hacking things together.

Although it isn't a full "Cloudron using Tailscale" story, it is nice that the default Tailscale configuration is to leave all your public networks alone. As a result, the Cloudron host can be set up to replicate backups elsewhere very quickly and easily.

@andreasdueren I've packaged one or two things for myself on Cloudron, and I took a look at Frappe/ERPnext.

First, for the thread: Cloudron does not run Docker images "as-is." Or, if you prefer, simply because a project runs in Docker does not mean that it will immediately be runnable under Cloudron. A Cloudron app needs to be packaged up so that it will "play nicely" with the control architecture that Cloudron provides. Put simply, to get that friendly Cloudron experience, some work is needed when packaging an app.

In the case of ERPnext, it has a compose file that specifies many software services. Traefik is used for routing and load balancing (I assume); Nginx fronts the service; it seems like Frappe (the API backend) is written in Python (another service). There's worker processes of several flavors, a scheduling service, a Redis cache, MariaDB (which, for porting to Cloudron, we'd want to integrate with it's built-in DB add-on), the site creator service... and a large number of storage volumes.

Cloudron does not, to the best of my understanding, support running docker-compose files. As a result, to package this, we'd have to pull all of these services into a single container image. That would take some thinking, especially since Docker "likes" to have one process per container. Or, if there is another way/it is possible, I don't personally know how to package up a multi-container Cloudron application.

The Cloudron team may have something else to say, but I thought I'd drop a note in the thread that helps explain why this app is a more complex proposition than others (perhaps) when it comes to packaging. Yes, it is open, and yes, it installs easily on a VM when you do a docker-compose up. Unfortunately, that is not the same as packaging things up to run under the Cloudron framework.

I've started work on this, and will update this thread when I have it in a repo. That might be later today, it might be in another day or two. I managed to get:

1. A modified Dockerfile
2. A modified startup script
3. The image pushing to a (local) private repo
4. An install into my local Cloudron

And, at that point, I have more environment variables to set.

So, it seems possible/I'll make the work public shortly.

2FA with authenticator apps are, by-and-large, all using TOTPs (https://en.wikipedia.org/wiki/Time-based_one-time_password), and therefore are effectively standardized. Whether you use Google's Authenticator, Authy, FreeOTP, Keepass, Vaultwarden, or something else, it doesn't matter. Or, if you find a provider where it does matter, you might want to be concerned.

https://alternativeto.net/software/google-authenticator/?license=opensource

You can also, in many 2FA contexts, use a hardware key.

https://www.yubico.com/

which have some added benefits (and drawbacks, mostly "it's a thing you can lose). Or

https://www.crowdsupply.com/sutajio-kosagi/precursor

if you really want a serious bit of kit from an open-and-secure perspective.

In short, and with kindness: I think you're searching for a boogeyman where there isn't one. I want 2FA on every account that matters to me, and I especially want stronger authentication frameworks in my software supply chain. I want 2FA on my bank accounts, I want 2FA on my email... really, I want something that goes beyond a single, salted/hashed password everywhere.

I'm not saying you shouldn't want to self-host your code on your own stack, and only use the most libre of free software. However, I think worrying about TOTP/2FA is like worrying about the "forced" transition to HTTPS everywhere. It's actually a good thing, and it isn't a "give us all your information" play. 2FA is a smart thing to do.

That said, I'm not keen on biometrics as a second factor.

@Neiluj

I would welcome input from a member of the team on this.

Docker is intended to run a single process in a single container. When you want to run multiple services, you run multiple containers.

This is where you (typically) would use some kind of tool to orchestrate or compose those containers. For example, a docker-compose.yml will define a set of services, and how they connect and interact with each-other.

Cloudron is designed to host singleton containers. Unlike industrial-scale platforms as a service, it provides limited tooling for how to define connections to other services. The manifest allows you to connect to the services that exist; for example, I can say "connect to the Cloudron-provided Postgres server." However, I have no way to say "I have chosen to run an S3 server/Minio at location X, and please use it." As a result, it puts a significant burden on the user. Further, there is no way within the package to say "this should not boot if that service is not present." You have to write custom code in order to provide that logic.

Further, the Docs app itself wants to run multiple services. The frontend is separate from multiple backends in their design. The app itself has orchestration concerns and considerations.

So while I appreciate people saying "but it is all there!," we're not discussing what is required to make this a production-grade package.

1. You have to pull the codebase.
2. Study how it wants to be run, and decide how it can be run in Cloudron.
3. Write a custom build/containerization script for the app, compressing multiple services (that want to be in multiple containers) into a single container.
4. Write custom code to make sure external services that are not part of the Cloudron PaaS are present, and fail gracefully/in a way that a non-expert user can debug, so that they can add those services and connect them to the app.
5. Make this all automated and testable.
6. Maintain all of this while upstream rapidly evolves.

I'm forgetting things, I'm sure. I estimate 80-120h of work in this, and it is essentially devops work. It should bill at $85+/hour. Further, I'm notorious for underestimating how long it takes to develop features by a factor of 2x-4x. So, I think this is work worth at least $8K-$12K---to say nothing of having to maintain the package, against a large, fast-moving target. (And, while it is open source, governments tend to be very careful about accepting changes from third parties, because there are significant security and compliance burdens they must bear.)

Maybe I'm just accustomed to deployments in Ansible and Terraform, and am overstating the difficulty of this deploy. However, my experience is that when a system is designed to run one way, and you want it to run some other way, there's significant work involved.

So, in return, please forgive my ignorance. I may be misunderstanding things about packaging for Cloudron, and you may be right: this may be easier than I think.

I've started poking at a package for Planka to refresh myself on packaging, because it is a singleton Node app that only has Postgres as an external dependency. It is an example, in my mind, of an app that fits the Cloudron model perfectly. However, anything that wants orchestration beyond the core services Cloudron provides---especially when some of those services are custom components internal to the application itself---is, in my mind, significantly more effort.

@girish Hi Girish. You got it.

I have a small IoT service hosted in the home network that adding auth/protection to would be painful. (It doesn't have a notion of auth, I don't think. So, Cloudron-as-LDAP-authority doesn't even save me.) The App Proxy feature made it easy to bounce through to it from outside, but I don't want it unauthenticated.

So, yes. You nailed exactly what I was wondering.

Flatpack and Appimage formats do not provide the kind of process and filesystem isolation that Docker does. I would not be comfortable running run a service like Cloudron if it allowed running things other than well-defined/managed containers.

Yes, there are ways to break isolation and escalate out of a Docker container, but flatpack apps can, essentially, write anywhere on my system. Updating the ecosystem is also much more difficult; by comparison, it is possible to auto-build Docker containers from source and track recent updates/manage timely security vulns in a way that flatpack/appimage packages do not/rarely do.

Someone more familiar with packaging Cloudron apps would be able to answer better than me. However, I find that whenever a docker-compose.yml is involved, it is probably hard to move the app to Cloudron.

https://github.com/AppFlowy-IO/AppFlowy-Cloud/blob/main/docker-compose.yml

In this case:

• It wants nginx. That might be avoidable, or it might be serving static assets/code for the app.
• It wants minio. This could probably be accommodated by requiring users to run minio on their Cloudron before installing this.
• It wants postgres, which might be able to be leveraged from the internal stack.
• It wants redis. Again, possibly from the default stack... I can't remember.
• gotrue is an auth component from supabase. This will need its own container, and may (or may not) play nice with the SMTP/OAuth running on Cloudron.
• appflowy_cloud is the hosted app. It wants its own container, and configuration information for all of the services included.
• admin_frontend has its own Dockerfile. I haven't looked. Looks like more things.
• ai. I have no idea. It looks like it wants some kind of OpenAI. This is getting heavy in terms of resources.
• appflowy_history is... another Dockerfile. Looks like a rust application that has been Dockerized.

The problem, I think, is that Cloudron assumes/is structured such that applications run as single containers. The compose is suggesting that this application has a number of independent components. Perhaps those could be bundled up/run separately... but, it might be a real trick to make work.

This isn't to say it isn't possible, but that's what I see that needs to run, and it isn't clear to me that this is an easy app architecture to move over to Cloudron. YMMV, etc.

OK.

This was very exciting.

I read some documentation. Specifically, https://docs.opnsense.org/manual/how-tos/nat_reflection.html.

Once I slowed down, undid all the weird thrashing I did with various DNS shortcuts for route domains internally/directly (e.g. changing my unbound config, or creating aliases for my domain), and instead read the documentation for both reflection and hairpin NAT in OpnSense, I was good to go.

Thank you for joining me on this journey where I create noise on the forum and discover that, by reading the manual, I can solve my own problems.

@humptydumpty I run OPNsense in front of Cloudron. I'm not doing anything fancy with it, but it does live between the world and my self-hosted Cloudron instance.

I have no idea what would happen if the machine was DDoS'd. I'm pretty sure it would fall over. At this point, I'm just excited that I have cron'd backups locally and to offsite.

Looking at the Docker Compose, this one is actually a very clean/light app.

1. The license is the GNU Affero, so there should be no licensing issues.
2. The Dockerfile is small, and basically a node app.
3. The docker-compose.yml is actually simple! It spins up the image for Planka, it has a whole bunch of environment variables (commented out as documentation, mostly), and it expects Postgres as a database.
4. It has a backup script!
5. It has an active repo and community!

I'm... in a rather tough spot in terms of time, but this feels like a good packaging target. @girish , @nebulon , is this app one that you would want to see in the app store? (To me, that's an important first question.) If so, do you have additional requirements (e.g. tests, automation) that you would want to see in place?

Hi all,

This thread is intended to document my reading of other support tickets, and what I've done in an attempt to bring my cloudron back.

BLUF: I succeeded, but I don't like how I did it. Ultimately, I had to edit resolv.conf. This ticket can likely be closed.

Initial problem

My experience so far is similar to this thread: https://forum.cloudron.io/topic/12294/autoupdate-cloudron-aborts/9

I pushed the button for the update to 8.0.3. Unfortunately, I did this shortly before leaving the house for two days. I thought "everything will be fine."

Return from trip

I came back to find apps and services down. (Unbound was down, for example.) I did a reboot to see what would happen. ("Turn it off and turn it back on again.") The services came back, but the apps did not.

Looking at one of the apps, I see this error:

An error occurred during the configure operation: Network Error: Network error downloading icon : getaddrinfo EAI_AGAIN api.cloudron.io

Explore the forum

I tried some possibly obvious fixes from the forum.

Update unbound settings

Following this page:

https://docs.cloudron.io/networking/#dns

I added a file to /etc/unbound/unbound.conf.d called override.conf, and it contains the following:

server:
    val-permissive-mode: yes
forward-zone:
    name: "."
    forward-addr: 10.0.0.1

My cloudron lives behind an OpnSense instance which serves as my router, and it is at 10.0.0.1. If you are following this, your DNS server is probably not 10.0.0.1. So, if you're also trying to fix something similar... you cannot just "plug and play" the values I use, but you might be able to follow the process.

After that change and an unbound restart, I could do:

host www.cloudron.io 127.0.0.150

and it reported back

Using domain server:
Name: 127.0.0.150
Address: 127.0.0.150#53
Aliases: 

www.cloudron.io has address 165.227.67.76
www.cloudron.io has IPv6 address 2604:a880:800:10::b66:f001

This is good. I then tried to configure a failing app, and it still would not talk to api.cloudron.io. So, the problem did not go away.

Update netplan

I still had a DNS problem. Based on this thread:

https://forum.cloudron.io/topic/12266/auto-update-to-8-0-3-systemd-resolved-empty-response/6

I considered the possibility that the issue might be deeper down. I went into my netplan, and modified my configuration. It was set for straight DHCP. My netplan now reads as:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s31f6:
      dhcp4: true
      dhcp6: false
      dhcp4-overrides:
        use-dns: false
      nameservers:
        addresses:
          - 10.0.0.1
          - 1.1.1.1
          - 8.8.8.8

This should, I think, set my router as the first DNS option, with Cloudflare and Google as fallbacks.

I did a netplan apply and then I rebooted the machine (shutdown -r now), and everything came back. I also enabled DNS query forwarding in OpnSense at this point, just in case. I don't know that it mattered.

host www.cloudron.io 127.0.0.150

works.

curl https://releases.cloudron.io/versions.json

does not work. So, curl is not picking up the DNS; it seems to be looking to the default :53, which I suspect is not unbound, but instead... something else. dig has the same problem.

 dig google.com
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> google.com
;; global options: +cmd
;; no servers could be reached

Edit /etc/resolv.conf

I don't like the solution I ended up with.

Based on this thread:

https://forum.cloudron.io/topic/12286/dns-failure/8

I decided that my empty resolv.conf was a problem. (I don't know why it was empty, and I don't like editing a file that claims it will be overwritten.)

I entered

nameserver 10.0.0.1

(again... for me is a good value.)

Reboot

After that change, I was able to manually configure/restart two apps. From there, I decided the fastest way to bring everything back (and confirm things might be good in general) would be to do a reboot from the admin panel.

This is not fix all of the apps. I had to retry the configure task on each, and Restart App for each.

Conclusion

At this point, I now have all my apps back. Cloudron claims v8.0.3 (Ubuntu 22.04.1 LTS), which is the most recent version.

Platform Version: v8.0.3 (Ubuntu 22.04.1 LTS)
Vendor: Dell Inc.
Product: OptiPlex 7040
CPU: 4 Core "Intel(R) Core(TM) i5-6500T CPU @ 2.50GHz"
Memory: 33.51 GB RAM & 8.59 GB Swap
Uptime: 9 minutes

I think this ticket can be closed, but given that a number of updates to 8.0.3 seem to have had DNS issues, I am going to submit this here. It may also be that I fixed the problem incorrectly.

Hi all,

I once, long ago, ran an exim mailserver with a colleague. At some point, we got zero-day'd, and I decided that running a mailserver was less fun than I thought. I've never looked back.

I maintain a domain on an external provider almost entirely for the email forwarding. That is, I have a domain and addresses that I only forward on to other email hosts (e.g. Gmail).

I could move that domain to my Cloudron. At that point, I would be putting all of my personal infrastructure on a box in my basement, and be relying on it for my most important piece of comms infrastructure. And, I know I'd need to actually test my backup and restore process at that point, because I really couldn't afford to have an outage take out my family's email for (say) days.

Do people use Cloudron for production mail? (I mean, I assume they must.) Are there any concerns? Gotchas? Are there other paths people have walked? I've tried experimenting with Cloudflare's email forwarding solution, but was unable to get it to work reliably (a number of months ago).

Many thanks,
Matt

Agreed. I'm not offering thoughts from a spirit of "GIVE UP!," by any means. It is more from the perspective of "I think this one is trickier than it seems at first glance."

But, I am still learning. So, the staff may say "this is actually easy!" Or, they might say "Yep, it's kinda tricky." And, as a result, we all learn more.

I would like to politely ask for a bit more info around cert updates, please.

I received an email yesterday from LetsEncrypt saying I had one day left on my cert renewal for the my domain in my Cloudron instance. It is not spam or a scam; I believe this email was legitimately from certbot.

I pushed the "renew certs" button in the control panel, but... did it work? How would I know? I read the logs, and played with openssl; of course, the latter is an external measure, and may be confused by my Cloudflare proxy. I think the update worked... but, visual log parsing is the hard way to figure out if a complex process succeeded. Was that null result buried in a JSON log an indicator of success? It was late, I was tired, and perhaps tomorrow my certs will fail.

1. It would be nice if the dashboard/control panel told me, after pressing the button to update certs, if the system thought it succeeded.
2. It would be nice if that same panel area are simply told me "Certs last renewed on <date>, expire on <date>, and the next cert update will run on <date>."

Cheers,
Matt

@girish No, I don't think I have any particular DDoS protection configured in OPNsense. However, this conversation makes me curious to investigate it.

I run the DNS through Cloudflare, which... may or may not provide me with some protection. (I'm not being cagey, so much as haven't really dug in to understand how/if Cloudflare protects my Cloudron domain/subdomains.)

I'm happy to experiment with OPNsense configuration, if it's a space of question. However, it serves more for internal protection than external (at the moment). I mostly use it to partition the Cloudron machine off from the rest of the house via VLAN as a "just-in-case" measure. It keeps Cloudron separate from dumb internet lightbulbs, which I consider a kind of mutual protection.

The Docs installation requirements:

https://github.com/suitenumerique/docs/blob/main/docs/installation.md

This requires Kubernetes. Or, their "native" deployment method assumes Kubernetes.

From that page:

an OIDC provider (if you don't have one, we will provide an example)
a PostgreSQL server (if you don't have one, we will provide an example)
a Memcached server (if you don't have one, we will provide an example)
a S3 bucket (if you don't have one, we will provide an example)

In theory, it might be possible to take their repository, and deploy this in a single container as a Django app that uses

• Minio for S3
• The Cloudron-native Postrges instance
• The Cloudron-native OIDC
• ... mumble mumble memcached...

To make this viable, the right thing would be to extend Cloudron to provide a "native" Minio and Memcached instance for apps to use. (That is, have them present from time of install along side the various databases.) I could imagine making a product case for Minio, I think we'd struggle to convince girish and nebulon that the additional complexity of including memached everywhere was worth it.

Either way, this would be a major packaging effort. Without digging deeper, I'm going to assume that the Django app is not abstracted away from the services as well as we'd like, and that K8s requirement will make it very difficult to package/deploy to Cloudron. "Very difficult" means "many, many weeks of work, including changes to the core infrastructure of Cloudron."

(It looks like they have a compose (link) which I suspect is for local development of the app. It uses Redis instead of Memcached, still needs Minio, but also wants an Nginx instance and Celery instance... and Keycloack... and the app is in multiple containers/pieces...)

Long-and-short, there is no apparent easy path to bring this to Cloudron as a single-container/single Dockerfile application.

@nebulon Many thanks, @nebulon and @girish . The concern wasn't so much that I could not figure out what the status of my certs were external to Cloudron, but more that it would be nice if the area of the dashboard regarding certs would, as a matter of course, just say "You have 47 days remaining, and Cloudron should automatically update your certs in 17 days."

And, if I do mash the button to manually run a cert update, it would be nice to get a response in the dash that says "Success! New certs will expire in 90 days!" (Or, whatever it would say.)

I was mostly surprised that I got a certbot email saying I only had one day left, making me wonder what was up. (I did do a domain registration move at some point, and possibly other things that could have somehow upset the automatic update process. So, this isn't a bug report.) Not having a simple UI response to the act of hitting "update certs" (and instead being dumped into the log) is all I'm poking at.

I don't know how long my personal instance has been running (a month or two now), but it has been a joy. Thank you.

This is all good.

@luckow , @Kubernetes , thanks for the context. The... difference between a data center is mostly immaterial, I think. I mean, unless you consider that there is no power infrastructure in my basement, the ethernet cables are tacked to the ceiling, and there's no redundancy... OK. So, it's a little different. That's a separate issue, though, from the question of "Cloudron as my mail solution." So, thank you for the +1s.

@fbartels , @Dave-Swift , and @BrutalBirdie , I think you've all raised good questions.

• How will I send mail out? Do I relay? If so, what service?
• Will my ISP allow port 25 all the way down? I don't know. I know they're letting :80 and :443 in, because I'm hosting Cloudron at the end of my fiber connection. But, that doesn't mean they're not blocking :25.

Many thanks all. Good considerations.

Ah. I tend to look at the compose, because if there is a large list of additional services, cramming them into a single container can become a problem from my point-of-view.

I agree 100% with your assessment. (I think that was what I was implying with my #2.)

If the Cloudron team thought this was good to add to the stable of apps, I'd give it a go. And, I'd want to do the work to integrate it into whatever build/test frameworks that are in place. However, my workplace is going through some complex and public difficulties, so my energy at the end of the day tends to be limited. So, that's a "yes, but" on diving in on packaging right now.