Cloudron Forum

Home Assistant / home-assistant.io

Awaken your home

Open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a Raspberry Pi or a local server.

Pretty awesome application and community. Today I was able to set it up on my Pi3, connected a zigbee usb controller, set up a few light bulbs, and already created scripted automations all within an hour or so. Strikingly straightforward to set up.

I think it would be great if we were able to manage an instance in Cloudron, maybe even on a Pi. I think this would be a great use case for the multi-cloudron feature coming in 7.0.

Home assistant has a Docker-based deployment option; though it also has a bunch of device / hw integrations (zigbee usb controller, for example) which could delay a fully-featured implementation.

Matrix just announced Matrix 2.0 with sliding sync:

https://matrix.org/blog/2023/09/matrix-2-0/

Commentary including the Matrix project lead, Arathorn: https://news.ycombinator.com/item?id=37599510

I suspected some of my apps were consuming a lot of space recently, and I had to do a fair amount of digging before the details became evident. To me this was particularly concerning for backups because they don't do any deduplication and I could end up with many multiples of this increased disk consumption.

It would be nice to have the size of each backup recorded in the metadata for backups. Even if it's a little off compared to my backup host it would still be nice to have an order-of-magnitude estimate.

I'm gonna try to package it up for Cloudron this evening and report back here with my results.

Notea / https://github.com/QingWei-Li/notea

Self hosted note taking app stored on S3. Notea is a privacy-first, open-source note-taking application. It supports Markdown syntax, sharing, responsive and more.

Brought to my attention by @yusf on Notion-like FOSS app thread.

I started a cloudron app package for Notea, you can find the source here: https://source.infogulch.com/infogulch/notea-app If anyone wants to try it, the docker image for this source is infogulch/notea-app:0.0.1-6.

I did run into some trouble after getting it to start successfully:

• I'm using a minio instance on the same cloudron as the data store for Notea since it doesn't support local file storage directly. This means that it's a bit awkward to configure; after installing the app, open the terminal and edit /app/data/env.sh to add the s3/minio backend storage config, then restart the app.
• I used proxyAuth because I think it's the bees knees but that mostly just caused me extra problems because the notea app implements authentication itself and redirects to /login which obviously doesn't work very well with proxyauth. I did open an issue to request an option to disable the login page.
  • In the meantime, it would probably be more expedient to relent and disable proxyAuth, and use notea's built-in authentication which is just a password configured by setting the PASSWORD env var in env.sh

Just to put my money where my mouth is, here is a formatted view of docker system df -v from one of my cloudrons which shows for each image what its "size", "shared size", and "unique size" is (I hope fits nicely):

REPOSITORY                                    SIZE      SHARED SIZE   UNIQUE SIZE
cloudron/org.navidrome.cloudronapp            2.212GB   2.176GB              36MB
cloudron/im.riot.cloudronapp                  2.223GB   2.176GB              47MB
cloudron/com.github.bitwardenrs               3.286GB   2.176GB            1110MB
cloudron/org.matrix.synapse                   2.555GB   2.176GB             379MB
cloudron/org.jupyter.cloudronapp              3.358GB   2.176GB            1183MB
cloudron/io.gitea.cloudronapp                 2.326GB   2.176GB             150MB
cloudron/github.pages.cloudronapp             2.286GB   2.176GB             109MB
cloudron/net.minecraft.cloudronapp            2.723GB   2.176GB             547MB
cloudron/org.fireflyiii.cloudronapp           3.269GB   2.176GB            1093MB
cloudron/com.docker.registry                  2.284GB   2.176GB             108MB
cloudron/mail                                 2.844GB   2.176GB             668MB
cloudron/sftp                                 2.194GB   2.176GB              18MB
cloudron/io.minio.cloudronapp                 2.236GB   2.176GB              59MB
cloudron/org.freshrss.cloudronapp             2.184GB   2.176GB               8MB
cloudron/postgresql                           2.343GB   2.176GB             167MB
cloudron/net.roundcube.cloudronapp            2.239GB   2.176GB              63MB
infogulch/promnesia-app                       2.224GB   2.176GB              48MB
cloudron/graphite                             2.246GB   2.176GB              70MB
cloudron/turn                                 2.182GB   2.176GB               5MB
cloudron/com.electerious.lychee.cloudronapp   2.248GB   2.176GB              72MB
cloudron/mysql                                2.495GB   2.176GB             318MB
cloudron/redis                                2.182GB   2.176GB               6MB
cloudron/mongodb                              2.299GB   2.176GB             122MB
infogulch/terminusdb-app                      3.331GB   2.176GB            1155MB
cloudron/org.radicale.cloudronapp2            2.187GB   2.176GB              11MB
cloudron/base                                 2.176GB   2.176GB                0B

You might notice that the "size" of every image is at least 2.1GB, but that 2.1 GB is shared between all the apps (namely, cloudron/base) because they all use the same base image. And I don't see any world where I wouldn't pay 2.1 GB to have every one of my images be useably-debuggable in case I needed it.

All this said, there may still be opportunities to reduce image sizes by using multi-stage builds to only copy release files (and not source code) to the final image -- but there's no reason for that final image to not be based on cloudron/base.

Oh cool thanks for pointing that out, I hadn't noticed before.

I guess the downsides don't really apply to me right now, but I'm not sure I'd want to enable arbitrary masquerading if there were a bunch of other people on this cloudron.

I still think it's reasonable for users to be able to send with a subaddress of any of their aliases, without opening the floodgates and allowing them to send from any address at all.

@klawitterb Nice thanks for sharing!

it also requires a S3 compatible storage like minio

Maybe this is a good time to bring up the Addon request: blob storage (s3) thread, which proposes a new "s3" addon in the same vein as the database addons that would create and manage s3-compatible buckets for apps automatically upon request in the app definition's manifest.

Some apps, especially recently, store and access data via an s3-compatible api. I think Cloudron should add s3 storage as a new addon that can be requested in CloudronManifest.json to better support packaging such apps.

Some potential apps that would use this addon:

• Outline
• Notea
• Nextcloud

Here are some numbered opinions so you can refer to (refute) them:

1. I think it should be implemented as a Minio server which is set up like the postgresql or mysql addons in that the one instance is configured automatically and can serve a number of different apps. Say, each app that has the s3 addon gets a new random bucket and creds for that bucket.
2. Ideally it can be configured in either local or gateway mode. Users can use local if there is enough storage on their server, or configure a gateway to point to any S3-compatible, Azure, or NAS service. Note that many service providers offer an S3-compatible storage API, so the list of supported providers is already much bigger than just those three, and includes hosts like Backblaze, Linode, DigitalOcean among many others.
3. Figuring out how backups should work will probably be the biggest issue. Here are a few potential solutions:
   a. Just clone / replicate the whole bucket on every backup. Might cost a lot of storage, but that's how backups work today so... See: Bucket Replication Guide
   b. Something-something object versions? Though it might be difficult to, say, take a snapshot of a whole bucket. Bucket Versioning Guide
   c. Minio contributors don't seem very receptive to the general idea of bucket-level shapshots. Taking Backups of MinIO #4398, Snapshots #9376
   d. Just don't support backups at all? (not a fan of this option, but it's technically an option)

Thoughts?

I'm not sure I'd want fancy, distributed filesystems on by default for most apps. I feel like most apps would need custom changes to explicitly support distributed storage, and I'm skeptical that a blanket drop-in distributed-fs solution could meet the performance and reliability needs of the diversity of cloudron users.

I'd rather have multi-node app management than distributed app runtime. Manage all your cloudron nodes and assign apps between them, migrate them etc, but most apps can still only be deployed to one cloudron instance at a time. At least I think this would be a better scaling/ha goal for a v1 implementation.

https://github.com/austinvhuang/openmemex

OpenMemex is an open source, local-first knowledge integration platform (aka "second brain" or "knowledge garden") optimized for automation (including caching and indexing of content) as well as enabling neural network machine learning integrations.

This looks like a pretty neat entry in the ever-growing "second brain apps" arena. Some highlights:

• ... OpenMemex is fully functional as a self-hosted application.
• ... SQLite is the central data storage medium
• ... data is organized automatically by timestamp. Topical/conceptual connections can be automatically linked ...
• ... the focus is on automated persistence, retrieval, and (future work) optimizing compression/consumption of information over UI-heavy notetaking tools.

The author is honest about the status "The implementation is currently at functioning pre-alpha MVP maturity", but it may be something worth watching.

I might take a crack at packaging it for Cloudron.

@LoudLemur By "not self-hostable" I mean the sync service. This is the cloudron forum, in this context I don't really care that it's an open source and cross platform local application -- that's great, but it doesn't help me run the cloud-dependent bits myself.

If I self hosted my own Data Syncing service, where would I put that address in the app?:

https://www.zotero.org/support/sync :

The first step to syncing your Zotero library is to create a Zotero account

Now that Cloudron adds authentication headers with the proxyAuth addon, maybe the easiest solution to integrate authentication is to use those headers in outline. I guess that might be as simple as adding a new file to the /server/routes/auth/providers dir. This would simplify the cloudron package because it wouldn't have to set up and run keycloak.

https://github.com/outline/outline/tree/main/server/routes/auth/providers

Then the only barrier to packaging for cloudron would be the hypothetical s3/minio addon mentioned above.

From what I've seen looking at the frappe/erpnext system design, it looks like a lot of the service complexity comes from their "bench" system tool that provides multitenancy / "environment duplication" capabilities via their backup and restore system. From my experience being able to branch environments is vital for these kinds of tools, but... that's basically cloudron's core competency. With a bit of work cloudron could probably match any missing features that "bench" has over it.

With bench factored out and its features provided by cloudron directly, I think you'd be left with a single python service that needs redis, mariadb/mysql, and postgresql, which cloudron provides as services already.

Notes:

• "frappe" is the company and also the name of the development/deployment framework for erpnext and other applications developed by the same team
• The frappe/frappe-worker docker image referred to in the frappe_docker/compose.yaml file is built from the source in frappe dir in the frappe/frappe repo
• The bench worker --queue short command (and similar) runs scheduler.py:start_worker()
• The queue workers are a Python RQ-based job scheduling/background task system. RQ uses redis queues

There's a post about bridges from 2017 that outlines the general approaches bridges can take. The matrix-puppet-bridge project mentioned in that post now points to two other efforts Sorunome's mx-puppet-*, and tulir's mautrix-*, both of which feature in the matrix-docker-ansible project mentioned above.

It seems all Matrix plugins/bots/bridges/etc connect through the network as Application Services which (in Synapse) is configured manually by setting homeserver.yaml/registration.yaml to include authentication tokens used by the app to authorize its connection to the homeserver. This is part of what the ansible project does.

I think that ansible project is probably the best bet -- at least as a source of truth even if we don't end up using it directly, though extracting usable knowledge about correct configuration from it is not trivial. It has an faq.

---

I did some exploration, tracing the discord bridge's registration.yaml and noted the configuration dependency path through the playbook. I'm only mildly familiar with both ansible & matrix, but I think this helped me get a clearer picture of the configuration required to set it up; perhaps it will also help someone else:

• global secret defined
• discord appservice token is derived
• token used to build discord registration.yaml
• registration.yaml file saved to disk at a location defined here
• discord registration.yaml added to list to mount into synapse container, and added to a list to configure synapse
• the list of registration yaml files is finally expanded into homeserver.yaml to be used by synapse

Also:

• The discord bridge is configured in this config.yaml template, but it's not clear to me how it gets the appservice token to authenticate itself with the homeserver.
• The main tasks executed to install the bridge include downloading the bridge's docker image and installing a systemd service file that runs it.

---

The main issue I see with running the playbook on a Cloudron system is that it expects to be run as root on the base server. E.g. by default it creates docker containers, opens firewalls, sets systemd services etc. I don't think using it blind is a good idea and would almost certainly run into conflicts with Cloudron.

But lets say we could disable the problematic features, and use it to define and manage the bridge containers -- you still have the issue of getting the app service tokens into Cloudron-managed-Synapse's config.

This doesn't sound right. The number of iterations has to be stored in the database, and it is very often stored with the password hash. Changing to a "unique" number doesn't have any meaningful impact on security, aside from being big enough..

The iteration count is designed to be a flexible way to increase the computational effort required for each cracking attempt. This is helpful because Moore's Law is quite real and instead of inventing a new hash every 2 years, users and operators can just bump the iteration count to maintain the same expected level of effort an attacker would have to expend with new hardware.

I recently encountered some issues after I switched from wildcard DNS configuration to provider-based automatic DNS configuration.

When you create an app with provider-DNS configured, cloudron will use your provider's API to automatically create the DNS subdomain record to point to your server so cloudron can serve the app. But if you create the app while you have wildcard DNS configured, no specific subdomain will be created. This is expected.

However, if you create the app under wildcard-DNS, then switch to provider-DNS, then the subdomains that would normally have been created when running under provider-DNS are not created automatically when switching between DNS configurations. This can cause a problem when the user deletes their wildcard DNS record because they assume that Cloudron will use provider-DNS to provision DNS entries for all apps.

I propose that Cloudron add a feature to check the DNS entries and reprovision DNS records as required when the user switches DNS configurations. I could imagine this could be implemented by triggering automatically as soon as the user switches DNS configurations, or as a button on the dashboard to recheck that dns entries for all apps are correctly assigned (including my.).

Project page: https://github.com/tonarino/innernet

Announcement post: https://blog.tonari.no/introducing-innernet

https://github.com/valeriansaliou/sonic

🦔 Fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM.

Lightweight as in their single-thread benchmark which indexed 1M records and sustained 1k rps search traffic never went above 28MB ram (!). There are some limitations: it's schemaless and only supports integer keys, so it may need to be paired with some other data store to replace solr (depending on which features are used^), but if possible it would allow virtually all users to enable email FTS regardless of their server memory capacity. The sonic-channel nodejs library describes the api.

As a data point, sonic is used by ArchiveBox to provide FTS features to archived pages. ArchiveBox is on the App Wishlist.

^ I was gonna look into this myself, but I wasn't able to find the mail app on cloudron git, am I blind or is it not published?