Agate+ (dual protocol server to serve gemini/http from one source)

[DidierMalenfant]

Latest update works! Thanks for the fixes.

I'm going to play with it for a bit but I'll leave a few more things I noticed in case you want to address those too:

• If you do git add --chmod=+x cld.sh locally you can then commit the executable flag with your repo and skip a step in the README.
• The cld.sh script should probably have a set -e line at the top (after the #!/bin/bash) to make sure the script stop executing if something goes wrong with one of the commands.
• Currently everything in /app/data is owned by root and I believe agate itself is launched as a root owned app. It's probably safer to make all the files in in /app/data owned by cloudron:cloudron and launch programs with /usr/local/bin/gosu cloudron:cloudron to run as the cloudron user. I might be off here but that was my understanding of recommended behavior looking at the documentation. Someone else can deny or confirm.

Great job on the app!

[timconsidine]

@DidierMalenfant Good points, thank you, will get on to them.

I'm needing to review deployment of multiple instances as it is not working 100%, but hoping to find a "cunning plan".

[LoudLemur]

nearly...

App installation error: Installation failed: Unable to pull image docker_registry.domain.tld/cloudron-agate-plus:v2. registry error: {"reason":"server error","statusCode":500,"json":null}

[timconsidine]

Is that really the address that you’re pulling the image from ? Or you spoofed it to hide the real one in an internet forum (searched and exposed by the great satan google) ?

If it’s the real one, it will never work.
You need to build the image locally
Push it to your docker repo
Then pull it from there.

[LoudLemur]

@timconsidine ha! thanks, tim. will keep trying.

[timconsidine]

Significant update to be uploaded to git.cloudron.io repo
Hopefully I've fixed the issues about installing multiple instances of this app.
Further testing underway before upload.

[timconsidine]

Phew, that was a wild ride!
Major update and refactor of the Agate+ app completed.
Updates pushed to https://git.cloudron.io/timconsidine/cloudron-agate-plus

WHAT CHANGED :

• overcame significant routing challenges arising from combination of (a) gemini clients only work on port 1965 unless you specify the port in the address (gem.domain1.tld:19xx), and (b) Cloudron does not support hostname (SNI) checking for a tcpPort

• multiple instances of Agate+ can now be deployed (e.g. gem.domain1.tld, diamond.domain1.tld, ruby.domain2.tld) and work fully (famous last words )

• certificate creation majorly simplified

CHALLENGES

• the original Agate app supports virtual hosts, so why take a different approach ? Because in that case Cloudron features of managing users access to apps and their content would be compromised ( user would get access to all sites in the virtual host)

• gemini clients (Lagrange, Deedum, Amfora) will ONLY connect to port 1965 unless you put the port in the address (gem.domain1.tld:1966). I think this is cumbersome and unacceptable for daily usage.

• Cloudron does not currently support hostname/SNI checking of traffic on a tcpPort, just sends all to port 1965, because that's what the client instructed. But without the hostname check, gemini client requests go to the wrong instance.

SOLUTION
A little bit "creative".

• Agate+ provides its own proxy in the app

• the first "master" deployment on port 1965 not only serves out content to clients for its own domain (gem.domain1.tld), but also runs the proxy so, when a client makes a non-port-specified request to ruby.domain2.tld and Cloudron sends this to 1965 because that's what the client said to do, the proxy fetches the correct content from ruby.domain2.tld (running on e.g. port 1966) and returns it to the client, even though the user didn't specify 1966 in the address and the client did not request it from 1966. Cool, hey?

• additional deployments of Agate+ recognise that they are not the first Agate+ (i.e. not running on port 1965) and function as pure gemini servers (in fact, proxy features actually removed automatically on deployment)

• there is a small setup step for additional deployments, but I think I have made this really easy : see POSTINSTALL.md

Is this complex ?
Yes. No. Well, a little, but not really, unless you really want to deep dive into it. Otherwise just follow the instructions.

Is this essential ?
Yes. Multiple Gemini servers simply cannot function on a single Cloudron instance (common IP address) without some form of reverse proxy because Gemini clients will ONLY send to port 1965 if no port is put in the address. And Cloudron does not yet support hostname/SNI routing of tcpPort (only nginx https traffic supports hostname routing).

I think this is the neatest solution to a potentially incompatible set of operating conditions (gemini client behaviour, lack of Cloudron support for hostname tcpPort routing).

I will be testing more, but you can never have enough testing, so feedback much appreciated.

Agate/gemini protocol has got me excited about potential uses, hopefully it can help others too.

[LoudLemur]

amazing effort and engrossing to read your development report. thank you! will test later...

[nebulon]

@timconsidine nice work, so basically one app which runs on the default port does the same thing for gemini, what nginx on port 80/443 does for http. The probably only more elegant way to get around this would be to run a gemini reverse proxy directly as a service on Cloudron. Given that gemini:// is quite the geeky fringe protocol for the moment, I think your solution is best suited. If you want, I will look into your package and see what further needs polishing for adopting into the main app libarary then.

[nebulon]

Just some random things, which popped up while glancing over the current package:

Instead of https://git.cloudron.io/timconsidine/cloudron-agate-plus/-/blob/master/start.sh?ref_type=heads#L52 you could just place the logfile into /run/.. and Cloudron will actually run logrotate for you out of the box: https://docs.cloudron.io/packaging/cheat-sheet/#logging

For the http server, which is currently run as a background job of the start.sh via node /app/code/proxy.js & it would be better to use supervisord for this and also the gemini server. The reason being that if one of the processes crash, they would get autorestarted by supervisor. Without that the app would require a manual restart by an admin to restart that node process internally.

Otherwise this is already looking really good!

[timconsidine]

Thank you for the pointers @nebulon
I’m away delivering aid for Ukraine, so will get on to this when I’m back.

[LoudLemur]

I haven't got round to testing the latest agate+ yet, but one thing I can say for sure:

It was far easier to convert our astro starlight documentation to gemini-protocol format and publish them using agate as a gemlog than it was to try and publish via cubby, which we still haven't managed!

[timconsidine]

@LoudLemur
I will make and publish an image after the final tweaks.

[timconsidine]

New version of the app uploaded to https://git.cloudron.io/timconsidine/cloudron-agate-plus

CHANGELOG :

• implemented @nebulon suggestion about logs rotation
• implemented @nebulon suggestion about using supervisord
• added /admin page for http mode to provide basic UI for managing content (no need of File Manager) to create and edit .gmi files and upload images

App is now a quick way to host a basic site (no fancy html/css/js blah blah) on both Gemini and HTTPS from one set of source content.

In-app content editing via /admin in http browser means quick and easy to update site content, even from your mobile phone 'on-the-go'.

REMEMBER to change the default user/password in /app/data/.env using File Manager and RESTART the app.
[it's in the app dashboard checklist].

[timconsidine]

I have uploaded a built docker image for those who want to test without building :

cloudron install —image tcmbp132021/cloudron-agate-plus:v6.36

Not sure it helps much because you still need the Cloudron CLI with your login to your Cloudron instance, and the CloudronManifest.json locally from the cloudron git repo.

[robi]

@timconsidine so far the easiest way we found was last year with @Kubernetes EasyInstall script, as you might recall.

[timconsidine]

@robi yep, I need to remind myself about it.
I have a vague distant feeling that it didn’t deal with the core issue that less techie users would still struggle with it. But hey, grey hair, don’t even recall my name some days, let me review it. Maybe the truth is I didn’t understand it at the time

[timconsidine]

@LoudLemur hey, did you see my post about CCAI ?
You can now use https://ccai.appx.uk to deploy custom apps to your own cloudron.
Anonymous - no login details saved.

Details : https://forum.cloudron.io/post/110907

[LoudLemur]

@timconsidine Hey, thanks! No, I missed that. I shall take a look.

WOW!

So all we need is to link a publicly accessible git and Docker image, and it does the magic for us?

Amazing! Surely, there must be more to it than that...?

[timconsidine]

@LoudLemur basically yes, simple as that, "magic" is behind the scenes

I'm cautious about the word "link", could be understood differently.

CCAI does this :

1. asks for :

• your cloudron domain e.g. my.cloudron-instance.tld (never stored)
• your user name (never stored)
• your password (never stored)
• your 2FA code (if you've enabled this, otherwise leave blank)
  If it logs in successfully,

2. asks for :

• git repo of the target custom app, e.g. https://git.cloudron.io/timconsidine/cloudron-agate-plus
• prebuilt docker image, e.g. tcmbp132021/cloudron-agate-plus:v6.36
• location where you want the custom app to deploy, e.g. your-gemini.your-domain.tld
• any port binding needed : optional in most cases, also optional for your first installation of Agate+, but subsequent instances need "GEMINI_PORT=1967" or higher number

So it's a bit more than just linking, but it's much easier than installing and using CLI, installing and using docker to build, installing and using a docker registry.