Agate+ (dual protocol server to serve gemini/http from one source)

[timconsidine]

@LoudLemur said in Agate+ (dual protocol gemini/http browser):

It asked where it should be installed, and then this error happened

try port 1965 ?
maybe you had a keyboard moment : 19 is not right

[LoudLemur]

@timconsidine thanks!

"keyboard moment" ahaha!

fails, with conflicting port. (Already have agate running, which perhaps is using port 1965. Try again though on 1966 (when England won the World Cup!) and it works... until it gives an

App installation error: Installation failed: Unable to pull image docker_registry.domain.tld/cloudron-agate-plus:v2. registry error: {"reason" : "server error"", "statusCode": 500, "json":null}

By the way, this is using the "running multiple instances" method, even though this is the first agate+ instsance.

[timconsidine]

@DidierMalenfant I fixed that hard-coding

I was struggling with self-signed certs for the scenario where a cloudron users wants to deploy multiple instances of agate+, e.g. for different sites on different subdomains. Easy for one domain, not so easy for multiple.

Cloudrono handles this quite easily, but gemini protocol is quite tight on certs, and my 2nd/3rd test instances were failing.

So I have to make a wildcard self-signed cert to handle any subdomain like xxxx.example.com.

There is still some issue, because while xxxx.example.com and zzzz.example.com will be fine under this version, probably aaaa.test.com will not be. Need to do some more testing.

Uploading fixed file to repo in a mo.

[LoudLemur]

Tried just now with a new download. we had an error:

App is being installed.

 => Queued ...
 => Deleting old containers
 => Registering location agateplus.example.com ................
 => Downloading image .....................................

App installation error: Installation failed: Unable to pull image docker_registry.domain.tld/cloudron-agate-plus:v2. registry error: {"reason":"server error","statusCode":500,"json":null}

During the course of testing this, we tried several ports (as it seemed that the default was occupied, perhaps by the previous effort, which was still using it.) We also tried a few sub-domains, as sometimes the port was accepted but the subdomain was already in use eg agateplus.example.com, agateplus2.example.com, agateplus3.example.com.

After we eventually manage to get agate+ running, how do we go about closing those ports again and stopping the "phantom" use of the sub-domains, which never worked anyway?

[timconsidine]

@LoudLemur couple of points raised :

1. cloudron only allows one assigned port like 1965 to one app. Once done, it's occupied/blocked. You need to uninstall the deployed app to clear it.
   From CLI :
   cloudron uninstall --app app.domain.tld
   Or from the app dashboard 'Uninstall' option

2. you gave a code shot which mentions "agateplus.example.com". I'm guessing you obscure the actual app URL which is fine and wise. But it also refers to a docker registry. That's an example.
   Just to clarify, did you build the app and deploy it to your registry ?
   That's necessary for custom apps, and that's probably why the 'unable to pull' message was generated.

I am still tweaking the handling of multiple deployments. Cloudron supports it, and agateplus too, but I am resolving some content caching issues.

Advice for now is stick to one deployment until I solve the cache issue.

[DidierMalenfant]

Latest update works! Thanks for the fixes.

I'm going to play with it for a bit but I'll leave a few more things I noticed in case you want to address those too:

• If you do git add --chmod=+x cld.sh locally you can then commit the executable flag with your repo and skip a step in the README.
• The cld.sh script should probably have a set -e line at the top (after the #!/bin/bash) to make sure the script stop executing if something goes wrong with one of the commands.
• Currently everything in /app/data is owned by root and I believe agate itself is launched as a root owned app. It's probably safer to make all the files in in /app/data owned by cloudron:cloudron and launch programs with /usr/local/bin/gosu cloudron:cloudron to run as the cloudron user. I might be off here but that was my understanding of recommended behavior looking at the documentation. Someone else can deny or confirm.

Great job on the app!

[timconsidine]

@DidierMalenfant Good points, thank you, will get on to them.

I'm needing to review deployment of multiple instances as it is not working 100%, but hoping to find a "cunning plan".

[LoudLemur]

nearly...

App installation error: Installation failed: Unable to pull image docker_registry.domain.tld/cloudron-agate-plus:v2. registry error: {"reason":"server error","statusCode":500,"json":null}

[timconsidine]

Is that really the address that you’re pulling the image from ? Or you spoofed it to hide the real one in an internet forum (searched and exposed by the great satan google) ?

If it’s the real one, it will never work.
You need to build the image locally
Push it to your docker repo
Then pull it from there.

[LoudLemur]

@timconsidine ha! thanks, tim. will keep trying.

[timconsidine]

Significant update to be uploaded to git.cloudron.io repo
Hopefully I've fixed the issues about installing multiple instances of this app.
Further testing underway before upload.

[timconsidine]

Phew, that was a wild ride!
Major update and refactor of the Agate+ app completed.
Updates pushed to https://git.cloudron.io/timconsidine/cloudron-agate-plus

WHAT CHANGED :

• overcame significant routing challenges arising from combination of (a) gemini clients only work on port 1965 unless you specify the port in the address (gem.domain1.tld:19xx), and (b) Cloudron does not support hostname (SNI) checking for a tcpPort

• multiple instances of Agate+ can now be deployed (e.g. gem.domain1.tld, diamond.domain1.tld, ruby.domain2.tld) and work fully (famous last words )

• certificate creation majorly simplified

CHALLENGES

• the original Agate app supports virtual hosts, so why take a different approach ? Because in that case Cloudron features of managing users access to apps and their content would be compromised ( user would get access to all sites in the virtual host)

• gemini clients (Lagrange, Deedum, Amfora) will ONLY connect to port 1965 unless you put the port in the address (gem.domain1.tld:1966). I think this is cumbersome and unacceptable for daily usage.

• Cloudron does not currently support hostname/SNI checking of traffic on a tcpPort, just sends all to port 1965, because that's what the client instructed. But without the hostname check, gemini client requests go to the wrong instance.

SOLUTION
A little bit "creative".

• Agate+ provides its own proxy in the app

• the first "master" deployment on port 1965 not only serves out content to clients for its own domain (gem.domain1.tld), but also runs the proxy so, when a client makes a non-port-specified request to ruby.domain2.tld and Cloudron sends this to 1965 because that's what the client said to do, the proxy fetches the correct content from ruby.domain2.tld (running on e.g. port 1966) and returns it to the client, even though the user didn't specify 1966 in the address and the client did not request it from 1966. Cool, hey?

• additional deployments of Agate+ recognise that they are not the first Agate+ (i.e. not running on port 1965) and function as pure gemini servers (in fact, proxy features actually removed automatically on deployment)

• there is a small setup step for additional deployments, but I think I have made this really easy : see POSTINSTALL.md

Is this complex ?
Yes. No. Well, a little, but not really, unless you really want to deep dive into it. Otherwise just follow the instructions.

Is this essential ?
Yes. Multiple Gemini servers simply cannot function on a single Cloudron instance (common IP address) without some form of reverse proxy because Gemini clients will ONLY send to port 1965 if no port is put in the address. And Cloudron does not yet support hostname/SNI routing of tcpPort (only nginx https traffic supports hostname routing).

I think this is the neatest solution to a potentially incompatible set of operating conditions (gemini client behaviour, lack of Cloudron support for hostname tcpPort routing).

I will be testing more, but you can never have enough testing, so feedback much appreciated.

Agate/gemini protocol has got me excited about potential uses, hopefully it can help others too.

[LoudLemur]

amazing effort and engrossing to read your development report. thank you! will test later...

[nebulon]

@timconsidine nice work, so basically one app which runs on the default port does the same thing for gemini, what nginx on port 80/443 does for http. The probably only more elegant way to get around this would be to run a gemini reverse proxy directly as a service on Cloudron. Given that gemini:// is quite the geeky fringe protocol for the moment, I think your solution is best suited. If you want, I will look into your package and see what further needs polishing for adopting into the main app libarary then.

[nebulon]

Just some random things, which popped up while glancing over the current package:

Instead of https://git.cloudron.io/timconsidine/cloudron-agate-plus/-/blob/master/start.sh?ref_type=heads#L52 you could just place the logfile into /run/.. and Cloudron will actually run logrotate for you out of the box: https://docs.cloudron.io/packaging/cheat-sheet/#logging

For the http server, which is currently run as a background job of the start.sh via node /app/code/proxy.js & it would be better to use supervisord for this and also the gemini server. The reason being that if one of the processes crash, they would get autorestarted by supervisor. Without that the app would require a manual restart by an admin to restart that node process internally.

Otherwise this is already looking really good!

[timconsidine]

Thank you for the pointers @nebulon
I’m away delivering aid for Ukraine, so will get on to this when I’m back.

[LoudLemur]

I haven't got round to testing the latest agate+ yet, but one thing I can say for sure:

It was far easier to convert our astro starlight documentation to gemini-protocol format and publish them using agate as a gemlog than it was to try and publish via cubby, which we still haven't managed!

[timconsidine]

@LoudLemur
I will make and publish an image after the final tweaks.

[timconsidine]

New version of the app uploaded to https://git.cloudron.io/timconsidine/cloudron-agate-plus

CHANGELOG :

• implemented @nebulon suggestion about logs rotation
• implemented @nebulon suggestion about using supervisord
• added /admin page for http mode to provide basic UI for managing content (no need of File Manager) to create and edit .gmi files and upload images

App is now a quick way to host a basic site (no fancy html/css/js blah blah) on both Gemini and HTTPS from one set of source content.

In-app content editing via /admin in http browser means quick and easy to update site content, even from your mobile phone 'on-the-go'.

REMEMBER to change the default user/password in /app/data/.env using File Manager and RESTART the app.
[it's in the app dashboard checklist].

[timconsidine]

I have uploaded a built docker image for those who want to test without building :

cloudron install —image tcmbp132021/cloudron-agate-plus:v6.36

Not sure it helps much because you still need the Cloudron CLI with your login to your Cloudron instance, and the CloudronManifest.json locally from the cloudron git repo.