Agate+ (dual protocol server to serve gemini/http from one source)

[timconsidine]

Clarification : when installing multiple instances of the app, slightly different CLI command is needed :
cloudron install --image docker_registry.domain.tld/cloudron-agate-plus:v2 -p GEMINI_PORT=19xx

This avoids conflict with other instances of this app.

If it makes into the App Store, user should be able to adjust the port when launching the GUI installation (obviously cannot test this at the moment).

[timconsidine]

Repo updated
Although cloudron could deploy multiple instances of the app, agate could not handle cert resolution of multiple instances hosted on the same ip address (even though different URL)
Now using an internal wildcard self-signed cert for agate, this is now supported.

I would prefer to use the letsencrypt cert, but it seems that is not possible, as it is not exposed to the app.
The public part can be captured, but the private part is hidden and not accessible.
If I am wrong on this, then maybe [@]staff or other app dev can advise me and show me how.

[robi]

How does the conversion to html happen? What serves http?

[timconsidine]

@robi I looked at some off-the-shelf utilities but for some reason I find difficult to express, I don’t like them. And become worried about dependencies.

So I wrote my own node script to read the .gmi file, which is only plain text markdown after all, and wrap the elements in basic plain html and plain css.

So the html view is generated on the fly in the app and served to a browser.
No external utilities or proxies.
Not even an internal proxy.

[robi]

Cool idea and great simple implementation. I take it your node app then serves the plain HTML/CSS.

Also a great solution for the app package liveness check since gemini runs on another port.

Two birds one stone!

[timconsidine]

@robi
yes, the node component handles the healthcheck and the ongoing rendering of the gemini source in .gmi files into clean html/css for serving to a traditional browser on an http port, while the agate component handles natively serving .gmi files to a gemini client on a tcp port (1965 unless changed by the user because of multiple installations).

It bugged me that markdown source in .gmi is so easy to render into html, so why not do it. I'm late to the gemini space, so there might be a reason, but I haven't found it yet.

I have a lurking suspicion that hard-core gemini fans almost deliberately foster the segregation of http/gemini. Seems silly to me, because the gemini project explicitly states the 2 worlds can co-exist. So let's make them co-exist.

[DidierMalenfant]

Testing it now...

• You need a chmod +x cld.sh in the repo.
• The installation never completes for me. It gets stuck at Waiting for propagation of newsite.malenfant.dev even though the DNS is updated. The app doesn't seem to launch.

Could be something with the DNS on cloudflare... Log shows:

Jul 16 17:28:25 box:dns/waitfordns resolveIp: Checking A for newsite.malenfant.dev at 162.159.38.40
Jul 16 17:28:25 box:dns/waitfordns resolveIp: No A. Checking CNAME for newsite.malenfant.dev at 162.159.38.40
Jul 16 17:28:25 box:dns/waitfordns isChangeSynced: NS davina.ns.cloudflare.com (162.159.38.40) errored when resolve newsite.malenfant.dev (A): Error: queryCname ENODATA newsite.malenfant.dev

but if I ping newsite.malenfant.dev from the command line it gets resolved correctly... odd.

Could it be that the domain is hardcoded in cld.sh?

# Generate self-signed wildcard certificate for all subdomains
echo "Generating self-signed wildcard certificate for *.appx.uk..."
# Extract the base domain (appx.uk) from CLOUDRON_APP_DOMAIN
BASE_DOMAIN="appx.uk"
WILDCARD_DOMAIN="*.${BASE_DOMAIN}"

[timconsidine]

Oh wow, big errors, will fix today
That was a test setup because of problems with self-signed certs. Should have been removed.

[LoudLemur]

@timconsidine I never thought of it as a potential Ghost replacement, but now you mention it, that is definitely one thing Agate+ could do. That dual support for html is cool!

Agate+ could also be used to plonk down when you want to park your domain or as a default page for a cloudron that is running apps on sub-domains.

[LoudLemur]

@timconsidine The geminauts are going to love your simple solution to making html available!

I tried the installation just now. and it went well up to a point. It asked where it should be installed, and then this error happened:

Port GEMINI_PORT: 19
Failed to install app: 400 message: 19 for GEMINI_PORT is not in permitted range in ports

[LoudLemur]

@timconsidine
The way this post is titled might be a bit confusing. It sounds like Agate+ is a browser that can visit gemini:// and html:// sites, rather than a gemini server which supports both gemini and html browsers.

[nebulon]

So while this app being very geeky, it seems to work rather well what @timconsidine has put together. If the individual processes within the app get run by supervisor like in other packages, to avoid just backgrounding them within start.sh I think we can add some basic integration tests and get this published. Especially since it bundles an http server it seems a much better fit for Cloudron out of the box.

Great job so far!

[LoudLemur]

@nebulon maybe test integration with goaccess?

https://goaccess.io/

[timconsidine]

@LoudLemur sorry, yes, it's broken with come code I am fixing.
Standy by please

[timconsidine]

@LoudLemur said in Agate+ (dual protocol gemini/http browser):

The way this post is titled might be a bit confusing.

err, yes, you're right - will fix somehow

[timconsidine]

@LoudLemur said in Agate+ (dual protocol gemini/http browser):

It asked where it should be installed, and then this error happened

try port 1965 ?
maybe you had a keyboard moment : 19 is not right

[LoudLemur]

@timconsidine thanks!

"keyboard moment" ahaha!

fails, with conflicting port. (Already have agate running, which perhaps is using port 1965. Try again though on 1966 (when England won the World Cup!) and it works... until it gives an

App installation error: Installation failed: Unable to pull image docker_registry.domain.tld/cloudron-agate-plus:v2. registry error: {"reason" : "server error"", "statusCode": 500, "json":null}

By the way, this is using the "running multiple instances" method, even though this is the first agate+ instsance.

[timconsidine]

@DidierMalenfant I fixed that hard-coding

I was struggling with self-signed certs for the scenario where a cloudron users wants to deploy multiple instances of agate+, e.g. for different sites on different subdomains. Easy for one domain, not so easy for multiple.

Cloudrono handles this quite easily, but gemini protocol is quite tight on certs, and my 2nd/3rd test instances were failing.

So I have to make a wildcard self-signed cert to handle any subdomain like xxxx.example.com.

There is still some issue, because while xxxx.example.com and zzzz.example.com will be fine under this version, probably aaaa.test.com will not be. Need to do some more testing.

Uploading fixed file to repo in a mo.

[LoudLemur]

Tried just now with a new download. we had an error:

App is being installed.

 => Queued ...
 => Deleting old containers
 => Registering location agateplus.example.com ................
 => Downloading image .....................................

App installation error: Installation failed: Unable to pull image docker_registry.domain.tld/cloudron-agate-plus:v2. registry error: {"reason":"server error","statusCode":500,"json":null}

During the course of testing this, we tried several ports (as it seemed that the default was occupied, perhaps by the previous effort, which was still using it.) We also tried a few sub-domains, as sometimes the port was accepted but the subdomain was already in use eg agateplus.example.com, agateplus2.example.com, agateplus3.example.com.

After we eventually manage to get agate+ running, how do we go about closing those ports again and stopping the "phantom" use of the sub-domains, which never worked anyway?

[timconsidine]

@LoudLemur couple of points raised :

1. cloudron only allows one assigned port like 1965 to one app. Once done, it's occupied/blocked. You need to uninstall the deployed app to clear it.
   From CLI :
   cloudron uninstall --app app.domain.tld
   Or from the app dashboard 'Uninstall' option

2. you gave a code shot which mentions "agateplus.example.com". I'm guessing you obscure the actual app URL which is fine and wise. But it also refers to a docker registry. That's an example.
   Just to clarify, did you build the app and deploy it to your registry ?
   That's necessary for custom apps, and that's probably why the 'unable to pull' message was generated.

I am still tweaking the handling of multiple deployments. Cloudron supports it, and agateplus too, but I am resolving some content caching issues.

Advice for now is stick to one deployment until I solve the cache issue.