Cloudron on a Raspberry pi?
-
@nebulon huh, that makes sense, but I thought this was supposed to be done by cloudron during setup? I don’t remember having to manually set the hostname in any other cloudron setup before, but I might just have forgotten...
-
It looks like the blocking issue is the DNS does not work? Does
host my.pi.mydomain.comandhost my.pi.mydomain.com 127.0.0.1work on the pi? -
If there's nothing private in it; could you post your /etc/hosts file?
-
Yup, so I'm testing it again from scratch.
@jamesgallagher, here's the entire contents of my
/etc/hostsright after runningcloudron-setup, but before setting up the domain in the browser:127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts@girish, the record is already there with the previous IP, and right now I'm just after
cloudron-setup, but beforesetupdns.htmlon the browser. I've ran a bunch of commands, here are the results:host my.pi.<mydomain.com>- not foundhost my.pi.<mydomain.com> 127.0.0.1- not foundhost www.google.com 127.0.0.1- ok (huh, unbound seems to be somewhat working)host code.<mydomain.com> 127.0.0.1(gitea instance on production cloudron) - OK!host my.<mydomain.com> 127.0.0.1(production cloudron admin) - NOT FOUND! (wat?)
And I ran
host my.pi.<mydomain.com>on my local machine, outside the pi, and it works and returns the old record (192.168.0.109from my previous attempt).I'm now setting up DNS in the browser, just to check what happens, I'll re-run the commands and post here.
-
This post is deleted! -
Okay, so some weird stuff seems to be going on with the DNS lookup thing, I think. Here's what happened after running cloudron domain setup (
setupdns.htmlon the browser):I ran it pointing to a subdomain of my domain (
pi.<mydomain.com>), and pinted it to a network interface (wlan0, in this pi's case), as @girish suggested. So I'm tailing/home/yellowtent/platformdata/logs/box.log, and here's what I see:2020-10-07T23:43:30.489Z box:dns/waitfordns waitForDns (try 20): my.pi.<mydomain.com> to be 192.168.0.6 in zone <mydomain.com> 2020-10-07T23:43:30.497Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has A record at 205.251.196.70 2020-10-07T23:43:30.498Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has A record at 205.251.199.85 2020-10-07T23:43:30.500Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has A record at 205.251.192.234 2020-10-07T23:43:30.501Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has A record at 205.251.195.139 2020-10-07T23:43:35.499Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has CNAME record at 205.251.196.70 2020-10-07T23:43:35.501Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has CNAME record at 205.251.199.85 2020-10-07T23:43:35.503Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has CNAME record at 205.251.192.234 2020-10-07T23:43:35.504Z box:dns/waitfordns resolveIp: Checking if my.pi.<mydomain.com> has CNAME record at 205.251.195.139 2020-10-07T23:43:35.516Z box:dns/waitfordns isChangeSynced: NS ns-1094.awsdns-08.org (205.251.196.70) errored when resolve my.pi.<mydomain.com> (A): Error: queryCname ENODATA my.pi.<mydomain.com> 2020-10-07T23:43:35.516Z box:dns/waitfordns waitForDns: my.pi.<mydomain.com> not done ns: ["ns-1094.awsdns-08.org","ns-1877.awsdns-42.co.uk","ns-234.awsdns-29.com","ns-907.awsdns-49.net"] 2020-10-07T23:43:35.525Z box:dns/waitfordns isChangeSynced: NS ns-1877.awsdns-42.co.uk (205.251.199.85) errored when resolve my.pi.<mydomain.com> (A): Error: queryCname ENODATA my.pi.<mydomain.com> 2020-10-07T23:43:35.629Z box:dns/waitfordns isChangeSynced: NS ns-234.awsdns-29.com (205.251.192.234) errored when resolve my.pi.<mydomain.com> (A): Error: queryCname ENODATA my.pi.<mydomain.com> 2020-10-07T23:43:35.635Z box:dns/waitfordns isChangeSynced: NS ns-907.awsdns-49.net (205.251.195.139) errored when resolve my.pi.<mydomain.com> (A): Error: queryCname ENODATA my.pi.<mydomain.com>So I go to my LOCAL machine to check the records:
$ host my.pi.<mydomain.com> my.pi.<mydomain.com> has address 192.168.0.6 $ host my.pi.<mydomain.com> ns-1094.awsdns-08.org Using domain server: Name: ns-1094.awsdns-08.org Address: 2600:9000:5304:4600::1#53 Aliases: my.pi.<mydomain.com> has address 192.168.0.6 $ host ns-1094.awsdns-08.org ns-1094.awsdns-08.org has address 205.251.196.70 ns-1094.awsdns-08.org has IPv6 address 2600:9000:5304:4600::1 $ host my.pi.<mydomain.com> 205.251.196.70 ;; connection timed out; no servers could be reachedSo it seems when I look up the records using Amazon's DNS server IP's instead of their names, I can't look them up! EVEN ON MY LOCAL MACHINE! I have no idea what might be going on. Then, going back to the pi:
$ host my.pi.<mydomain.com> ns-1094.awsdns-08.org Using domain server: Name: ns-1094.awsdns-08.org Address: 2600:9000:5304:4600::1#53 Aliases: my.pi.<mydomain.com> has address 192.168.0.6 $ host my.pi.<mydomain.com> 205.251.196.70 ;; connection timed out; no servers could be reached $ host my.pi.<mydomain.com> 127.0.0.1 ;; connection timed out; no servers could be reached $ host my.pi.<mydomain.com> localhost ;; connection timed out; no servers could be reachedSo... Yeah, this is where I'm at right now. Totally at a loss hahah! NO IDEA what might possibly be going on now...
-
@malvim Check the unbound logs maybe? I think in
journalctl -u unbound -
@girish meh, I can't see anything wrong
Oct 08 03:53:12 ubuntu systemd[1]: Started Unbound DNS Resolver. Oct 08 03:53:12 ubuntu unbound[29715]: [29715:0] notice: init module 0: subnet Oct 08 03:53:12 ubuntu unbound[29715]: [29715:0] notice: init module 1: validator Oct 08 03:53:12 ubuntu unbound[29715]: [29715:0] notice: init module 2: iterator Oct 08 03:53:12 ubuntu unbound[29715]: [29715:0] info: start of service (unbound 1.6.7). Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: service stopped (unbound 1.6.7). Oct 08 03:54:49 ubuntu systemd[1]: Stopping Unbound DNS Resolver... Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: server stats for thread 0: 35 queries, 1 answers from cache, 34 recursions, 0 prefetch, 0 rejected by ip ratelimiting Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: server stats for thread 0: requestlist max 18 avg 10.6765 exceeded 0 jostled 0 Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: average recursion processing time 12.169607 sec Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: histogram of recursion processing times Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: [25%]=0.371371 median[50%]=1.375 [75%]=25.3333 Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: lower(secs) upper(secs) recursions Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: 0.131072 0.262144 3 Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: 0.262144 0.524288 3 Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: 0.524288 1.000000 1 Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: 1.000000 2.000000 4 Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: 16.000000 32.000000 3 Oct 08 03:54:49 ubuntu systemd[1]: Stopped Unbound DNS Resolver. Oct 08 03:54:49 ubuntu unbound[29715]: [29715:0] info: 32.000000 64.000000 3 -- Reboot -- Oct 08 03:54:55 ubuntu systemd[1]: Started Unbound DNS Resolver. Oct 08 03:54:56 ubuntu unbound[2090]: [2090:0] notice: init module 0: subnet Oct 08 03:54:56 ubuntu unbound[2090]: [2090:0] notice: init module 1: validator Oct 08 03:54:56 ubuntu unbound[2090]: [2090:0] notice: init module 2: iterator Oct 08 03:54:56 ubuntu unbound[2090]: [2090:0] info: start of service (unbound 1.6.7).I disabled ipv6 on boot just to be sure, but I see nothing. completely at a loss here.
-
I'd love to test it on another network, but can't really go anywhere else right now...
-
So I found this code in
start.sh:echo "==> Setting up unbound" # DO uses Google nameservers by default. This causes RBL queries to fail (host 2.0.0.127.zen.spamhaus.org) # We do not use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!) # We listen on 0.0.0.0 because there is no way control ordering of docker (which creates the 172.18.0.0/16) and unbound # If IP6 is not enabled, dns queries seem to fail on some hosts. -s returns false if file missing or 0 size ip6=$([[ -s /proc/net/if_inet6 ]] && echo "yes" || echo "no") cp -f "${script_dir}/start/unbound.conf" /etc/unbound/unbound.conf.d/cloudron-network.confIt says unbound sometimes doesn't resolve names if ipv6 is not enabled? I had it enabled, and then disabled it thinking it might pose problems... The code seems to just set ip6 variable and nothing else, not sure whether this might be related to the problems I'm having or not.
-
@malvim I don't think IPv6 is the issue. IIRC, unbound won't even start without that flag. In your case, unbound is running.
So, I would debug this step by step: First, is DNS working at all? You can do
host my.pi.domain.com 8.8.8.8. This uses Google DNS and skips unbound altogether. Does this work? If not, we have to fix this first. Are you able to run the above command on other machines on the same network as the PI? -
@girish said in Cloudron on a Raspberry pi?:
I'm in the middle of running installation without unbound, and everything worked. I'll re-run it now with unbound again, but
host my.pi.domain.comdoes NOT work from any machine inside my home network, and does work from my main cloudron server, so I think there's more to debug here, unfortunately.
host my.pi.domain.comDOES work from my local machine, but not using Google's server, which is kind of weird.I guess I'll have to solve this issue (maybe with my ISP?) before proceeding with cloudron on the pi. Sucks.
-
Okay, so this was related to my ISP. After a few calls with tech support and buying a new router, DNS issues are gone and I was able to install cloudron from start to finish!
Now I ssh into the pi and, as expected, the first containers (mysql, turn, sftp, graphite) are continuosly restarting, since they're installed from the production amd64 images which won't work.
So now I'm thinking I'll just clone, say, the mysql addon (from
https://git.cloudron.io/cloudron/mysql-addon.git), try to build it for the pi, publish the image on my repo and try to use it, see if mysql works, and go from there. I'll try to do that tonight and get back to you guys.What do you guys say? @girish ? Is that the right path to start on?
Baby steps.
-
@malvim w00t, awesome progress. If you can create a branch of your box changes, that will help others as well. Yes, starting with one of the addon containers is a good start. They all have automatic tests, it's easy to run them as well.
-
Ah, just noticed I might need a new cloudron baseimage, right? Hahah! This is gonna be fun
-
Hey,
So I'm building
docker-base-imageand the other initial docker containers, but I can't seem to find them on cloudron's git repo.I found this code in
infra_version.js:'images': { 'turn': { repo: 'cloudron/turn', tag: 'cloudron/turn:1.1.0@sha256:e1dd22aa6eef5beb7339834b200a8bb787ffc2264ce11139857a054108fefb4f' }, 'mysql': { repo: 'cloudron/mysql', tag: 'cloudron/mysql:2.3.1@sha256:c1145d43c8a912fe6f5a5629a4052454a4aa6f23391c1efbffeec9d12d72a256' }, 'postgresql': { repo: 'cloudron/postgresql', tag: 'cloudron/postgresql:3.1.0@sha256:261c38d332a20cd4160930d7395fd342496159e94c522d92fde8163c680adc98' }, 'mongodb': { repo: 'cloudron/mongodb', tag: 'cloudron/mongodb:3.0.0@sha256:59e50b1f55e433ffdf6d678f8c658812b4119f631db8325572a52ee40d3bc562' }, 'redis': { repo: 'cloudron/redis', tag: 'cloudron/redis:2.3.0@sha256:0e31ec817e235b1814c04af97b1e7cf0053384aca2569570ce92bef0d95e94d2' }, 'mail': { repo: 'cloudron/mail', tag: 'cloudron/mail:2.10.0@sha256:3aff92bfc85d6ca3cc6fc381c8a89625d2af95cc55ed2db692ef4e483e600372' }, 'graphite': { repo: 'cloudron/graphite', tag: 'cloudron/graphite:2.3.0@sha256:b7bc1ca4f4d0603a01369a689129aa273a938ce195fe43d00d42f4f2d5212f50' }, 'sftp': { repo: 'cloudron/sftp', tag: 'cloudron/sftp:2.0.2@sha256:cbd604eaa970c99ba5c4c2e7984929668e05de824172f880e8c576b2fb7c976d' } }And I could find the database addons (
mysql-addon.postgresql-addonandmongodb-addon), but I'm not sure they're what you use to build these images, and also I wasn't able to find anything to do with turn, redis, sftp, graphite...I was able to build
mysql-addonon top of my arm64 base image, but can't seem to find the others in order to keep going. Are these Dockerfiles somewhere else we have access to? -
@malvim said in Cloudron on a Raspberry pi?:
Thanks for the replies, guys. I'll take a stab at it, tomorrow or on Friday, and see what it looks like.
I honestly had about 0 confidence that we could go from there to :
@malvim said in Cloudron on a Raspberry pi?:
I was able to install cloudron from start to finish!
in like a month...
Hat's off
-
The mysql-addon repo is at https://git.cloudron.io/cloudron/mysql-addon and according to that repo naming convention also the others.
The sftp addon is at https://git.cloudron.io/cloudron/docker-sftp (not sure why that repo name ended up like it is)
On a side topic, does anyone know of some good naming convention for docker images when it comes to supporting multi-arch?
For example:
cloudron/mysql-amd64:2.3.1(amd64)
cloudron/mysql-arm64:2.3.1(arm64) -
@nebulon said in Cloudron on a Raspberry pi?:
On a side topic, does anyone know of some good naming convention for docker images when it comes to supporting multi-arch?
I think the preferred way is to not have the architecture in the name or tag, but rather populate the manifest properly: https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/
-
@malvim we build images manually using docker build and push them out. When building you have to tag it locally as cloudron/mysqladdontest (just
docker build -t cloudron/mysqladdontest.). Do the tests work? -
@nebulon thanks! I was able to find most of them, but still missing graphite, turn and mail. Are those regular docker containers as well, and if so, are their dockerfiles published on git.cloudron? A search for those terms did not yield any meaningfule results for me.
On your side topic: I went looking for the arm64 version of the base ubuntu image, and it's published as multi-platform, so I guess there's no need to publish under another name, like @fbartels already answered. What I did have to do, though, was to strip the sha256 hash from the
FROMstatement. When I tried building on the rpi keeping the hash, docker selected the amd64 version of the image. Not sure how we'd go about making sure the image passes the integrity check (which I assume is why the has is there in the first place) while also being able to build for different architectures.@mehdi Man, I appreciate it! It was almost about a month just to get my hands on one, then weird networking stuff, trying to sneak in a few hours of banging on this problem in between work... Haha! A lot of fun though! So thanks for the kudos, they're highly appreciated, especially coming from you!
@girish yeah, I'm publishing to a private docker container (which I packaged together with verdaccio in a custom app just to have docker and npm private registers haha), so that part is good, I'm tagging them pointing to it and it's all good.
Haven't run the tests yet, though.
-
Okay, so another question: Do we depend on mongodb being 4.0, or can we upgrade it to 4.2?
It seems mongodb 4.0 on arm64 only has support for ubuntu 16.04 (as per https://docs.mongodb.com/manual/administration/production-notes/)
-
https://git.cloudron.io/cloudron/turn-addon is the turn addon and https://git.cloudron.io/cloudron/docker-graphite/ is graphite (the names are a bit here and there). The mail server is not open, I have sent you an invite though.
-
@girish Thanks, I'll check them out!
-
@malvim said in Cloudron on a Raspberry pi?:
Okay, so another question: Do we depend on mongodb being 4.0, or can we upgrade it to 4.2?
yeah, we can bump it. But it requires the usual round of testing against all apps.
-
@girish Cool. I'll bump it to 4.2 (4.4 needs ubuntu 20.04) and keep pushing hehehe. I'll ping people here when I have something we can run tests on.
-
@malvim Are you using buildx? We bumped 5.6.2 to have docker 19 so we can have buildx
-
@girish Not yet, I tried installing it for cross-platform building, but had some errors and didn't want to waste time, so I'm now just building the images from the pi itself. Later today I might have to start using it again for installing cloudron, so that might become enought of a hassle that I try again.
Anyone here with experience building for other architectures using buildx?
-
Hey, all.
Happy to say this is now going on on my raspberry pi:
I have NO idea wheter stuff is really working hahaha.
I'll probably choose a simple app and build it for arm64, then try to install it from command line and run tests, maybe?Not sure how to run tests against addons, or even apps. Is there some documentation around about this? @girish, could you point me somewhere? Thanks!
-
@malvim Wow, that's some incredible progress! If the status indicator is green, it's pretty sure that the addon containers are responding to health checks!
For the test for the addons, there is a test/ inside the repo of each addon. You can just do
npm installandnpm test. -
To be clearer, like this (say with postgres addon). The tests will always test the latest
cloudron/{addonname}testimage.$ docker build -t cloudron/postgresqladdontest . $ cd test $ npm install # only have to do this once $ npm test > postgresql-addon@1.0.0 test /home/girish/yellowtent/postgresql-addon > mocha --bail ./test/test.js Postgresql Addon Error response from daemon: network with name cloudron already exists auth ✓ fails without access_token ✓ fails with invalid access_token ✓ succeeds add database ✓ succeeds (410ms) ✓ succeeds when added again remove database ✓ succeeds (140ms) use the database ✓ can create extension (49ms) ✓ can create table foo ✓ can insert into table foo ✓ can read from table foo ✓ restart (5241ms) ✓ can read from table foo backup and restore ✓ succeeds to create backup (392ms) ✓ succeeds to create new database (422ms) ✓ succeeds to clear new database (449ms) ✓ succeeds to restore backup (875ms) ✓ succeeds to check restore data (47ms) restore of invalid dump fails ✓ succeeds to create backup (175ms) ✓ succeeds to clear new database (434ms) ✓ fails to restore backup (174ms) restore of existing dump ✓ succeeds (1895ms) 21 passing (35s) -
@malvim Our e2e tests use this app - https://git.cloudron.io/cloudron/test-app . You can just build it like any other app and deploy it on Cloudron.
-
Hey, @girish!
So I was trying to run the tests on the raspberry pi, had some failures, but they're also failing on my regular amd64 laptop with ubuntu, so I'm not sure what's going on.
I tried specifically the postgresql addon like you mentioned, with no changes, the
authandadd databasetests pass, but I always get a timeout on theremove databasetest, every time. Mail and sftp addon also fail at different points, so I'm not trusting the failed tests on the rpi.I thought about trying to run the tests on my production cloudron server, but not sure I should.
Have you ever been through this?
-
I think ARM is the future so I'll be following this thread, love what I'm reading so far. Good stuff.
@malvim But if your test is failing on x86 and ARM, it probably isn't a CPU related issue IMO.
-
@Lonk Yup, those are my thoughts as well. The thing is, it seems the devs are able to run the tests themselves, so it seems there are no obvious problems anywhere heheh.
-
Okay, so I ran the tests on my production cloudron and they all passed. It seems they don't run on a regular machine previously to installing cloudron? Is that true? And SHOULD they? Maybe I'm testing it wrong heheh.
But it would be nice if we could just run the tests outside of any cloudron installation, so I could test the images themselves, separately, on arm64 before going with another full install.
What do you guys say?
-
@malvim said in Cloudron on a Raspberry pi?:
Okay, so I ran the tests on my production cloudron and they all passed. It seems they don't run on a regular machine previously to installing cloudron? Is that true? And SHOULD they? Maybe I'm testing it wrong heheh.
Your steps for testing it on local are different right? Since production is already running. What are your testing steps in each scenario?
But it would be nice if we could just run the tests outside of any cloudron installation, so I could test the images themselves, separately, on arm64 before going with another full install.
What do you guys say?
I'd say that's possible. We could just use Docker itself? That's what I do locally.
Anyway, what Rasberry Pi are you developing this on? I'll go run out and get one to see if I run into similar issues.
️ Very interested in Cloudron on ARM in 2030! -
@Lonk said in Cloudron on a Raspberry pi?:
Your steps for testing it on local are different right? Since production is already running. What are your testing steps in each scenario?
Yeah, I'm doing what @girish outlined in his latest response in this thread: clone an addon repo,
docker buildthe image tagging it with whatever name the test runner uses (cloudron/postgresqladdontestin this case),npm installandnpm test.It turns out I'm getting failures in all that I've tried so far, so I don't feel comfortable saying I "have successfully built the images" when I can't run the tests. But I can't run them on my laptop as well, so who knows? Haha!
Anyway, what Rasberry Pi are you developing this on? I'll go run out and get one to see if I run into similar issues.
️ Very interested in Cloudron on ARM in 2030!I had never played with one till this year, they're fun! I got the latest model, the Raspberry Pi 4. Powerful little thing! It'd be great to have another pair of hands on this.
-
@malvim We never run tests on the Cloudron itself, only on the laptop! Can you give the output of the
npm testcommand? What is the error? -
@malvim Well, consider yourself not alone in this endeavor! I'd love to help and am very interested in porting this to arm. Depends on the number of dependencies from the base image as well as Cloudron itself.
I'll post back here when I get my Raspberry Pi; I'm excited!
️ -
@girish said in Cloudron on a Raspberry pi?:
@malvim We never run tests on the Cloudron itself, only on the laptop! Can you give the output of the
npm testcommand? What is the error?malvim@zem:~/docker/postgresql-addon/test$ npm i audited 152 packages in 2.419s found 3 vulnerabilities (2 low, 1 high) run `npm audit fix` to fix them, or `npm audit` for details malvim@zem:~/docker/postgresql-addon/test$ npm test > postgresql-addon@1.0.0 test /home/malvim/docker/postgresql-addon > mocha --bail ./test/test.js Postgresql Addon Error: No such container: postgresql auth ✓ fails without access_token ✓ fails with invalid access_token ✓ succeeds add database ✓ succeeds (676ms) ✓ succeeds when added again (124ms) remove database 1) succeeds 5 passing (2m) 1 failing 1) Postgresql Addon remove database succeeds: Error: Timeout of 100000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/malvim/docker/postgresql-addon/test/test.js) at listOnTimeout (internal/timers.js:554:17) at processTimers (internal/timers.js:497:7)Hey, @girish!
This is the output, and it just hangs after that and never exits. This happens on my laptop and on another machine I used for testing purposes. On my Cloudron, these tests passed and only the last one failed, the backup/restore one iirc.
-
@malvim said in Cloudron on a Raspberry pi?:
Postgresql Addon
Error: No such container: postgresql
auththis seems to be more of a problem than the timeout later
-
@robi said in Cloudron on a Raspberry pi?:
@malvim said in Cloudron on a Raspberry pi?:
Postgresql Addon
Error: No such container: postgresql
auththis seems to be more of a problem than the timeout later
It's really not. This happens because the test always tries to remove the currentl running postgresql container before running the tests, so the first time you run, the container is not running and you get this message. Still runs the tests and some of them work.
-
@malvim ok thanks, otherwise it looks like it's missing
..which would partially explain the timeout..
-
@malvim said in Cloudron on a Raspberry pi?:
This is the output, and it just hangs after that and never exits
Is there anything in
docker logs -f postgresql? -
Is it getting the token last when it should be getting it first in
auth? I've gotta re-read this whole thread cause I'm missing what this unit test even is supposed to prove. That one module could run in a Cloudron environment? -
@girish said in Cloudron on a Raspberry pi?:
Is there anything in
docker logs -f postgresql?Nothing that I thought was strange... Here's the output:
Creating new installation [39/54] The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "C". The default text search configuration will be set to "english". Data page checksums are disabled. fixing permissions on existing directory /var/lib/postgresql/11/main ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default timezone ... Etc/UTC selecting dynamic shared memory implementation ... posix creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok syncing data to disk ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. Success. You can now start the database server using: /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start CREATE ROLE ALTER ROLE waiting for server to shut down.... done server stopped Generating SSL certificate Generating a RSA private key .......................+++++ .....................................................................+++++ writing new private key to '/run/postgresql.cloudron.key' ----- Starting supervisor 2020-10-21 01:20:33,612 CRIT Supervisor running as root (no user in config file) 2020-10-21 01:20:33,613 INFO Included extra file "/etc/supervisor/conf.d/postgresql-service.conf" during parsing 2020-10-21 01:20:33,613 INFO Included extra file "/etc/supervisor/conf.d/postgresql.conf" during parsing 2020-10-21 01:20:33,628 INFO RPC interface 'supervisor' initialized 2020-10-21 01:20:33,628 CRIT Server 'inet_http_server' running without any HTTP authentication checking 2020-10-21 01:20:33,630 INFO RPC interface 'supervisor' initialized 2020-10-21 01:20:33,631 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2020-10-21 01:20:33,631 INFO supervisord started with pid 1 2020-10-21 01:20:34,635 INFO spawned: 'postgresql' with pid 53 2020-10-21 01:20:34,639 INFO spawned: 'postgresql-service' with pid 54 2020-10-21 01:20:34.680 UTC [53] LOG: listening on IPv4 address "0.0.0.0", port 5432 2020-10-21 01:20:34.680 UTC [53] LOG: listening on IPv6 address "::", port 5432 2020-10-21 01:20:34.714 UTC [53] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2020-10-21 01:20:34.788 UTC [61] LOG: database system was shut down at 2020-10-21 01:20:33 UTC 2020-10-21 01:20:34.825 UTC [53] LOG: database system is ready to accept connections Postgresql service endpoint listening on https://:::3000 [GET] /healthcheck 2020-10-21 01:20:36,382 INFO success: postgresql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-10-21 01:20:36,382 INFO success: postgresql-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) [GET] /healthcheck [GET] / [GET] / [GET] / [POST] /databases [POST] /databases [POST] /databases [DELETE] /databases/removetestdatabase -
So, could anyone here just try and run tests for addons on their machines, without any cloudron-related stuff, then just clone an addon and try to test it?
I've ran tests on my laptop, and on a server I have access to that doesn't run cloudron, and both of them time out in the same place.
DON'T RUN this tests on your production cloudron like I did, it will delete and recreate your postgresql (or whatever addon you're trying to use) and apps will go down heheh. I had to restore a couple of apps' backups, but it's all good now.
-
@malvim Maybe you can try to see if the mysql addon tests work in the meantime?
I ran the postgresl tests now and it did work for me:
> postgresql-addon@1.0.0 test /home/girish/yellowtent/postgresql-addon > mocha --bail ./test/test.js Postgresql Addon Error: No such container: postgresql Error response from daemon: network with name cloudron already exists auth ✓ fails without access_token ✓ fails with invalid access_token ✓ succeeds add database ✓ succeeds (401ms) ✓ succeeds when added again remove database ✓ succeeds (147ms) use the database ✓ can create extension (77ms) ✓ can create table foo ✓ can insert into table foo ✓ can read from table foo ✓ restart (5309ms) ✓ can read from table foo backup and restore ✓ succeeds to create backup (386ms) ✓ succeeds to create new database (743ms) ✓ succeeds to clear new database (439ms) ✓ succeeds to restore backup (1280ms) ✓ succeeds to check restore data (44ms) restore of invalid dump fails ✓ succeeds to create backup (178ms) ✓ succeeds to clear new database (426ms) ✓ fails to restore backup (180ms) restore of existing dump ✓ succeeds (1876ms) 21 passing (38s) -
Hey, @girish
Mysql addon tests run perfectly on all three machines (laptop, server, pi)!
I'm having failing tests on mail and sftp addons, and this hanging problem with postgresql. All behaviors are the same on the three machines, which is a... good thing, I guess? Haha! I'll try the others and see where they go.
Couldn't download graphite, though, says I don't have permission rights.
-
@malvim The mail tests are kind of hard to run, they also require a special setup with DNS/Docker. We can skip that for now. Mongo and redis tests work too? That's really good progress then. Can you tell me what you face with the sftp addon? You might need the latest "build" of sftp since I really just fixed the test 2-3 days ago.
-
Hey,
It's a bit past midnight here and I've JUST realized my laptop doesn't have ssh HOST keys (I never ssh into it), and the test mounts the host's
/etc/sshdirectory into the container's/etc/ssh, and that's why it's not running.I'll go get some sleep and keep going tomorrow night.
-
So I installed
openssh-serverand now I get this:2020-10-22 04:13:28,130 INFO spawned: 'proftpd' with pid 48 Wrong passphrase for this key. Please try again. Wrong passphrase for this key. Please try again. Wrong passphrase for this key. Please try again. 2020-10-22 04:13:28,150 c7a6e160fe35 proftpd[48] c7a6e160fe35: mod_sftp/0.9.9: error reading passphrase for SFTPHostKey '/etc/ssh/ssh_host_rsa_key': (unknown) 2020-10-22 04:13:28,150 c7a6e160fe35 proftpd[48] c7a6e160fe35: mod_sftp/0.9.9: unable to use key in SFTPHostKey '/etc/ssh/ssh_host_rsa_key', exiting 2020-10-22 04:13:28,151 INFO exited: proftpd (exit status 0; not expected) 2020-10-22 04:13:31,160 INFO spawned: 'proftpd' with pid 49 Wrong passphrase for this key. Please try again. Wrong passphrase for this key. Please try again. Wrong passphrase for this key. Please try again. 2020-10-22 04:13:31,183 c7a6e160fe35 proftpd[49] c7a6e160fe35: mod_sftp/0.9.9: error reading passphrase for SFTPHostKey '/etc/ssh/ssh_host_rsa_key': (unknown) 2020-10-22 04:13:31,183 c7a6e160fe35 proftpd[49] c7a6e160fe35: mod_sftp/0.9.9: unable to use key in SFTPHostKey '/etc/ssh/ssh_host_rsa_key', exiting 2020-10-22 04:13:31,185 INFO exited: proftpd (exit status 0; not expected)Not sure how to proceed when it asks me for passphrases...
-
are there pre-set keys or are they generated during setup?
since the dir wasn't there during setup, maybe the generation failed hence the errors.
-
@malvim I guess you are running this on Ubuntu 20? They changed the ssh keys format, so you have to generate the host keys again. See the test.js comment in startSftp().
-
@robi setup mounts the host's
/etc/sshdir, so it uses whatever is in the host machine, and it was... nothing!@girish hahahahahahahaha holy crap, I was looking at THIS EXACT LINE and completely disregarded the comment that explained perfectly what was going on! Thanks for the help once again, I'll check it out.
-
Hey, guys.
So, this is what's going on currently:
I was able to build and run tests for, with minimal adaptations, the
docker-base-imageproject, and the following addons:docker-sftpmongodb-addonmysql-addonredis-addonturn-addon
That leaves
mail-addon,docker-graphiteandpostgresql-addonMail needs more setup as @girish said, so I'm not doing it for now. I was not able to clone the
docker-graphiteproject, as I think I've mentioned before, so still waiting on what to do in this case.In trying to understand a bit more of what's going on with the
postgreql-addontests hanging, I found it hangs in any test that usesconnectClient, like theremove databaseanduse databasetests:describe('remove database', function () { const data = { database: 'removetestdatabase', username: 'removetestuser', password: 'somepassword', locale: 'C' }; before(function (done) { addDatabase(data, done); }); it('succeeds', function (done) { request.delete(`https://${ip}:3000/databases/${data.database}?access_token=${CLOUDRON_POSTGRESQL_TOKEN}&username=${data.username}`, { rejectUnauthorized: false }, function (error, response, body) { expect(error).to.be(null); expect(response.statusCode).to.equal(200); connectClient(data, function (error) { expect(error).to.not.eql(null); client.end(done); }); }); }); });In this case,
addDatabaseruns ok, and it makes sense since it is tested before.The function is called,
request.deleteis called, the twoexpects pass, and then the function INSIDEconnectClient(with theexpect(error)line) never runs.Can anyone else (besides @girish, who ran the tests and seen them run fine) run this and see what happens? This is happening on my laptop as well, not only on the Pi, but if I can't get this to go, I can't get the tests to pass on the pi, and we won't be able to trust everything runs okay.
-
@malvim Are you testing this on a mac?
-
@girish Nope. Ubuntu 20.04 on both my laptop and a server, Ubuntu 18.04 on the raspberry pi. All with the same result.
-
It works for me here atleast. I am guessing that you are unable to connect to the container IP maybe? Can you try installing psql tooling and connect via IP address? You can put a log in connectClient to see the credentials.
$ npm test > postgresql-addon@1.0.0 test /home/girish/yellowtent/postgresql-addon > mocha --bail ./test/test.js Postgresql Addon Error response from daemon: network with name cloudron already exists auth ✓ fails without access_token ✓ fails with invalid access_token ✓ succeeds add database ✓ succeeds (369ms) ✓ succeeds when added again remove database ✓ succeeds (125ms) use the database ✓ can create extension (82ms) ✓ can create table foo ✓ can insert into table foo ✓ can read from table foo ✓ restart (5091ms) ✓ can read from table foo backup and restore ✓ succeeds to create backup (443ms) ✓ succeeds to create new database (412ms) ✓ succeeds to clear new database (426ms) ✓ succeeds to restore backup (940ms) ✓ succeeds to check restore data restore of invalid dump fails ✓ succeeds to create backup (186ms) ✓ succeeds to clear new database (424ms) ✓ fails to restore backup (185ms) restore of existing dump ✓ succeeds (1721ms) 21 passing (36s)
