Cloudron on a Raspberry pi?

[girish]

@malvim We never run tests on the Cloudron itself, only on the laptop! Can you give the output of the npm test command? What is the error?

[Lonkle]

@malvim Well, consider yourself not alone in this endeavor! I'd love to help and am very interested in porting this to arm. Depends on the number of dependencies from the base image as well as Cloudron itself.

I'll post back here when I get my Raspberry Pi; I'm excited! ️

[malvim]

@girish said in Cloudron on a Raspberry pi?:

@malvim We never run tests on the Cloudron itself, only on the laptop! Can you give the output of the npm test command? What is the error?

malvim@zem:~/docker/postgresql-addon/test$ npm i
audited 152 packages in 2.419s
found 3 vulnerabilities (2 low, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details
malvim@zem:~/docker/postgresql-addon/test$ npm test

> postgresql-addon@1.0.0 test /home/malvim/docker/postgresql-addon
> mocha --bail ./test/test.js



  Postgresql Addon
Error: No such container: postgresql
    auth
      ✓ fails without access_token
      ✓ fails with invalid access_token
      ✓ succeeds
    add database
      ✓ succeeds (676ms)
      ✓ succeeds when added again (124ms)
    remove database
      1) succeeds

  5 passing (2m)
  1 failing

  1) Postgresql Addon
       remove database
         succeeds:
     Error: Timeout of 100000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/malvim/docker/postgresql-addon/test/test.js)
      at listOnTimeout (internal/timers.js:554:17)
      at processTimers (internal/timers.js:497:7)

Hey, @girish!

This is the output, and it just hangs after that and never exits. This happens on my laptop and on another machine I used for testing purposes. On my Cloudron, these tests passed and only the last one failed, the backup/restore one iirc.

[robi]

@malvim said in Cloudron on a Raspberry pi?:

Postgresql Addon
Error: No such container: postgresql
auth

this seems to be more of a problem than the timeout later

[malvim]

@robi said in Cloudron on a Raspberry pi?:

@malvim said in Cloudron on a Raspberry pi?:

Postgresql Addon
Error: No such container: postgresql
auth

this seems to be more of a problem than the timeout later

It's really not. This happens because the test always tries to remove the currentl running postgresql container before running the tests, so the first time you run, the container is not running and you get this message. Still runs the tests and some of them work.

[robi]

@malvim ok thanks, otherwise it looks like it's missing

..which would partially explain the timeout..

[girish]

@malvim said in Cloudron on a Raspberry pi?:

This is the output, and it just hangs after that and never exits

Is there anything in docker logs -f postgresql ?

[Lonkle]

Is it getting the token last when it should be getting it first in auth? I've gotta re-read this whole thread cause I'm missing what this unit test even is supposed to prove. That one module could run in a Cloudron environment?

[malvim]

@girish said in Cloudron on a Raspberry pi?:

Is there anything in docker logs -f postgresql ?

Nothing that I thought was strange... Here's the output:

Creating new installation                                                                                                                                                                                     [39/54]
The files belonging to this database system will be owned by user "postgres".                             
This user must also own the server process.                                                                                                                                                                          
                                                                                                                                                                                                                     
The database cluster will be initialized with locale "C".                                                 
The default text search configuration will be set to "english".                                           
                                                                                                          
Data page checksums are disabled.                    
                                                     
fixing permissions on existing directory /var/lib/postgresql/11/main ... ok                                                                                                                                          
creating subdirectories ... ok                                                                                                                                                                                       
selecting default max_connections ... 100                                                                 
selecting default shared_buffers ... 128MB                                                                                                                                                                           
selecting default timezone ... Etc/UTC                                                                    
selecting dynamic shared memory implementation ... posix                                                  
creating configuration files ... ok                  
running bootstrap script ... ok                      
performing post-bootstrap initialization ... ok                                                           
syncing data to disk ... ok                          

WARNING: enabling "trust" authentication for local connections                                            
You can change this by editing pg_hba.conf or using the option -A, or                                     
--auth-local and --auth-host, the next time you run initdb.                                               

Success. You can now start the database server using:                                                     

    /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start                     

CREATE ROLE                                          
ALTER ROLE                                           
waiting for server to shut down.... done             
server stopped                                       
Generating SSL certificate                           
Generating a RSA private key                         
.......................+++++                         
.....................................................................+++++                                
writing new private key to '/run/postgresql.cloudron.key'                                                 
-----                                                
Starting supervisor                                  
2020-10-21 01:20:33,612 CRIT Supervisor running as root (no user in config file)                          
2020-10-21 01:20:33,613 INFO Included extra file "/etc/supervisor/conf.d/postgresql-service.conf" during parsing                                                                                                     
2020-10-21 01:20:33,613 INFO Included extra file "/etc/supervisor/conf.d/postgresql.conf" during parsing  
2020-10-21 01:20:33,628 INFO RPC interface 'supervisor' initialized                                       
2020-10-21 01:20:33,628 CRIT Server 'inet_http_server' running without any HTTP authentication checking   
2020-10-21 01:20:33,630 INFO RPC interface 'supervisor' initialized                                       
2020-10-21 01:20:33,631 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-10-21 01:20:33,631 INFO supervisord started with pid 1                                               
2020-10-21 01:20:34,635 INFO spawned: 'postgresql' with pid 53                                            
2020-10-21 01:20:34,639 INFO spawned: 'postgresql-service' with pid 54                                    
2020-10-21 01:20:34.680 UTC [53] LOG:  listening on IPv4 address "0.0.0.0", port 5432                     
2020-10-21 01:20:34.680 UTC [53] LOG:  listening on IPv6 address "::", port 5432                          
2020-10-21 01:20:34.714 UTC [53] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-10-21 01:20:34.788 UTC [61] LOG:  database system was shut down at 2020-10-21 01:20:33 UTC
2020-10-21 01:20:34.825 UTC [53] LOG:  database system is ready to accept connections                     
Postgresql service endpoint listening on https://:::3000                                                  
[GET] /healthcheck                                   
2020-10-21 01:20:36,382 INFO success: postgresql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-21 01:20:36,382 INFO success: postgresql-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[GET] /healthcheck                                   
[GET] /                                              
[GET] /                                              
[GET] /                                              
[POST] /databases                                    
[POST] /databases                                    
[POST] /databases                                    
[DELETE] /databases/removetestdatabase

[malvim]

So, could anyone here just try and run tests for addons on their machines, without any cloudron-related stuff, then just clone an addon and try to test it?

I've ran tests on my laptop, and on a server I have access to that doesn't run cloudron, and both of them time out in the same place.

DON'T RUN this tests on your production cloudron like I did, it will delete and recreate your postgresql (or whatever addon you're trying to use) and apps will go down heheh. I had to restore a couple of apps' backups, but it's all good now.

[girish]

@malvim Maybe you can try to see if the mysql addon tests work in the meantime?

I ran the postgresl tests now and it did work for me:

> postgresql-addon@1.0.0 test /home/girish/yellowtent/postgresql-addon
> mocha --bail ./test/test.js



  Postgresql Addon
Error: No such container: postgresql
Error response from daemon: network with name cloudron already exists
    auth
      ✓ fails without access_token
      ✓ fails with invalid access_token
      ✓ succeeds
    add database
      ✓ succeeds (401ms)
      ✓ succeeds when added again
    remove database
      ✓ succeeds (147ms)
    use the database
      ✓ can create extension (77ms)
      ✓ can create table foo
      ✓ can insert into table foo
      ✓ can read from table foo
      ✓ restart (5309ms)
      ✓ can read from table foo
    backup and restore
      ✓ succeeds to create backup (386ms)
      ✓ succeeds to create new database (743ms)
      ✓ succeeds to clear new database (439ms)
      ✓ succeeds to restore backup (1280ms)
      ✓ succeeds to check restore data (44ms)
    restore of invalid dump fails
      ✓ succeeds to create backup (178ms)
      ✓ succeeds to clear new database (426ms)
      ✓ fails to restore backup (180ms)
    restore of existing dump
      ✓ succeeds (1876ms)


  21 passing (38s)

[malvim]

Hey, @girish

Mysql addon tests run perfectly on all three machines (laptop, server, pi)!

I'm having failing tests on mail and sftp addons, and this hanging problem with postgresql. All behaviors are the same on the three machines, which is a... good thing, I guess? Haha! I'll try the others and see where they go.

Couldn't download graphite, though, says I don't have permission rights.

[girish]

@malvim The mail tests are kind of hard to run, they also require a special setup with DNS/Docker. We can skip that for now. Mongo and redis tests work too? That's really good progress then. Can you tell me what you face with the sftp addon? You might need the latest "build" of sftp since I really just fixed the test 2-3 days ago.

[malvim]

Hey,

It's a bit past midnight here and I've JUST realized my laptop doesn't have ssh HOST keys (I never ssh into it), and the test mounts the host's /etc/ssh directory into the container's /etc/ssh, and that's why it's not running.

I'll go get some sleep and keep going tomorrow night.

[malvim]

So I installed openssh-server and now I get this:

2020-10-22 04:13:28,130 INFO spawned: 'proftpd' with pid 48

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.
2020-10-22 04:13:28,150 c7a6e160fe35 proftpd[48] c7a6e160fe35: mod_sftp/0.9.9: error reading passphrase for SFTPHostKey '/etc/ssh/ssh_host_rsa_key': (unknown)
2020-10-22 04:13:28,150 c7a6e160fe35 proftpd[48] c7a6e160fe35: mod_sftp/0.9.9: unable to use key in SFTPHostKey '/etc/ssh/ssh_host_rsa_key', exiting
2020-10-22 04:13:28,151 INFO exited: proftpd (exit status 0; not expected)
2020-10-22 04:13:31,160 INFO spawned: 'proftpd' with pid 49

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.
2020-10-22 04:13:31,183 c7a6e160fe35 proftpd[49] c7a6e160fe35: mod_sftp/0.9.9: error reading passphrase for SFTPHostKey '/etc/ssh/ssh_host_rsa_key': (unknown)
2020-10-22 04:13:31,183 c7a6e160fe35 proftpd[49] c7a6e160fe35: mod_sftp/0.9.9: unable to use key in SFTPHostKey '/etc/ssh/ssh_host_rsa_key', exiting
2020-10-22 04:13:31,185 INFO exited: proftpd (exit status 0; not expected)

Not sure how to proceed when it asks me for passphrases...

[robi]

are there pre-set keys or are they generated during setup?

since the dir wasn't there during setup, maybe the generation failed hence the errors.

[girish]

@malvim I guess you are running this on Ubuntu 20? They changed the ssh keys format, so you have to generate the host keys again. See the test.js comment in startSftp().

[malvim]

@robi setup mounts the host's /etc/ssh dir, so it uses whatever is in the host machine, and it was... nothing!

@girish hahahahahahahaha holy crap, I was looking at THIS EXACT LINE and completely disregarded the comment that explained perfectly what was going on! Thanks for the help once again, I'll check it out.

[malvim]

Hey, guys.

So, this is what's going on currently:

I was able to build and run tests for, with minimal adaptations, the docker-base-image project, and the following addons:

• docker-sftp
• mongodb-addon
• mysql-addon
• redis-addon
• turn-addon

That leaves mail-addon, docker-graphite and postgresql-addon

Mail needs more setup as @girish said, so I'm not doing it for now. I was not able to clone the docker-graphite project, as I think I've mentioned before, so still waiting on what to do in this case.

In trying to understand a bit more of what's going on with the postgreql-addon tests hanging, I found it hangs in any test that uses connectClient, like the remove database and use database tests:

describe('remove database', function () {
    const data = { database: 'removetestdatabase', username: 'removetestuser', password: 'somepassword', locale: 'C' };

    before(function (done) {
        addDatabase(data, done);
    });

    it('succeeds', function (done) {
        request.delete(`https://${ip}:3000/databases/${data.database}?access_token=${CLOUDRON_POSTGRESQL_TOKEN}&username=${data.username}`, { rejectUnauthorized: false }, function (error, response, body) {
            expect(error).to.be(null);
            expect(response.statusCode).to.equal(200);

            connectClient(data, function (error) {
                expect(error).to.not.eql(null);
                client.end(done);
            });
        });
    });
});

In this case, addDatabase runs ok, and it makes sense since it is tested before.

The function is called, request.delete is called, the two expects pass, and then the function INSIDE connectClient (with the expect(error) line) never runs.

Can anyone else (besides @girish, who ran the tests and seen them run fine) run this and see what happens? This is happening on my laptop as well, not only on the Pi, but if I can't get this to go, I can't get the tests to pass on the pi, and we won't be able to trust everything runs okay.

[girish]

@malvim Are you testing this on a mac?