Cloudron on a Raspberry pi?

[girish]

@malvim Maybe you can try to see if the mysql addon tests work in the meantime?

I ran the postgresl tests now and it did work for me:

> postgresql-addon@1.0.0 test /home/girish/yellowtent/postgresql-addon
> mocha --bail ./test/test.js



  Postgresql Addon
Error: No such container: postgresql
Error response from daemon: network with name cloudron already exists
    auth
      ✓ fails without access_token
      ✓ fails with invalid access_token
      ✓ succeeds
    add database
      ✓ succeeds (401ms)
      ✓ succeeds when added again
    remove database
      ✓ succeeds (147ms)
    use the database
      ✓ can create extension (77ms)
      ✓ can create table foo
      ✓ can insert into table foo
      ✓ can read from table foo
      ✓ restart (5309ms)
      ✓ can read from table foo
    backup and restore
      ✓ succeeds to create backup (386ms)
      ✓ succeeds to create new database (743ms)
      ✓ succeeds to clear new database (439ms)
      ✓ succeeds to restore backup (1280ms)
      ✓ succeeds to check restore data (44ms)
    restore of invalid dump fails
      ✓ succeeds to create backup (178ms)
      ✓ succeeds to clear new database (426ms)
      ✓ fails to restore backup (180ms)
    restore of existing dump
      ✓ succeeds (1876ms)


  21 passing (38s)

[malvim]

Hey, @girish

Mysql addon tests run perfectly on all three machines (laptop, server, pi)!

I'm having failing tests on mail and sftp addons, and this hanging problem with postgresql. All behaviors are the same on the three machines, which is a... good thing, I guess? Haha! I'll try the others and see where they go.

Couldn't download graphite, though, says I don't have permission rights.

[girish]

@malvim The mail tests are kind of hard to run, they also require a special setup with DNS/Docker. We can skip that for now. Mongo and redis tests work too? That's really good progress then. Can you tell me what you face with the sftp addon? You might need the latest "build" of sftp since I really just fixed the test 2-3 days ago.

[malvim]

Hey,

It's a bit past midnight here and I've JUST realized my laptop doesn't have ssh HOST keys (I never ssh into it), and the test mounts the host's /etc/ssh directory into the container's /etc/ssh, and that's why it's not running.

I'll go get some sleep and keep going tomorrow night.

[malvim]

So I installed openssh-server and now I get this:

2020-10-22 04:13:28,130 INFO spawned: 'proftpd' with pid 48

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.
2020-10-22 04:13:28,150 c7a6e160fe35 proftpd[48] c7a6e160fe35: mod_sftp/0.9.9: error reading passphrase for SFTPHostKey '/etc/ssh/ssh_host_rsa_key': (unknown)
2020-10-22 04:13:28,150 c7a6e160fe35 proftpd[48] c7a6e160fe35: mod_sftp/0.9.9: unable to use key in SFTPHostKey '/etc/ssh/ssh_host_rsa_key', exiting
2020-10-22 04:13:28,151 INFO exited: proftpd (exit status 0; not expected)
2020-10-22 04:13:31,160 INFO spawned: 'proftpd' with pid 49

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.

Wrong passphrase for this key.  Please try again.
2020-10-22 04:13:31,183 c7a6e160fe35 proftpd[49] c7a6e160fe35: mod_sftp/0.9.9: error reading passphrase for SFTPHostKey '/etc/ssh/ssh_host_rsa_key': (unknown)
2020-10-22 04:13:31,183 c7a6e160fe35 proftpd[49] c7a6e160fe35: mod_sftp/0.9.9: unable to use key in SFTPHostKey '/etc/ssh/ssh_host_rsa_key', exiting
2020-10-22 04:13:31,185 INFO exited: proftpd (exit status 0; not expected)

Not sure how to proceed when it asks me for passphrases...

[robi]

are there pre-set keys or are they generated during setup?

since the dir wasn't there during setup, maybe the generation failed hence the errors.

[girish]

@malvim I guess you are running this on Ubuntu 20? They changed the ssh keys format, so you have to generate the host keys again. See the test.js comment in startSftp().

[malvim]

@robi setup mounts the host's /etc/ssh dir, so it uses whatever is in the host machine, and it was... nothing!

@girish hahahahahahahaha holy crap, I was looking at THIS EXACT LINE and completely disregarded the comment that explained perfectly what was going on! Thanks for the help once again, I'll check it out.

[malvim]

Hey, guys.

So, this is what's going on currently:

I was able to build and run tests for, with minimal adaptations, the docker-base-image project, and the following addons:

• docker-sftp
• mongodb-addon
• mysql-addon
• redis-addon
• turn-addon

That leaves mail-addon, docker-graphite and postgresql-addon

Mail needs more setup as @girish said, so I'm not doing it for now. I was not able to clone the docker-graphite project, as I think I've mentioned before, so still waiting on what to do in this case.

In trying to understand a bit more of what's going on with the postgreql-addon tests hanging, I found it hangs in any test that uses connectClient, like the remove database and use database tests:

describe('remove database', function () {
    const data = { database: 'removetestdatabase', username: 'removetestuser', password: 'somepassword', locale: 'C' };

    before(function (done) {
        addDatabase(data, done);
    });

    it('succeeds', function (done) {
        request.delete(`https://${ip}:3000/databases/${data.database}?access_token=${CLOUDRON_POSTGRESQL_TOKEN}&username=${data.username}`, { rejectUnauthorized: false }, function (error, response, body) {
            expect(error).to.be(null);
            expect(response.statusCode).to.equal(200);

            connectClient(data, function (error) {
                expect(error).to.not.eql(null);
                client.end(done);
            });
        });
    });
});

In this case, addDatabase runs ok, and it makes sense since it is tested before.

The function is called, request.delete is called, the two expects pass, and then the function INSIDE connectClient (with the expect(error) line) never runs.

Can anyone else (besides @girish, who ran the tests and seen them run fine) run this and see what happens? This is happening on my laptop as well, not only on the Pi, but if I can't get this to go, I can't get the tests to pass on the pi, and we won't be able to trust everything runs okay.

[girish]

@malvim Are you testing this on a mac?

[malvim]

@girish Nope. Ubuntu 20.04 on both my laptop and a server, Ubuntu 18.04 on the raspberry pi. All with the same result.

[girish]

It works for me here atleast. I am guessing that you are unable to connect to the container IP maybe? Can you try installing psql tooling and connect via IP address? You can put a log in connectClient to see the credentials.

$ npm test

> postgresql-addon@1.0.0 test /home/girish/yellowtent/postgresql-addon
> mocha --bail ./test/test.js



  Postgresql Addon
Error response from daemon: network with name cloudron already exists
    auth
      ✓ fails without access_token
      ✓ fails with invalid access_token
      ✓ succeeds
    add database
      ✓ succeeds (369ms)
      ✓ succeeds when added again
    remove database
      ✓ succeeds (125ms)
    use the database
      ✓ can create extension (82ms)
      ✓ can create table foo
      ✓ can insert into table foo
      ✓ can read from table foo
      ✓ restart (5091ms)
      ✓ can read from table foo
    backup and restore
      ✓ succeeds to create backup (443ms)
      ✓ succeeds to create new database (412ms)
      ✓ succeeds to clear new database (426ms)
      ✓ succeeds to restore backup (940ms)
      ✓ succeeds to check restore data
    restore of invalid dump fails
      ✓ succeeds to create backup (186ms)
      ✓ succeeds to clear new database (424ms)
      ✓ fails to restore backup (185ms)
    restore of existing dump
      ✓ succeeds (1721ms)


  21 passing (36s)

[malvim]

Hey,

So I commented out the remove database tests because I had trouble connecting from the container, but the next test, use the database, starts like this:

// psql -h localhost -U usetestuser  -d usetestdatabase
describe('use the database', function () {
    const data = { database: 'usetestdatabase', username: 'usetestuser', password: 'somepassword', locale: 'C' };

    before(function (done) {
        async.series([
            addDatabase.bind(null, data),
            connectClient.bind(null, data)
        ], done);
    });

So I ran "psql -h localhost -U usetestuser -d usetestdatabase", using 'somepassword' as a password, and was able to connect, both from inside the container and from my host machine.

connectClient still doesn't connect, and, I get this on the screen:

    auth
      ✓ fails without access_token
      ✓ fails with invalid access_token
      ✓ succeeds
    use the database
usetestuser somepassword usetestdatabase 172.18.0.2 5432
      1) "before all" hook

  3 passing (2m)
  1 failing

  1) Postgresql Addon
       use the database
         "before all" hook:
     Error: Timeout of 100000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/malvim/Projetos/Pi/docker/postgresql-addon/test/test.js)
      at listOnTimeout (internal/timers.js:551:17)
      at processTimers (internal/timers.js:494:7)

   
      2) "after all" hook

[malvim]

Also, it does not seem to be network-related, since I get the same behavior on an ubuntu 20.04 VPS I have...

[malvim]

I tried recreating the steps inside the container using node, and could connect. I'm really at a loss here right now...

[Lonkle]

@malvim Wish I had my Pi now so I could help test! Soon tho. ️

[malvim]

Hey, @Lonk! You mind testing this on your laptop? I'm not getting these errors only on the Pi, I'm getting them on my amd64 laptop and amd64 server as well!

Repo is here: https://git.cloudron.io/cloudron/postgresql-addon, it's just a matter of:

• cloning it;
• building the image locally (docker build -t cloudron/postgresqladdontest .)
• installing dependencies (npm install)
• running the test (npm test)

Could you run this somewhere and see if you have the same problem?

Thanks!

[malvim]

MAN, it seems I'm running into ALL KINDS of weird problems hahaha!

I started going into postgresql node client's code, and it seemed to be some weird behavior of EventEmitter, which was weird...

I installed nvm and tested with older node distributions, and it ran fine on my laptop! So it seems, @girish, that the test code for postgresql-addon does not run with node v14 or later on my machines. Tested with v12 and v13 and it was all fine. What version are you using for tests?

Moving on to the next hurdle, I guess hahah!

[girish]

@malvim said in Cloudron on a Raspberry pi?:

So it seems, @girish, that the test code for postgresql-addon does not run with node v14 or later on my machines

Aha! I feel like I have hit this issue before. Indeed, when we updated the box code to use node 14 lots of things fail (not sure why). We use node 10.18.1 everywhere. This is why the hotfix and code enforces it. For future reference, always use the node in scripts/createReleaseTarball in the box repo.

[malvim]

@girish okay, cool, I'll use that. It's good news, then, that the OTHER tests even ran!

So, for what it's worth, the code seemed to fail on the use of EventEmitter. A class was dispatching an event and the corresponding listener was not triggered, so if you ever need to investigate further, maybe that's a start.