Writing automated tests for packages

[girish]

Hi All,
A number of packages are currently marked as WIP now. Some of you have reached out and asked on what is blocking releasing them. What's blocking them essentially is automated tests. We maintain a large number of apps and as a small team what makes it possible for us to push reliable updates quickly is our automated tests. These tests, while it takes a bit of time to write them, help us in maintaining the package in the long run. Without the tests, we are simply not in a position to update apps quickly.

These automated tests are browser based selenium tests. The focus of the tests is not to focus on the app testing but to focus on the packaging. What the tests do is:

• Install the app the Cloudron. By convention, this is the 'test' subdomain.
• Open the app in a browser and login with admin credentials or Cloudron credentials and check if the login works
• Depending on the app, it will create something like a document or a post or file or chat message etc
• Restart the app and check if the app works
• Backup the app and then restore the app from the backup. Then we check if the above post/file/chat message is still around. This provides sanity test that the app is backing up properly
• Move the app to a different location. By convention, this is the 'test2' subdomain. Check if the app has the above post/file/chat. This checks if the package is setting up domain/app urls correctly.
• Finally, to test updates, it installs the current version of the app from the app store. It logs in and creates post/file/chat. It then applies the current app package build on top of the app store version and checks if data is still OK.

The tests while they follow a common pattern have to be individually tuned per app (specifically the login and the post/file/chat message is different for each app). Writing the tests requires some javascript knowledge and also some xpath knowledge but we have written tests for enough packages that we have automated all sorts of things. Just ask here and we can help.

Running existing tests

Before writing tests, the first thing to do is to simply start with ensuring you can run the tests of some existing package. For example, say https://git.cloudron.io/cloudron/minio-app . To run the tests:

• Install the cloudron CLI tool and do cloudron login
• git clone https://git.cloudron.io/cloudron/minio-app
• cloudron build
• cd minio-app/test
• npm install
• USERNAME=cloudronusername PASSWORD=cloudronpassword node_modules/.bin/mocha --bail test.js

The above will produce an output like this:

  Application life cycle test
    - build app
App is being installed.

 => Queued 
 => Cleaning up old install 
 => Registering subdomains 
 => Downloading image ..
 => Creating container .
 => Waiting for DNS propagation ..................
 => Wait for health check .....

App is installed.
    ✓ install app (37267ms)
    ✓ can get app information (735ms)
    ✓ can login (3062ms)
    ✓ can add bucket (1304ms)
...

Writing tests

To write tests, usually, I just copy an existing app's test directory straight into the new one. And then, I slowly start fixing the tests. Tip: you can just change it() to xit() to skip a test.

We have also not tested running tests on Mac or Windows. Though, I am confident that it works well or can be made to work well on Mac.

Long story short, if you can help us write tests for the packages, we can get it published almost immediately. Please let us know what information you require from us to help writing these tests.

[girish]

I recorded a quick video showing host the tests run. The browser on the left is my Firefox that just shows the Cloudron dashboard. The browser that appears on the right is Chrome and it is where the automated testing happens.

Youtube Video

EDIT: OK, this is funny. @nebulon was wondering why I had some dark music in the video. I thought I had recorded a silent video and did not even know there was some background music Investigating, I found out that my youtube at some point has started auto-playing "3 HOURS Most EPIC POWERFUL BATTLE MUSIC"! and OBS was happily recording that. Since, I don't have a headset connected to this setup, I had no idea. Funnily, when I uploaded this video, it immediately told me there is a "copyright claim" and I thought "yeah, right". Now I know why.

[murgero]

@girish Beautiful I'll get tests going for Directus then. Since I am being paid for it's packaging.

[marcusquinn]

@murgero Yay unit testing definitely a great string to add to your bow and hopefully time-saver for adding more apps without adding maintenance whack-a-mole overhead too when you'd hope the tests will do the monitoring so we don't need to so much.

[girish]

@murgero Thanks, if you have any question, feel free to ask and I am happy to help.

[doodlemania2]

Can you tell me a bit about your setup? I assume this is a Linux desktop distro? Any pointers on initial setup would be appreciated!

[doodlemania2]

@girish - i watched the video and was able to discern most of your setup, but would still appreciate a sanity check if you have time on how you got your test harness up and running. Having troubles getting Selinium to "run"

[girish]

@doodlemania2 Happy to help!

Yes, I am on Ubuntu but @nebulon is on Arch, so that should work as well. Which part are you stuck in? Did the npm install go through? What's your setup like?

[BrutalBirdie]

@girish Arg how did you enable the browser on the right? (Selenium/Chromedriver)

[nebulon]

It will be initialized by the test.js script. The requires are important as well as the before and after hook. See https://git.cloudron.io/cloudron/minio-app/-/blob/master/test/test.js

[BrutalBirdie]

@nebulon Yea I just wanted to reply never mind, I copied the tests from the alltube app which do not even use selenium so yeeaa my mistake.

Now I took the nextcloud tests and will learn from them

[erics]

This post is deleted!