Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Writing automated tests for packages

    App Packaging & Development
    7
    12
    683
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • girish
      girish Staff last edited by girish

      Hi All,
      A number of packages are currently marked as WIP now. Some of you have reached out and asked on what is blocking releasing them. What's blocking them essentially is automated tests. We maintain a large number of apps and as a small team what makes it possible for us to push reliable updates quickly is our automated tests. These tests, while it takes a bit of time to write them, help us in maintaining the package in the long run. Without the tests, we are simply not in a position to update apps quickly.

      These automated tests are browser based selenium tests. The focus of the tests is not to focus on the app testing but to focus on the packaging. What the tests do is:

      • Install the app the Cloudron. By convention, this is the 'test' subdomain.
      • Open the app in a browser and login with admin credentials or Cloudron credentials and check if the login works
      • Depending on the app, it will create something like a document or a post or file or chat message etc
      • Restart the app and check if the app works
      • Backup the app and then restore the app from the backup. Then we check if the above post/file/chat message is still around. This provides sanity test that the app is backing up properly
      • Move the app to a different location. By convention, this is the 'test2' subdomain. Check if the app has the above post/file/chat. This checks if the package is setting up domain/app urls correctly.
      • Finally, to test updates, it installs the current version of the app from the app store. It logs in and creates post/file/chat. It then applies the current app package build on top of the app store version and checks if data is still OK.

      The tests while they follow a common pattern have to be individually tuned per app (specifically the login and the post/file/chat message is different for each app). Writing the tests requires some javascript knowledge and also some xpath knowledge but we have written tests for enough packages that we have automated all sorts of things. Just ask here and we can help.

      Running existing tests

      Before writing tests, the first thing to do is to simply start with ensuring you can run the tests of some existing package. For example, say https://git.cloudron.io/cloudron/minio-app . To run the tests:

      • Install the cloudron CLI tool and do cloudron login
      • git clone https://git.cloudron.io/cloudron/minio-app
      • cloudron build
      • cd minio-app/test
      • npm install
      • USERNAME=cloudronusername PASSWORD=cloudronpassword node_modules/.bin/mocha --bail test.js

      The above will produce an output like this:

        Application life cycle test
          - build app
      App is being installed.
      
       => Queued 
       => Cleaning up old install 
       => Registering subdomains 
       => Downloading image ..
       => Creating container .
       => Waiting for DNS propagation ..................
       => Wait for health check .....
      
      App is installed.
          ✓ install app (37267ms)
          ✓ can get app information (735ms)
          ✓ can login (3062ms)
          ✓ can add bucket (1304ms)
      ...
      

      Writing tests

      To write tests, usually, I just copy an existing app's test directory straight into the new one. And then, I slowly start fixing the tests. Tip: you can just change it() to xit() to skip a test.

      We have also not tested running tests on Mac or Windows. Though, I am confident that it works well or can be made to work well on Mac.

      Long story short, if you can help us write tests for the packages, we can get it published almost immediately. Please let us know what information you require from us to help writing these tests.

      1 Reply Last reply Reply Quote 8
      • girish
        girish Staff last edited by girish

        I recorded a quick video showing host the tests run. The browser on the left is my Firefox that just shows the Cloudron dashboard. The browser that appears on the right is Chrome and it is where the automated testing happens.

        Youtube Video

        EDIT: OK, this is funny. @nebulon was wondering why I had some dark music in the video. I thought I had recorded a silent video and did not even know there was some background music 🙂 Investigating, I found out that my youtube at some point has started auto-playing "3 HOURS Most EPIC POWERFUL BATTLE MUSIC"! and OBS was happily recording that. Since, I don't have a headset connected to this setup, I had no idea. Funnily, when I uploaded this video, it immediately told me there is a "copyright claim" and I thought "yeah, right". Now I know why.

        murgero BrutalBirdie 2 Replies Last reply Reply Quote 12
        • murgero
          murgero App Dev @girish last edited by

          @girish Beautiful I'll get tests going for Directus then. Since I am being paid for it's packaging.

          --
          https://urgero.org
          ~ Professional Nerd. Freelance Programmer. ~
          Matrix: @murgero:urgero.org

          marcusquinn 1 Reply Last reply Reply Quote 4
          • marcusquinn
            marcusquinn @murgero last edited by

            @murgero Yay 😁 unit testing definitely a great string to add to your bow and hopefully time-saver for adding more apps without adding maintenance whack-a-mole overhead too when you'd hope the tests will do the monitoring so we don't need to so much.

            We're not here for a long time - but we are here for a good time :)
            Jersey/UK
            Work & Ecommerce Advice: https://brandlight.org
            Personal & Software Tips: https://marcusquinn.com

            1 Reply Last reply Reply Quote 1
            • girish
              girish Staff last edited by

              @murgero Thanks, if you have any question, feel free to ask and I am happy to help.

              1 Reply Last reply Reply Quote 1
              • doodlemania2
                doodlemania2 App Dev last edited by

                Can you tell me a bit about your setup? I assume this is a Linux desktop distro? Any pointers on initial setup would be appreciated!

                doodlemania2 1 Reply Last reply Reply Quote 0
                • doodlemania2
                  doodlemania2 App Dev @doodlemania2 last edited by

                  @girish - i watched the video and was able to discern most of your setup, but would still appreciate a sanity check if you have time on how you got your test harness up and running. Having troubles getting Selinium to "run"

                  1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff last edited by

                    @doodlemania2 Happy to help!

                    Yes, I am on Ubuntu but @nebulon is on Arch, so that should work as well. Which part are you stuck in? Did the npm install go through? What's your setup like?

                    1 Reply Last reply Reply Quote 0
                    • BrutalBirdie
                      BrutalBirdie Staff @girish last edited by

                      @girish Arg how did you enable the browser on the right? (Selenium/Chromedriver)

                      Like my work? Consider donating a beer 🍻 Cheers!

                      1 Reply Last reply Reply Quote 0
                      • nebulon
                        nebulon Staff last edited by nebulon

                        It will be initialized by the test.js script. The requires are important as well as the before and after hook. See https://git.cloudron.io/cloudron/minio-app/-/blob/master/test/test.js

                        BrutalBirdie 1 Reply Last reply Reply Quote 0
                        • BrutalBirdie
                          BrutalBirdie Staff @nebulon last edited by

                          @nebulon Yea I just wanted to reply never mind, I copied the tests from the alltube app which do not even use selenium so yeeaa my mistake.

                          Now I took the nextcloud tests and will learn from them 🙂

                          Like my work? Consider donating a beer 🍻 Cheers!

                          1 Reply Last reply Reply Quote 0
                          • E
                            erics App Dev last edited by erics

                            This post is deleted!
                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Powered by NodeBB