doc/pages/06.contribute/10.packaging_apps/40.testing/testing.md

title	template	taxonomy	routes
Testing your app	docs	category docs	default /packaging_testing

Once you're done writing you app package, you'll want to check that everything works correctly. At first, you can manually try to install your app on some test server of your own, but testing everything manually often becomes tedious ;).

The YunoHost maintains several tools to automatically and somewhat "objectively" analyze and tests our 400+ apps catalog.

Package linter

The package linter performs a static analysis of your app to check a bunch of recommended practice.

It is pretty straightforward to run considering that you should only need Python installed.

Package check

Package check is a more elaborate software that will tests many scenarios for you app such as:

• installing, removing and reinstalling your app + validating that the app can indeed be accessed on its URL endpoint (with no 404/502 error)
  • when installing on a root domain (domain.tld/)
  • when installing in a domain subpatch (domain.tld/foobar)
  • installing in private mode
  • installing multiple instances
• upgrading from the same version
• upgrading from older versions
• backup/restore
• ...

Package check will then summarize the results and compute a quality level ranging from 0 to 8.

To run its tests, package check uses a LXC container to manipulate the package in a clean environment without any previous installations. Installing LXC/LXD is unfortunately not that straighforward (it kinda depends on your OS/distribution), though we documented how to do so in the README.

Package check's tests.toml

Package check interfaces with your app's tests.toml which allows to control a few things about which tests are run - though it is usually pretty empty as many things can be deduced from the app's manifest.

More info about the syntax here are also available in the README

Continous integration (CI)

The YunoHost project also developed an interface called yunorunner which interfaces with package_check, handles a job queue, and automatically add jobs to the queue using some triggers.

The two major ones are:

• The "official" CI: This where the "official" quality level of each app comes from. Jobs are triggered after each commit on the repo's master branch.
• The "dev" CI: This is where people validate their pull request which is often more convenient than running package_check yourself, and has the advantage of the results being automatically public, which facilitates collective debugging.

Members of the YunoHost-Apps organization can trigger jobs on the dev CI directly from a pull request simply by commenting something like !testme (cf for example here). A .png summary of the tests will be automatically displayed once the job completes (and you can click the link to see the entire job execution and debug it).

Why create package_check + yunorunner rather than using well-known solutions like Gitlab-CI ?

Constrain 1 : Gitlab-CI or other similar solutions are mostly based around Docker, while we use LXC. In particular, we do want to reuse LXC snapshots of successful install during other tests (upgrade, backup/restore, ..) rather than reinstalling the app from scratch everytime, which drastically reduces the test time. We could do so using Gitlab artifacts, but such artifacts are automatically made public which is not convenient because they contain a full filesystem and their only use it to speed up the test process. Moreover, in the Gitlab-CI paradigm, jobs are not running on the same machine and they would need to download the snapshot which can be lengthy. The other mechanism, caching, is explicitly advertised as not reliable in Gitlab's-CI doc. What would be helpful would be some non-public artifact system (see similar discussion here)

Constrain 2 : Our test workflow is somewhat complex and we have 400+ apps to maintain. It's not acceptable to have to duplicate/maintain the CI logic in app each repository, so we need some sort of repo that handles the CI logic, while still being to supersed some behavior from the app repo itself. Gitlab-CI has some syntax which can allow this, but this remains quite laborious.

Constrain 3 : Having a common job queue/dashboard UI accross all app. In the Gitlab-CI paradigm, each repository has its own job queue/dashboard UI, but this is not really convenient.