YunoHost/doc
1
0
Fork 0
mirror of https://github.com/YunoHost/doc.git synced 2024-09-03 20:06:26 +02:00
Code Issues Projects Releases Packages Wiki Activity

app packaging rework: add scripts, doc and testing pages

Browse source
This commit is contained in:
Alexandre Aubin 2023-01-28 19:00:44 +01:00
parent ddeaa4ba65
commit 5674a80e2d
3 changed files with 201 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

120
pages/06.contribute/10.packaging_apps/20.scripts/scripts.md
View file

@ -8,17 +8,129 @@ routes:
---
# General architecture ?
App scripts are the essential logic defining what happens when an app is `install`ed, `remove`d, `upgrade`d, `backup`ed, or `restore`d. They are written in [Bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) which is the same thing as stuff you type in interactive mode in a terminal, though we added a bunch of custom YunoHost functions that we call `helpers` to standardize many common operations - for example adding the nginx configuration.
# Global args like $app, install args, etc.
Note that starting from packaging v2, the logic or what happens when an app is installed, etc. is also contained partially in the resource configuration in the `manifest.toml`.
In the `scripts` folder of an app, you can expect to find:
```text
manifest.toml
scripts/
- _common.sh # Some "common" definition, typically custom helpers or global variables used accross all scripts
- install # The install procedure
- remove # The remove procedure
- upgrade # The upgrade procedure
- backup # The backup procedure - in fact it only "declares" what should be backup / no actual real backup happens at this point except dumping SQL databases
- restore # The restore procedure
- change_url # Some apps do also provide a change url script, which corresponds to changing the URL endpoint of the app, which may be as simple as changing the nginx conf, or may involve significant changes in the app DB
```
Here is an example of the simple install script for the `helloworld` app:
```bash
#!/bin/bash
# This is where we load the official YunoHost helpers
source /usr/share/yunohost/helpers
#=================================================
# DOWNLOAD, CHECK AND UNPACK SOURCE
# This is where we would usually fetch the .tar.gz archive
# from the upstream and extract it into our local install dir
#=================================================
# At the beginning of each major operation, we call this helper
# that creates a progress bar
ynh_script_progression --message="Setting up source files..." --weight=1
echo "Hello world!" > $install_dir/index.html
chown -R www-data: "$install_dir"
#=================================================
# NGINX CONFIGURATION
# This is where the nginx conf snippet for the app is created using the
# configuration template provided in conf/nginx.conf
# and added to /etc/nginx/conf.d/$domain.d/$app.conf
#=================================================
ynh_script_progression --message="Configuring nginx web server..." --weight=1
ynh_add_nginx_config
```
Note that the scripts are run with the `set -eu` options (except for the remove script), which means that any failing command or use of non-existing variable will trigger an error and stop the script execution.
# Variables available in a script context
Special variables are automatically defined in the context of a script:
- `$app` is the app ID. It will typically be the ID from the app's manifest.toml, for example `helloworld`, but will be `helloworld__2`, `__3` etc for multi-instance installs.
- During install, answers to install questions are automatically available as bash variables. For example, the `$domain` setting corresponds to the `domain` question, same for `$prefered_pet`, etc... Note that - apart from special questions such as `init_main_permission` or user-provided passwords - they are also automatically saved as settings (cf next section).
- During other scripts, all app settings are also loaded and automatically available.
- Note that some settings are automatically created/updated by app ressources. For example, the `install_dir` setting will automatically be available too and corresponds to typically `/var/www/$app`
# Setting system
Application often need to store long term information in between scripts triggered by the admin. For this, YunoHost has a key-value store for each application called "setting" and is stored in `/etc/yunohost/apps/$app/settings.yml`.
Apps can interact with this key value store in this way:
```bash
# Retrieve a setting into variable "db_name"
db_name=$(ynh_app_setting_get --app=$app --key=db_name)
# Update a setting
ynh_app_setting_set --app=$app --key=db_name --value=$db_name
```
# Helper system
# Configuration/template system (ynh_add_config, __FOOBAR__ syntax, common helpers like ynh_add_nginx_config etc)
We call helpers a set of custom bash function created by the YunoHost project to standardize common operations accross all apps. They are all prefixed with `ynh_`. The full list and documentation of these helpers is available on [this page](/packaging_apps_helpers). Some of these helpers are now partially obsolete as they are now handled by the core via app resources.
# Common operations
Here is the list of the major ones:
- `ynh_app_setting_get` / `ynh_app_setting_set`
- `ynh_script_progression`
- `ynh_setup_source`
- nginx: `ynh_add_nginx_config` / `ynh_remove_nginx_config`
- php: `ynh_add_fpm_config` / `ynh_remove_fpm_config`
- systemd: `ynh_add_systemd_config` / `ynh_remove_systemd_config`
- fail2ban: `ynh_add_fail2ban_config` / `ynh_remove_fail2ban_config`
- custom: `ynh_add_config`
- nodejs: `ynh_install_nodejs` / `ynh_use_nodejs`
- `ynh_exec_warn_less`
- `ynh_local_curl`
- `ynh_secure_remove`
- `ynh_backup` / `ynh_restore_file`
# Configuration/template system
App scripts will often need to create a bunch of configuration files.
Configuration templates are canonically stored provided in the `conf/` folder of the app, such as `nginx.conf`, `extra_php-fpm.conf`, `systemd.conf`, or `some-custom-app-conf.env` ...
In these templates, you can use the syntax `__FOOBAR__` which will automatically be replaced by the variable `$foobar` during runtime, when the conf is installed via the `ynh_add_*_config` helpers.
# App sources
App scripts will often need to download the upsteam app sources to be deployed in the install dir, via `ynh_setup_source`.
The `ynh_setup_source` is coupled to another conf file, usually `conf/app.src` which describes where to download the sources, how to check integrity, and how to extract the source.
For example, this is an `app.src` for wordpress:
```text
SOURCE_URL=https://downloads.wordpress.org/release/wordpress-6.1.1.zip # The url of the sources
SOURCE_SUM=088280b34aebc7331693e729d8e6b05eb8b9998c.... # The sha256 checksum
SOURCE_SUM_PRG=sha256sum
SOURCE_FORMAT=zip # The format is zip
SOURCE_IN_SUBDIR=true # This is wether or not the sources are directly in the install root
SOURCE_EXTRACT=true # (yes, we want to extract the zip)
```
# Common operations (TODO/FIXME)
### installing/upgrading app sources
### adding configurations

39
pages/06.contribute/10.packaging_apps/30.doc/doc.md
View file

@ -7,7 +7,38 @@ routes:
default: '/packaging_app_doc'
---
- DESCRIPTION.md, ADMIN.md
- autogenerated README
- screenshots
- pre_install, post_install, pre_upgrade, post_upgrade
Properly documenting your app is important for user experience ;). YunoHost provides several mechanism to display information to users.
# Extensive description : `doc/DESCRIPTION.md` and `doc/screenshots/`
You are encouraged to add a `doc/DESCRIPTION.md` which should contain a more extensive description than the short description contained in `manifest.toml`. This usually corresponds to listing the key features of the app.
You are also encouraged to add a `.png` or `.jpg` screenshot of what the app looks like in `doc/screenshots/`. There is no constrain on the exact filename, but the file weight should be kept below 512kb.
This description and screenshot will be shown when the admin open the catalog page for this app and the app info page in the webadmin after the admin installs this app.
You can also add translated versions of the `.md` file in, for example, `doc/DESCRIPTION_fr.md`, `_es.md`, `_it.md`, etc.
If your app repository is part of the YunoHost-Apps org, the provided description will be used to auto-regenerate the README.md of your github repo via `yunohost-bot`.
# Specific notes for admins : `doc/ADMIN.md`, `doc/<whatever>.md`
Sometimes, you may want to ship YunoHost-specific notes regarding the administration of this app. For example, integrating OnlyOffice inside Nextcloud.
You can do so by adding a `doc/ADMIN.md` file or even a `doc/<whatever>.md` page for each specific topic - and similarly add `_<lang>` suffix for translations.
Note that you can even use the `__FOOBAR__` syntax which will automatically be replaced with the `foobar` setting.
These notes will be available in the app info page in the webadmin after the app installation.
# Pre/post-install notes, pre/post-upgrade notes
Sometimes, you may want to display important information to the admin at key points in the app's life cycle. You can do so by providing special markdown files:
- `doc/PRE_INSTALL.md`: displayed right before the installation (for example to warn that « This app install is expected to take around 30 minutes, be patient! » or « This app will automatically add 1GB swap to your system »)
- NB: try to not overlap with the anti-feature tags from the catalog (cf Publishing your app in the catalog) which can be used to warn that the app's upstream is alpha-stage or deprecated among other things.
- `doc/POST_INSTALL.md`: displayed in a popup after the installation AND a dismissable note in the webadmin app info view.
- `doc/PRE_UPGRADE.md`: displayed right before any upgrade of this app (NB: the pre-upgrade note from the NEW version will be used, not the one from the installed version)
- You can also create `doc/PRE_UPGRADE.d/{version}.md` to have a note displayed before upgrading to a version equal or higher than `{version}`
- `doc/POST_UPGRADE.md`: displayed in a popup after the upgrade AND a dismissable note in the webadmin app info view.
Same as `ADMIN.md` and others: in these files, you can use the `__FOOBAR__` syntax which will automatically be replaced with the `foobar` setting.

51
pages/06.contribute/10.packaging_apps/40.testing/testing.md
View file

@ -7,5 +7,54 @@ routes:
default: '/packaging_testing'
---
- CI, quality tools (linter, package check, tests.toml, dashboards, triggering jobs on the CI)
Once you're done writing you app package, you'll want to check that everything works correctly. At first, you can manually try to install your app on some test server of your own, but testing everything manually often becomes tedious ;).
The YunoHost maintains several tools to automatically and somewhat "objectively" analyze and tests our 400+ apps catalog.
# Package linter
The [package linter](https://github.com/YunoHost/package_linter) performs a static analysis of your app to check a bunch of recommended practice.
It is pretty straightforward to run considering that you should only need Python installed.
# Package check
[Package check](https://github.com/YunoHost/package_check) is a more elaborate software that will tests many scenarios for you app such as:
- installing, removing and reinstalling your app + validating that the app can indeed be accessed on its URL endpoint (with no 404/502 error)
- when installing on a root domain (`domain.tld/`)
- when installing in a domain subpatch (`domain.tld/foobar`)
- installing in private mode
- installing multiple instances
- upgrading from the same version
- upgrading from older versions
- backup/restore
- ...
Package check will then summarize the results and compute a quality level ranging from 0 to 8.
To run its tests, package check uses a LXC container to manipulate the package in a clean environment without any previous installations. Installing LXC/LXD is unfortunately not that straighforward (it kinda depends on your OS/distribution), though we documented how to do so [in the README](https://github.com/YunoHost/package_check#deploying-package_check).
## Package check's `tests.toml`
Package check interfaces with your app's `tests.toml` which allows to control a few things about which tests are run - though it is usually pretty empty as many things can be deduced from the app's manifest.
More info about the syntax here are also available [in the README](https://github.com/YunoHost/package_check#teststoml-syntax)
# Continous integration (CI)
The YunoHost project also developed an interface called [`yunorunner`](https://github.com/YunoHost/yunorunner) which interfaces with `package_check`, handles a job queue, and automatically add jobs to the queue using some triggers.
The two major ones are:
- [The "official" CI](https://ci-apps.yunohost.org/ci): This where the "official" quality level of each app comes from. Jobs are triggered after each commit on the repo's master branch.
- [The "dev" CI](https://ci-apps-dev.yunohost.org/ci/): This is where people validate their pull request which is often more convenient than running `package_check` yourself, and has the advantage of the results being automatically public, which facilitates collective debugging.
Members of the YunoHost-Apps organization can trigger jobs on the dev CI directly from a pull request simply by commenting something like `!testme` (cf for example [here](https://github.com/YunoHost-Apps/nextcloud_ynh/pull/532#issuecomment-1402751409)). A .png summary of the tests will be automatically displayed once the job completes (and you can click the link to see the entire job execution and debug it).
### Why create `package_check` + `yunorunner` rather than using well-known solutions like Gitlab-CI ?
Constrain 1 : Gitlab-CI or other similar solutions are mostly based around Docker, while we use LXC. In particular, we do want to reuse LXC snapshots of successful install during other tests (upgrade, backup/restore, ..) rather than reinstalling the app from scratch everytime, which drastically reduces the test time. We could do so using Gitlab artifacts, but such artifacts are automatically made public which is not convenient because they contain a full filesystem and their only use it to speed up the test process. Moreover, in the Gitlab-CI paradigm, jobs are not running on the same machine and they would need to download the snapshot which can be lengthy. The other mechanism, caching, is explicitly advertised as not reliable in Gitlab's-CI doc. What would be helpful would be some non-public artifact system (see similar discussion [here](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/336))
Constrain 2 : Our test workflow is somewhat complex and we have 400+ apps to maintain. It's not acceptable to have to duplicate/maintain the CI logic in app each repository, so we need some sort of repo that handles the CI logic, while still being to supersed some behavior from the app repo itself. Gitlab-CI has some syntax which can allow this, but this remains quite laborious.
Constrain 3 : Having a common job queue/dashboard UI accross all app. In the Gitlab-CI paradigm, each repository has its own job queue/dashboard UI, but this is not really convenient.