diff --git a/.gitignore b/.gitignore index f0ff6f7..cda4d28 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ *.swp *~ Notes +config diff --git a/README-fr.md b/README-fr.md deleted file mode 100644 index e8ac95c..0000000 --- a/README-fr.md +++ /dev/null @@ -1,207 +0,0 @@ -Package checker pour YunoHost -================== - -[Projet YunoHost](https://yunohost.org/#/) - -> [Read this readme in english](README.md) - -Ensemble de tests unitaires pour vérifier les packages Yunohost. -Le script `package_check.sh` effectue une succession de test sur un package afin de vérifier sa capacité à s'installer et se désinstaller dans différents cas. -Le résultats des tests est affiché directement et stocké dans le fichier Test_results.log - -Le script est capable d'effectuer les tests suivant: -- Vérification du package avec [package linter](https://github.com/YunoHost/package_linter) -- Installation en sous-dossier -- Installation à la racine du domaine -- Installation sans accès par url (Pour les applications n'ayant pas d'interface web) -- Désinstallation -- Réinstallation après désinstallation -- Installation en privé -- Installation en public -- Upgrade depuis la même version du package -- Upgrade depuis une précédente version du package -- Backup -- Restore après suppression de l'application -- Restore sans installation préalable -- Installation multi-instance -- Test de port déjà utilisé -- Test du script change_url -- Test des actions et configurations disponible dans le config-panel - -Package check utilise un conteneur LXC pour créer un environnement de test propre sans résidus d'installations précédentes. - -Usage: -Pour une app dans un dossier: `./package_check.sh APP_ynh` -Pour une app sur github: `./package_check.sh https://github.com/USER/APP_ynh` - -Il est nécessaire de fournir, à la racine du package de l'app à tester, un fichier `check_process` pour indiquer au script les arguments attendu et les tests à effectuer. -Si ce fichier n'est pas présent, package_check sera utilisé en mode dégradé. Il va tenter de repérer les arguments domain, path et admin dans le manifest pour exécuter un nombre restreint de test, en fonction des arguments trouvés. - ---- -## Déploiement de package_check - -Package_check ne peut être installer que sur Debian Stretch ou Debian Buster. - -``` -git clone https://github.com/YunoHost/package_check -package_check/sub_scripts/lxc_build.sh - -package_check/package_check.sh APP_ynh -``` - ---- -## Syntaxe du fichier `check_process` -> A l'exception des espaces, la syntaxe du fichier doit être scrupuleusement respectée. - -``` -;; Nom du test -# Commentaire ignoré - ; pre-install - echo -n "Placez ici vos commandes a exéxuter dans le conteneur, " - echo "avant chaque installation de l'application." - ; Manifest - domain="domain.tld" (DOMAIN) - path="/path" (PATH) - admin="john" (USER) - language="fr" - is_public=1 (PUBLIC|public=1|private=0) - password="password" - port="666" (PORT) - ; Actions - action_argument=arg1|arg2 - is_public=1|0 - ; Config_panel - main.categorie.config_example=arg1|arg2 - main.overwrite_files.overwrite_phpfpm=1|0 - main.php_fpm_config.footprint=low|medium|high|specific - main.php_fpm_config.free_footprint=20 - main.php_fpm_config.usage=low|medium|high - ; Checks - pkg_linter=1 - setup_sub_dir=1 - setup_root=1 - setup_nourl=0 - setup_private=1 - setup_public=1 - upgrade=1 - upgrade=1 from_commit=65c382d138596fcb32b4c97c39398815a1dcd4e8 - backup_restore=1 - multi_instance=1 - port_already_use=1 (XXXX) - change_url=1 - actions=1 - config_panel=1 -;;; Levels - Level 5=auto -;;; Options -Email= -Notification=none -;;; Upgrade options - ; commit=65c382d138596fcb32b4c97c39398815a1dcd4e8 - name=Name of this previous version - manifest_arg=domain=DOMAIN&path=PATH&admin=USER&password=pass&is_public=1& -``` -### `;; Nom du test` -Nom du scénario de test qui sera effectué. -On peut créer autant de scénario de test que voulu, tous ayant la même syntaxe. -Les différents scénarios de test seront exécutés successivement. - -### `; pre-install` -*Instruction optionnelle* -Si vous devez exécuter une commande ou un groupe de commandes avant l'installation, vous pouvez utiliser cette instruction. -Toutes les commandes ajoutées après l'instruction `; pre-install` seront exécutées dans le conteneur avant chaque installation de l'application. - -### `; Manifest` -Ensemble des clés du manifest. -Toutes les clés du manifest doivent être renseignée afin de procéder à l'installation. -> Les clés de manifest données ici ne le sont qu'à titre d'exemple. Voir le manifest de l'application. - -Certaines clés de manifest sont indispensables au script pour effectuer certains test. Ces clés doivent être mises en évidence afin que le script soit capable de les retrouver et de changer leur valeur. -`(DOMAIN)`, `(PATH)`, `(USER)` et `(PORT)` doivent être mis en bout de ligne des clés correspondantes. Ces clés seront modifiées par le script. -`(PUBLIC|public=1|private=0)` doit, en plus de correspondre à la clé de visibilité public, indiquer les valeurs du manifest pour public et privé. - -### `; Actions` -List des arguments pour chaque action nécessitant un argument. -`action_argument` est le nom de l'argument, ainsi que vous pouvez le trouver à la fin de [action.arguments.**action_argument**]. -`arg1|arg2` sont les différents arguments à utiliser pour les test. Vous pouvez mettre autant d'arguments que désiré, séparé par `|`. - -*Seul `actions.toml` peut être testé par package_check, pas `actions.json`.* - -### `; Config_panel` -List des arguments pour chaque configuration de config_panel. -`main.categorie.config_example` est l'entrée toml complète pour l'argument de cette configuration. -`arg1|arg2` sont les différents arguments à utiliser pour les test. Vous pouvez mettre autant d'arguments que désiré, séparé par `|`. - -*Seul `config_panel.toml` peut être testé par package_check, pas `config_panel.json`.* - -### `; Checks` -Ensemble des tests à effectuer. -Chaque test marqué à 1 sera effectué par le script. -Si un test est absent de la liste, il sera ignoré. Cela revient à le noter à 0. -- `pkg_linter`: Vérification du package avec [package linter](https://github.com/YunoHost/package_linter) -- `setup_sub_dir`: Installation dans un path. -- `setup_root`: Installation à la racine du domaine. -- `setup_nourl`: Installation sans accès http. Ce test ne devrait être choisi que pour les applications ne disposant pas d'une interface web. -- `setup_private`: Installation en privé. -- `setup_public`: Installation en public. -- `upgrade`: Upgrade du package sur la même version. Test uniquement le script upgrade. -- `upgrade from_commit`: Upgrade du package à partir du commit spécifié vers la dernière version. -- `backup_restore`: Backup et restauration. -- `multi_instance`: Installation de l'application 2 fois de suite, pour vérifier sa capacité à être multi-instance. -- `port_already_use`: Provoque une erreur sur le port en l'ouvrant avant le script d'install. - Le test` port_already_use` peut éventuellement prendre en argument un numéro de port. Si celui-ci n'est pas dans le manifest. - Le numéro de port doit alors être noté entre parenthèse, il servira au test de port. -- `change_url`: Test le script change_url de 6 manières différentes, Root vers un path, path vers un autre path et path vers root. Et la même chose avec un autre domaine. -- `actions`: Toutes les actions disponible dans actions.toml -- `config_panel`: Toutes les configurations disponible dans config_panel.toml - -### `;;; Levels` -Les [niveaux](https://yunohost.org/#/packaging_apps_levels_fr) 1 à 8 sont déterminés automatiquement. -A l'exception du niveau 5, vous ne pouvez plus forcer une valeur pour un niveau. -Le niveau 5 est déterminé par les résultats de [package linter](https://github.com/YunoHost/package_linter). -La valeur par défaut pour ce niveau est `auto`, cependant, si nécessaire, vous pouvez forcer la valeur pour ce niveau en la fixant à `1`, pour un résultat positif, ou à `0`, pour un résultat négatif. -Si vous le faites, veuillez ajouter un commentaire pour justifier pourquoi vous forcez ce niveau. - -### `;;; Options` -Options supplémentaires disponible dans le check_process. -Ces options sont facultatives. - -- `Email` : Permet d'indiquer un email alternatif à celui qui est présent dans le manifest pour les notifications de package check, lorsque celui-ci s'exécute en contexte d'intégration continue. -- `Notification` : Degré de notification souhaité pour l'application. Il y a 3 niveaux de notification disponible. - - `down` : Envoi un mail seulement si le niveau de l'application a baissé. - - `change` : Envoi un mail seulement si le niveau de l'application a changé. - - `all` : Envoi un mail pour chaque test de l'application, quel que ce soit le résultat. - -### `;;; Upgrade options` -*Instruction optionnelle* -Pour chaque commit indiqué pour un upgrade, permet d'indiquer un nom pour cette version et les paramètres du manifest à utiliser lors de l'installation préliminaire. -En cas d'absence de nom, le commit sera utilisé. -De même en cas d'absence d'arguments pour le manifest, les arguments du check_process seront utilisés. -> 3 variables doivent être utilisées pour les arguments du manifest, DOMAIN, PATH et USER. - ---- -Le script `package_check.sh` accepte 6 arguments en plus du package à tester. -- `--bash-mode`: Rend le script autonome. Aucune intervention de l'utilisateur ne sera nécessaire. - La valeur de auto_remove est ignorée. -- `--branch=nom-de-branche`: Teste une branche du dépôt plutôt que de tester master. Permet de tester les pull request. - Vous pouvez utiliser une url avec une branche, https://github.com/YunoHost-Apps/APP_ynh/tree/my_branch, pour utiliser implicitement cet argument. -- `--build-lxc`: Installe LXC et créer le conteneur debian Yunohost si nécessaire. -- `--force-install-ok`: Force la réussite des installations, même si elles échouent. Permet d'effectuer les tests qui suivent même si l'installation a échouée. -- `--interrupt`: Force l'option auto_remove à 0, le script marquera une pause avant chaque suppression d'application. -- `--help`: Affiche l'aide du script - ---- -## LXC - -Package check utilise la virtualisation en conteneur pour assurer l'intégrité de l'environnement de test. -L'usage de LXC apporte une meilleure stabilité au processus de test, un test de suppression échoué n'entraine pas l'échec des tests suivant, et permet de garder un environnement de test sans résidus de test précédents. En revanche, l'usage de LXC augmente la durée des tests, en raison des manipulations du conteneur et de la réinstallation systématique des dépendances de l'application. - -Il faut prévoir également un espace suffisant sur l'hôte, au minimum 6Go pour le conteneur, ses snapshots et sa copie de sauvegarde. - -L'usage de LXC est facilité par 4 scripts, permettant de gérer la création, la mise à jour, la suppression et la réparation du conteneur. -- `lxc_build.sh`: lxc_build installe LXC et ses dépendances, puis créer le conteneur debian. - Il ajoute ensuite le support réseau, installe YunoHost et le configure. Et enfin configure un accès ssh. - L'accès ssh par défaut est `ssh -t pchecker_lxc` -- `lxc_upgrade.sh`: Effectue la mise à jour du conteneur à l'aide d'apt-get et recréer le snapshot. -- `lxc_remove.sh`: Supprime le conteneur LXC, son snapshot et sa sauvegarde. Désinstalle LXC et déconfigure le réseau associé. -- `lxc_check.sh`: Vérifie le conteneur LXC et tente de le réparer si nécessaire. diff --git a/README.md b/README.md index 50098a8..9c487ea 100644 --- a/README.md +++ b/README.md @@ -3,26 +3,23 @@ Package checker for YunoHost [YunoHost project](https://yunohost.org/#/) -> [Lire ce readme en francais](README-fr.md) - Set of unit tests to check YunoHost packages. The `package_check.sh` script perform a series of tests on a package for verify its capability to be installed and removed in different situation. The test results are printed directly in the terminal and stored in the log file Test_results.log The script is able to perform the following tests: - Check the package with [package linter](https://github.com/YunoHost/package_linter) -- Installation in a subdir - Installation at the root of a domain -- Installation without url access (For apps without web UI) -- Uninstallation -- Reinstallation after uninstallation +- Installation in a subpath +- Installation without url access (For non-web apps) +- Removal +- Reinstallation after removal - Private installation -- Public installation - Upgrade from same version of the package - Upgrade from a previous version of the package - Backup -- Restore the application after uninstallation -- Restoration without an previous installation +- Restore from a backup after removal +- Restore from a backup on a fresh system - Multi-instances installation - Test with the port already used - Test of change_url script @@ -40,13 +37,11 @@ If this file is not present, package_check will be used in downgraded mode. It w --- ## Deploying package_check -Package_check can only be installed on Debian Stretch or Debian Buster. - ``` git clone https://github.com/YunoHost/package_check -package_check/sub_scripts/lxc_build.sh - -package_check/package_check.sh APP_ynh +cd package_check +./build_base_lxc.sh +./package_check.sh your_app_ynh ``` --- @@ -54,19 +49,18 @@ package_check/package_check.sh APP_ynh > Except spaces, the syntax of this file must be respected. ``` -;; Test name +;; Default test serie # Comment ignored ; pre-install echo -n "Here your commands to execute in the container" echo ", before each installation of the app." ; Manifest - domain="domain.tld" (DOMAIN) - path="/path" (PATH) - admin="john" (USER) + # You need to provide default values for installation parameters ... + # EXCEPT for special args: domain, path, admin, and is_public + # which will be filled automatically during tests language="fr" - is_public=1 (PUBLIC|public=1|private=0) password="password" - port="666" (PORT) + port="666" ; Actions action_argument=arg1|arg2 is_public=1|0 @@ -82,26 +76,21 @@ package_check/package_check.sh APP_ynh setup_root=1 setup_nourl=0 setup_private=1 - setup_public=1 upgrade=1 upgrade=1 from_commit=65c382d138596fcb32b4c97c39398815a1dcd4e8 backup_restore=1 multi_instance=1 - port_already_use=1 (XXXX) - change_url=1 - actions=1 - config_panel=1 -;;; Levels - Level 5=auto -;;; Options -Email= -Notification=none + port_already_use=1 (66) + change_url=0 + actions=0 + config_panel=0 ;;; Upgrade options ; commit=65c382d138596fcb32b4c97c39398815a1dcd4e8 name=Name of this previous version manifest_arg=domain=DOMAIN&path=PATH&admin=USER&password=pass&is_public=1& ``` -### `;; Test name` + +### `;; Default test serie` A name for the series of tests to perform. It's possible to create multiple tests series, all with the same syntax. All different series will be performed sequentially. @@ -116,10 +105,6 @@ List of manifest keys. All manifest keys need to be filled to perform the installation. > The manifest keys already in the file here are simply examples. Check the package manifest. -Some manifest keys are mandatory for the script to performs some tests. This keys must be highlighted, so the script is able to find them and modify their values. -`(DOMAIN)`, `(PATH)`, `(USER)` and `(PORT)` must be placed at the end of corresponding key. These keys will be changed by the script. -`(PUBLIC|public=1|private=0)` must, aside of marking the public key, indicate the values for public and private. - ### `; Actions` List of arguments for each action that needs an argument. `action_argument` is the name of the argument, as you can find at the end of [action.arguments.**action_argument**]. @@ -139,11 +124,10 @@ List of tests to perform. Each test set to 1 will be performed by the script. If a test is not in the list, it will be ignored. It's similar to set the test at 0. - `pkg_linter`: Check the package with [package linter](https://github.com/YunoHost/package_linter) -- `setup_sub_dir`: Installation in a path. - `setup_root`: Installation at the root of a domain. -- `setup_nourl`: Installation without http access. This test should be perform only for apps that does not have a web interface. +- `setup_sub_dir`: Installation in a path. +- `setup_nourl`: Installation with no domain/path. This test is meant for non-web apps - `setup_private`: Private installation. -- `setup_public`: Public installation. - `upgrade`: Upgrade the package to the same version. Only to test the upgrade script. - `upgrade from_commit`: Upgrade the package from the specified commit to the latest version. - `backup_restore`: Backup then restore. @@ -155,23 +139,6 @@ If a test is not in the list, it will be ignored. It's similar to set the test a - `actions`: All actions available in actions.toml - `config_panel`: All configurations available in config_panel.toml -### `;;; Levels` -From [levels](https://yunohost.org/#/packaging_apps_levels_fr) 1 to 8, levels are determined automatically. -Except the level 5, you can't force a value for a level anymore. -The level 5 is determined by the results of [package linter](https://github.com/YunoHost/package_linter). -The default value for this level is `auto`, however, if needed, you can force the value for this level by setting it at `1`, for a positive result, or at `0`, for a negative one. -If you do so, please add a comment to justify why you force this level. - -### `;;; Options` -Supplementary options available in the check_process. -These options are facultative. - -- `Email` : Allow to specify an alternative email than the one in the manifest for notification by package check, when in a context of a continuous integration server. -- `Notification` : Level of notification for this package. There are 3 available levels. - - `down` : Send an email only if the level of the package has decreased. - - `change` : Send an email if the level of the package has changed. - - `all` : Send an email for each test on this package, whatever the result. - ### `;;; Upgrade options` *Optional instruction* For each specified commit for an upgrade, allow to give a name for this version and the manifest parameters which will be used for the preliminary installation. @@ -181,27 +148,8 @@ And if there's no manifest arguments, the default arguments of the check process --- The `package_check.sh` script accept 6 arguments in addition of the package to be checked. -- `--bash-mode`: The script will work without user intervention. - auto_remove value is ignored - `--branch=branch-name`: Check a branch of the repository instead of master. Allow to check a pull request. You can use an url with a branch, https://github.com/YunoHost-Apps/APP_ynh/tree/my_branch, to implicitly use this argument. -- `--build-lxc`: Install LXC and create the Debian YunoHost container if necessary. -- `--force-install-ok`: Force success of installations, even if they fail. Allow to perform following tests even if an installation fails. -- `--interrupt`: Force auto_remove value, break before each remove. +- `--interactive`: Wait for user input between each tests - `--help`: Display help. ---- -## LXC - -Package check uses the virtualization in containers to ensure the integrity of the testing environment. -Using LXC provides a better stability to the test process, a failed test doesn't impact the following tests and provides a testing environment without residues of previous tests. However, using LXC increases the length of tests, because of the manipulations of the container and reinstallations of dependencies. - -It uses also some space on the host, at least 6GB for the container, its snapshots and backup have to available. - -Using LXC is eased by 4 scripts, allowing to manage the creation, update, deletion and repair of the container. -- `lxc_build.sh`: lxc_build install LXC and its dependencies, then create a Debian container. - It add network support, install YunoHost and configure it. And then configure ssh. - The default ssh access is `ssh -t pchecker_lxc` -- `lxc_upgrade.sh`: Perform an upgrade of the container with apt-get and recreate the snapshot. -- `lxc_remove.sh`: Delete the LXC container, its snapshots and backup. Uninstall LXC and deconfigure the associated network. -- `lxc_check.sh`: Check the LXC container and try to fix it if necessary. diff --git a/sub_scripts/build_base_lxc.sh b/build_base_lxc.sh similarity index 95% rename from sub_scripts/build_base_lxc.sh rename to build_base_lxc.sh index 9c7ebcd..f488a45 100755 --- a/sub_scripts/build_base_lxc.sh +++ b/build_base_lxc.sh @@ -1,7 +1,7 @@ #!/bin/bash -cd $(dirname $(realpath $0) | sed 's@/sub_scripts$@@g') -source "./sub_scripts/common.sh" +cd $(dirname $(realpath $0)) +source "./lib/common.sh" function rebuild_base_lxc() { diff --git a/check_process b/check_process deleted file mode 100644 index a5631af..0000000 --- a/check_process +++ /dev/null @@ -1,47 +0,0 @@ -;; Test name -# Comment ignored - ; pre-install - echo -n "Here your commands to execute in the container" - echo ", before each installation of the app." - ; Manifest - domain="domain.tld" (DOMAIN) - path="/path" (PATH) - admin="john" (USER) - language="fr" - is_public=1 (PUBLIC|public=1|private=0) - password="password" - port="666" (PORT) - ; Actions - action_argument=arg1|arg2 - is_public=1|0 - ; Config_panel - main.categorie.config_example=arg1|arg2 - main.overwrite_files.overwrite_phpfpm=1|0 - main.php_fpm_config.footprint=low|medium|high|specific - main.php_fpm_config.free_footprint=20 - main.php_fpm_config.usage=low|medium|high - ; Checks - pkg_linter=1 - setup_sub_dir=1 - setup_root=1 - setup_nourl=0 - setup_private=1 - setup_public=1 - upgrade=1 - upgrade=1 from_commit=65c382d138596fcb32b4c97c39398815a1dcd4e8 - backup_restore=1 - multi_instance=1 - port_already_use=1 (66) - change_url=0 - actions=0 - config_panel=0 -;;; Levels - # If the level 5 (Package linter) is forced to 1. Please add justifications here. - Level 5=auto -;;; Options -Email= -Notification=none -;;; Upgrade options - ; commit=65c382d138596fcb32b4c97c39398815a1dcd4e8 - name=Name of this previous version - manifest_arg=domain=DOMAIN&path=PATH&admin=USER&password=pass&is_public=1& diff --git a/levels.list b/levels.list deleted file mode 100644 index 6f1a478..0000000 --- a/levels.list +++ /dev/null @@ -1,11 +0,0 @@ -0 Broken -1 Installable -2 Installable in all situations -3 Can be updated -4 Backup and restore support -5 Clean -6 Open to contributions from the community -7 Successfully pass all functional tests and linter tests -8 Maintained and long-term good quality -9 High quality app -10 Package assessed as perfect diff --git a/lib/analyze_test_results.py b/lib/analyze_test_results.py new file mode 100644 index 0000000..3465172 --- /dev/null +++ b/lib/analyze_test_results.py @@ -0,0 +1,284 @@ + +# Levels + +#0 Broken +#1 Installable +#2 Installable in all situations +#3 Can be updated +#4 Backup and restore support +#5 Clean +#6 Open to contributions from the community +#7 Successfully pass all functional tests and linter tests +#8 Maintained and long-term good quality +#9 High quality app +#10 Package assessed as perfect + +# Linter stuff: + + +# # Check we qualify for level 6, 7, 8 +# # Linter will have a warning called "app_in_github_org" if app ain't in the +# # yunohost-apps org... +# if ! cat "./temp_linter_result.json" | jq ".warning" | grep -q "app_in_github_org" +# then +# local pass_level_6="true" +# fi +# if cat "./temp_linter_result.json" | jq ".success" | grep -q "qualify_for_level_7" +# then +# local pass_level_7="true" +# fi +# if cat "./temp_linter_result.json" | jq ".success" | grep -q "qualify_for_level_8" +# then +# local pass_level_8="true" +# fi +# +# # If there are any critical errors, we'll force level 0 +# if [[ -n "$(cat "./temp_linter_result.json" | jq ".critical" | grep -v '\[\]')" ]] +# then +# local pass_level_0="false" +# # If there are any regular errors, we'll cap to 4 +# elif [[ -n "$(cat "./temp_linter_result.json" | jq ".error" | grep -v '\[\]')" ]] +# then +# local pass_level_4="false" +# # Otherwise, test pass (we'll display a warning depending on if there are +# # any remaning warnings or not) +# else +# if [[ -n "$(cat "./temp_linter_result.json" | jq ".warning" | grep -v '\[\]')" ]] +# then +# log_report_test_warning +# else +# log_report_test_success +# fi +# local pass_level_4="true" +# fi + + + + + + + + + + + + + +# +# local test_serie_id=$1 +# source $TEST_CONTEXT/$test_serie_id/results +# +# # Print the test result +# print_result () { +# +# # Print the result of this test +# # we use printf to force the length to 30 (filled with space) +# testname=$(printf %-30.30s "$1:") +# if [ $2 -eq 1 ] +# then +# echo "$testname ${BOLD}${GREEN}SUCCESS${NORMAL}" +# elif [ $2 -eq -1 ] +# then +# echo "$testname ${BOLD}${RED}FAIL${NORMAL}" +# else +# echo "$testname Not evaluated." +# fi +# } +# +# # Print the result for each test +# echo -e "\n\n" +# print_result "Package linter" $RESULT_linter +# print_result "Install (root)" $RESULT_check_root +# print_result "Install (subpath)" $RESULT_check_subdir +# print_result "Install (no url)" $RESULT_check_nourl +# print_result "Install (private)" $RESULT_check_private +# print_result "Install (multi-instance)" $RESULT_check_multi_instance +# print_result "Upgrade" $RESULT_check_upgrade +# print_result "Backup" $RESULT_check_backup +# print_result "Restore" $RESULT_check_restore +# print_result "Change URL" $RESULT_change_url +# print_result "Port already used" $RESULT_check_port +# print_result "Actions and config-panel" $RESULT_action_config_panel +# +# # Determine the level for this app +# +# # Each level can has 5 different values +# # 0 -> If this level can't be validated +# # 1 -> If this level is forced. Even if the tests fails +# # 2 -> Indicates the tests had previously validated this level +# # auto -> This level has not a value yet. +# # na -> This level will not be checked, but it'll be ignored in the final sum +# +# # Set default values for level, if they're empty. +# test -n "${level[1]}" || level[1]=auto +# test -n "${level[2]}" || level[2]=auto +# test -n "${level[3]}" || level[3]=auto +# test -n "${level[4]}" || level[4]=auto +# test -n "${level[5]}" || level[5]=auto +# test -n "${level[5]}" || level[5]=auto +# test -n "${level[6]}" || level[6]=auto +# test -n "${level[7]}" || level[7]=auto +# test -n "${level[8]}" || level[8]=auto +# test -n "${level[9]}" || level[9]=0 +# test -n "${level[10]}" || level[10]=0 +# +# pass_level_1() { +# # FIXME FIXME #FIXME +# return 0 +# } +# +# pass_level_2() { +# # -> The package can be install and remove in all tested configurations. +# # Validated if none install failed +# [ $RESULT_check_subdir -ne -1 ] && \ +# [ $RESULT_check_root -ne -1 ] && \ +# [ $RESULT_check_private -ne -1 ] && \ +# [ $RESULT_check_multi_instance -ne -1 ] +# } +# +# pass_level_3() { +# # -> The package can be upgraded from the same version. +# # Validated if the upgrade is ok. Or if the upgrade has been not tested but already validated before. +# [ $RESULT_check_upgrade -eq 1 ] || \ +# ( [ $RESULT_check_upgrade -ne -1 ] && \ +# [ "${level[3]}" == "2" ] ) +# } +# +# pass_level_4() { +# # -> The package can be backup and restore without error +# # Validated if backup and restore are ok. Or if backup and restore have been not tested but already validated before. +# ( [ $RESULT_check_backup -eq 1 ] && \ +# [ $RESULT_check_restore -eq 1 ] ) || \ +# ( [ $RESULT_check_backup -ne -1 ] && \ +# [ $RESULT_check_restore -ne -1 ] && \ +# [ "${level[4]}" == "2" ] ) +# } +# +# pass_level_5() { +# # -> The package have no error with package linter +# # -> The package does not have any alias_traversal error +# # Validated if Linter is ok. Or if Linter has been not tested but already validated before. +# [ $RESULT_alias_traversal -ne 1 ] && \ +# ([ $RESULT_linter -ge 1 ] || \ +# ( [ $RESULT_linter -eq 0 ] && \ +# [ "${level[5]}" == "2" ] ) ) +# } +# +# pass_level_6() { +# # -> The package can be backup and restore without error +# # This is from the linter, tests if app is the Yunohost-apps organization +# [ $RESULT_linter_level_6 -eq 1 ] || \ +# ([ $RESULT_linter_level_6 -eq 0 ] && \ +# [ "${level[6]}" == "2" ] ) +# } +# +# pass_level_7() { +# # -> None errors in all tests performed +# # Validated if none errors is happened. +# [ $RESULT_check_subdir -ne -1 ] && \ +# [ $RESULT_check_upgrade -ne -1 ] && \ +# [ $RESULT_check_private -ne -1 ] && \ +# [ $RESULT_check_multi_instance -ne -1 ] && \ +# [ $RESULT_check_port -ne -1 ] && \ +# [ $RESULT_check_backup -ne -1 ] && \ +# [ $RESULT_check_restore -ne -1 ] && \ +# [ $RESULT_change_url -ne -1 ] && \ +# [ $RESULT_action_config_panel -ne -1 ] && \ +# ([ $RESULT_linter_level_7 -ge 1 ] || +# ([ $RESULT_linter_level_7 -eq 0 ] && \ +# [ "${level[8]}" == "2" ] )) +# } +# +# pass_level_8() { +# # This happens in the linter +# # When writing this, defined as app being maintained + long term quality (= +# # certain amount of time level 5+ in the last year) +# [ $RESULT_linter_level_8 -ge 1 ] || \ +# ([ $RESULT_linter_level_8 -eq 0 ] && \ +# [ "${level[8]}" == "2" ] ) +# } +# +# pass_level_9() { +# list_url="https://raw.githubusercontent.com/YunoHost/apps/master/apps.json" +# curl --silent $list_url | jq ".[\"$app_id\"].high_quality" | grep -q "true" +# } +# +# # Check if the level can be changed +# level_can_change () { +# # If the level is set at auto, it's waiting for a change +# # And if it's set at 2, its value can be modified by a new result +# [ "${level[$1]}" == "auto" ] || [ "${level[$1]}" -eq 2 ] +# } +# +# if level_can_change 1; then pass_level_1 && level[1]=2 || level[1]=0; fi +# if level_can_change 2; then pass_level_2 && level[2]=2 || level[2]=0; fi +# if level_can_change 3; then pass_level_3 && level[3]=2 || level[3]=0; fi +# if level_can_change 4; then pass_level_4 && level[4]=2 || level[4]=0; fi +# if level_can_change 5; then pass_level_5 && level[5]=2 || level[5]=0; fi +# if level_can_change 6; then pass_level_6 && level[6]=2 || level[6]=0; fi +# if level_can_change 7; then pass_level_7 && level[7]=2 || level[7]=0; fi +# if level_can_change 8; then pass_level_8 && level[8]=2 || level[8]=0; fi +# if level_can_change 9; then pass_level_9 && level[9]=2 || level[9]=0; fi +# +# # Level 10 has no definition yet +# level[10]=0 +# +# # Initialize the global level +# global_level=0 +# +# # Calculate the final level +# for i in `seq 1 10` +# do +# +# # If there is a level still at 'auto', it's a mistake. +# if [ "${level[i]}" == "auto" ] +# then +# # So this level will set at 0. +# level[i]=0 +# +# # If the level is at 1 or 2. The global level will be set at this level +# elif [ "${level[i]}" -ge 1 ] +# then +# global_level=$i +# +# # But, if the level is at 0, the loop stop here +# # Like that, the global level rise while none level have failed +# else +# break +# fi +# done +# +# # If some witness files was missing, it's a big error ! So, the level fall immediately at 0. +# if [ $RESULT_witness -eq 1 ] +# then +# log_error "Some witness files has been deleted during those tests ! It's a very bad thing !" +# global_level=0 +# fi +# +# # If the package linter returned a critical error, the app is flagged as broken / level 0 +# if [ $RESULT_linter_broken -eq 1 ] +# then +# log_error "The package linter reported a critical failure ! App is considered broken !" +# global_level=0 +# fi +# +# if [ $RESULT_alias_traversal -eq 1 ] +# then +# log_error "Issue alias_traversal was detected ! Please see here https://github.com/YunoHost/example_ynh/pull/45 to fix that." +# fi +# +# # Then, print the levels +# # Print the global level +# verbose_level=$(grep "^$global_level " "./levels.list" | cut -c4-) +# +# log_info "Level of this application: $global_level ($verbose_level)" +# +# # And print the value for each level +# for i in `seq 1 10` +# do +# display="0" +# if [ "${level[$i]}" -ge 1 ]; then +# display="1" +# fi +# echo -e "\t Level $i: $display" +# done diff --git a/sub_scripts/common.sh b/lib/common.sh old mode 100755 new mode 100644 similarity index 98% rename from sub_scripts/common.sh rename to lib/common.sh index 2392ea5..297fdc2 --- a/sub_scripts/common.sh +++ b/lib/common.sh @@ -17,12 +17,22 @@ SUBDOMAIN="sub.$DOMAIN" TEST_USER="package_checker" LXC_BASE="ynh-appci-$DIST-$ARCH-base" -LXC_NAME="ynh-appci-$DIST" +LXC_NAME="ynh-appci-test" [[ -e "./config" ]] && source "./config" readonly lock_file="./pcheck.lock" +clean_exit () { + + LXC_RESET + + [ -n "$TEST_CONTEXT" ] rm -rf "$TEST_CONTEXT" + rm -f "$lock_file" + + exit $1 +} + #================================================= # LXC helpers #================================================= diff --git a/sub_scripts/lxc.sh b/lib/lxc.sh old mode 100755 new mode 100644 similarity index 86% rename from sub_scripts/lxc.sh rename to lib/lxc.sh index 7aa1e0a..2524245 --- a/sub_scripts/lxc.sh +++ b/lib/lxc.sh @@ -5,7 +5,8 @@ #================================================= LXC_CREATE () { - sudo lxc launch $LXC_NAME-base $LXC_NAME || exit 1 + sudo lxc image list $LXC_BASE | grep -q -w $LXC_BASE || log_critical "The base image $LXC_BASE doesn't exist yet. Consider using the build_base_lxc.sh to create it first" + sudo lxc launch $LXC_BASE $LXC_NAME || clean_exit 1 sudo lxc config set "$LXC_NAME" security.nesting true _LXC_START_AND_WAIT $LXC_NAME set_witness_files @@ -87,7 +88,7 @@ _LXC_START_AND_WAIT() { restart_container() { - sudo lxc stop "$1" + sudo lxc stop "$1" --timeout 15 &>/dev/null sudo lxc start "$1" } @@ -107,8 +108,7 @@ _LXC_START_AND_WAIT() { fi if [ "$j" == "10" ]; then - log_error 'Failed to start the container' - lxc info --show-log $1 + log_debug 'Failed to start the container ... restarting ...' failstart=1 restart_container "$1" @@ -124,7 +124,7 @@ _LXC_START_AND_WAIT() { fi if [ "$j" == "10" ]; then - log_error 'Failed to access the internet' + log_debug 'Failed to access the internet ... restarting' failstart=1 restart_container "$1" @@ -142,6 +142,8 @@ _LXC_START_AND_WAIT() { # Fail if the container failed to start if [ $i -eq $max_try ] && [ $failstart -eq 1 ] then + log_error "The container miserably failed to start or to connect to the internet" + lxc info --show-log $1 return 1 fi done diff --git a/sub_scripts/manifest_parsing.py b/lib/manifest_parsing.py old mode 100755 new mode 100644 similarity index 100% rename from sub_scripts/manifest_parsing.py rename to lib/manifest_parsing.py diff --git a/sub_scripts/testing_process.sh b/lib/tests.sh old mode 100755 new mode 100644 similarity index 77% rename from sub_scripts/testing_process.sh rename to lib/tests.sh index f17d31d..e80127c --- a/sub_scripts/testing_process.sh +++ b/lib/tests.sh @@ -1,113 +1,10 @@ #!/bin/bash -source sub_scripts/witness.sh - #================================================= -# Misc test helpers & coordination +# Logistic helpers #================================================= -RUN_ALL_TESTS() { - # Launch all tests successively - curl_error=0 - - # Be sure that the container is running - LXC_START "true" - - # Print the version of YunoHost from the LXC container - log_small_title "YunoHost versions" - LXC_START "sudo yunohost --version" - - # Init the value for the current test - current_test_number=1 - - # The list of test contains for example "TEST_UPGRADE some_commit_id - for testfile in $(ls $TEST_CONTEXT/tests/*.json); - do - TEST_LAUNCHER $testfile - done - -} - -TEST_LAUNCHER () { - local testfile="$1" - - # Start the timer for this test - start_timer - # And keep this value separately - local global_start_timer=$starttime - - current_test_id=$(basename $testfile | cut -d. -f1) - current_test_infos="$TEST_CONTEXT/tests/$current_test_id.json" - current_test_results="$TEST_CONTEXT/results/$current_test_id.json" - echo "{}" > $current_test_results - - local test_type=$(jq -r '.test_type' $testfile) - local test_arg=$(jq -r '.test_arg' $testfile) - - # Execute the test - $test_type $test_arg - - [ $? -eq 0 ] && SET_RESULT "success" main_result || SET_RESULT "failure" main_result - - break_before_continue - - # Restore the started time for the timer - starttime=$global_start_timer - # End the timer for the test - stop_timer 2 - - LXC_STOP - - # Update the lock file with the date of the last finished test. - # $$ is the PID of package_check itself. - echo "$1 $2:$(date +%s):$$" > "$lock_file" -} - -SET_RESULT() { - local result=$1 - local name=$2 - [ "$result" == "success" ] && log_report_test_success || log_report_test_failed - local current_results="$(cat $current_test_results)" - echo "$current_results" | jq --arg result $result ".$name=\$result" > $current_test_results -} - -#================================================= - -at_least_one_install_succeeded () { - - for TEST in $(ls $TEST_CONTEXT/tests/*.json) - do - local test_id=$(basename $TEST | cut -d. -f1) - jq -e '. | select(.test_type == "TEST_INSTALL")' $TEST >/dev/null \ - && jq -e '. | select(.main_result == "success")' $TEST_CONTEXT/results/$test_id.json >/dev/null \ - && return 0 - done - - log_error "All installs failed, therefore the following tests cannot be performed..." - return 1 -} - -break_before_continue () { - - if [ $interactive -eq 1 ] - then - echo "To enter a shell on the lxc:" - echo " sudo lxc exec $LXC_NAME bash" - read -p "Press a key to delete the application and continue...." < /dev/tty - fi -} - -start_test () { - - total_number_of_test=$(ls $TEST_CONTEXT/tests/*.json | wc -l) - - log_title "$1 [Test $current_test_number/$total_number_of_test]" - - # Increment the value of the current test - current_test_number=$((current_test_number+1)) -} - -RUN_YUNOHOST_CMD() { +_RUN_YUNOHOST_CMD() { log_debug "Running yunohost $1" @@ -124,38 +21,7 @@ RUN_YUNOHOST_CMD() { check_witness_files && return $returncode || return 2 } -this_is_a_web_app () { - - # Usually the fact that we test "nourl" - # installs should be a good indicator for the fact that it's not a webapp - for TEST in $(ls $TEST_CONTEXT/tests/*.json) - do - jq -e '. | select(.test_type == "TEST_INSTALL") | select(.test_arg == "nourl")' $TEST \ - && return 1 - done - - return 0 -} - -default_install_path() { - # All webapps should be installable at the root of a domain ? - this_is_a_web_app && echo "/" || echo "" -} - -path_to_install_type() { - local check_path="$1" - - [ -z "$check_path" ] && echo "nourl" \ - || [ "$check_path" == "/" ] && echo "root" \ - || echo "subdir" - -} - -#================================================= -# Install and remove an app -#================================================= - -INSTALL_APP () { +_INSTALL_APP () { local install_args="$(jq -r '.install_args' $current_test_infos)" local preinstall_template="$(jq -r '.preinstall_template' $current_test_infos)" @@ -171,7 +37,7 @@ INSTALL_APP () { # Exec the pre-install instruction, if there one if [ -n "$preinstall_template" ] then - log_small_title "Pre installation request" + log_small_title "Running pre-install steps" # Copy all the instructions into a script local preinstall_script="$TEST_CONTEXT/preinstall.sh" echo "$preinstall_template" > "$preinstall_script" @@ -187,14 +53,15 @@ INSTALL_APP () { fi # Install the application in a LXC container - RUN_YUNOHOST_CMD "app install --force /app_folder -a $install_args" + log_info "Running: yunohost app install --force /app_folder -a $install_args" + _RUN_YUNOHOST_CMD "app install --force /app_folder -a $install_args" local ret=$? [ $ret -eq 0 ] && log_debug "Installation successful." || log_error "Installation failed." return $ret } -LOAD_SNAPSHOT_OR_INSTALL_APP () { +_LOAD_SNAPSHOT_OR_INSTALL_APP () { local check_path="$1" local _install_type=$(path_to_install_type $check_path) @@ -203,7 +70,7 @@ LOAD_SNAPSHOT_OR_INSTALL_APP () { if [ ! -e "$LXC_SNAPSHOTS/$snapname" ] then LOAD_LXC_SNAPSHOT snap0 \ - && INSTALL_APP "path=$check_path" \ + &&_INSTALL_APP "path=$check_path" \ && log_debug "Creating a snapshot for $_install_type installation." \ && CREATE_LXC_SNAPSHOT $snapname else @@ -214,7 +81,7 @@ LOAD_SNAPSHOT_OR_INSTALL_APP () { } -REMOVE_APP () { +_REMOVE_APP () { # Remove an application break_before_continue @@ -222,18 +89,14 @@ REMOVE_APP () { log_small_title "Removing the app..." # Remove the application from the LXC container - RUN_YUNOHOST_CMD "app remove $app_id" + _RUN_YUNOHOST_CMD "app remove $app_id" local ret=$? [ "$ret" -eq 0 ] && log_debug "Remove successful." || log_error "Remove failed." return $ret } -#================================================= -# Try to access the app by its url -#================================================= - -VALIDATE_THAT_APP_CAN_BE_ACCESSED () { +_VALIDATE_THAT_APP_CAN_BE_ACCESSED () { local check_domain=$1 local check_path=$2 @@ -254,19 +117,19 @@ VALIDATE_THAT_APP_CAN_BE_ACCESSED () { then log_debug "Forcing public access using a skipped_uris setting" # Add a skipped_uris on / for the app - RUN_YUNOHOST_CMD "app setting $app_id_to_check skipped_uris -v \"/\"" + _RUN_YUNOHOST_CMD "app setting $app_id_to_check skipped_uris -v \"/\"" # Regen the config of sso - RUN_YUNOHOST_CMD "app ssowatconf" + _RUN_YUNOHOST_CMD "app ssowatconf" fi # Try to access to the url in 2 times, with a final / and without for i in $(seq 1 2) do - curl_check_path="${check_path:0:${#check_path}-1}" # First time we'll try without the trailing slash, # Second time *with* the trailing slash - [ $i -eq 1 ] || curl_check_path="$check_path/" + local curl_check_path="$(echo $check_path | sed 's@/$@@g')" + [ $i -eq 1 ] || curl_check_path="$curl_check_path/" # Remove the previous curl output rm -f "$curl_output" @@ -287,8 +150,10 @@ VALIDATE_THAT_APP_CAN_BE_ACCESSED () { # Call curl to try to access to the url of the app curl --location --insecure --silent --show-error \ --header "Host: $check_domain" \ - --resolve $check_domain:80:$LXC_IP \ - --resolve $check_domain:443:$LXC_IP \ + --resolve $DOMAIN:80:$LXC_IP \ + --resolve $DOMAIN:443:$LXC_IP \ + --resolve $SUBDOMAIN:80:$LXC_IP \ + --resolve $SUBDOMAIN:443:$LXC_IP \ --write-out "%{http_code};%{url_effective}\n" \ --output "$curl_output" \ $check_domain$curl_check_path \ @@ -380,62 +245,19 @@ VALIDATE_THAT_APP_CAN_BE_ACCESSED () { } - - #================================================= # The # Actual # Tests #================================================= - - - - PACKAGE_LINTER () { - # Package linter start_test "Package linter" # Execute package linter and linter_result gets the return code of the package linter ./package_linter/package_linter.py "$package_path" | tee -a "$complete_log" ./package_linter/package_linter.py "$package_path" --json | tee -a "$complete_log" > $current_test_results - -# # Check we qualify for level 6, 7, 8 -# # Linter will have a warning called "app_in_github_org" if app ain't in the -# # yunohost-apps org... -# if ! cat "./temp_linter_result.json" | jq ".warning" | grep -q "app_in_github_org" -# then -# local pass_level_6="true" -# fi -# if cat "./temp_linter_result.json" | jq ".success" | grep -q "qualify_for_level_7" -# then -# local pass_level_7="true" -# fi -# if cat "./temp_linter_result.json" | jq ".success" | grep -q "qualify_for_level_8" -# then -# local pass_level_8="true" -# fi -# -# # If there are any critical errors, we'll force level 0 -# if [[ -n "$(cat "./temp_linter_result.json" | jq ".critical" | grep -v '\[\]')" ]] -# then -# local pass_level_0="false" -# # If there are any regular errors, we'll cap to 4 -# elif [[ -n "$(cat "./temp_linter_result.json" | jq ".error" | grep -v '\[\]')" ]] -# then -# local pass_level_4="false" -# # Otherwise, test pass (we'll display a warning depending on if there are -# # any remaning warnings or not) -# else -# if [[ -n "$(cat "./temp_linter_result.json" | jq ".warning" | grep -v '\[\]')" ]] -# then -# log_report_test_warning -# else -# log_report_test_success -# fi -# local pass_level_4="true" -# fi } TEST_INSTALL () { @@ -454,8 +276,8 @@ TEST_INSTALL () { LOAD_LXC_SNAPSHOT snap0 # Install the application in a LXC container - INSTALL_APP "path=$check_path" "is_public=$is_public" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path $install_type + _INSTALL_APP "path=$check_path" "is_public=$is_public" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path $install_type local install=$? @@ -468,10 +290,10 @@ TEST_INSTALL () { && CREATE_LXC_SNAPSHOT $snapname # Remove and reinstall the application - REMOVE_APP \ + _REMOVE_APP \ && log_small_title "Reinstalling after removal." \ - && INSTALL_APP "path=$check_path" "is_public=$is_public" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path $install_type + &&_INSTALL_APP "path=$check_path" "is_public=$is_public" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path $install_type return $? } @@ -498,7 +320,7 @@ TEST_UPGRADE () { if [ "$commit" == "" ] then # If no commit is specified, use the current version. - LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" + _LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" local ret=$? else # Make a backup of the directory @@ -509,7 +331,7 @@ TEST_UPGRADE () { LOAD_LXC_SNAPSHOT snap0 # Install the application - INSTALL_APP "path=$check_path" + _INSTALL_APP "path=$check_path" local ret=$? # Then replace the backup @@ -518,13 +340,13 @@ TEST_UPGRADE () { fi # Check if the install worked - [ $ret -eq 0 ] || { log_error "Initial install failed... upgrade test ignore"; LXC_STOP; return 1; } + [ $ret -eq 0 ] || { log_error "Initial install failed... upgrade test ignore"; return 1; } log_small_title "Upgrade..." # Upgrade the application in a LXC container - RUN_YUNOHOST_CMD "app upgrade $app_id -f /app_folder" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path + _RUN_YUNOHOST_CMD "app upgrade $app_id -f /app_folder" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path return $? } @@ -541,11 +363,11 @@ TEST_MULTI_INSTANCE () { LOAD_LXC_SNAPSHOT snap0 log_small_title "First installation: path=$DOMAIN$check_path" \ - && INSTALL_APP "domain=$DOMAIN" "path=$check_path" \ + &&_INSTALL_APP "domain=$DOMAIN" "path=$check_path" \ && log_small_title "Second installation: path=$SUBDOMAIN$check_path" \ - && INSTALL_APP "path=$check_path" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $DOMAIN $check_path \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path "" ${app_id}__2 + &&_INSTALL_APP "path=$check_path" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $DOMAIN $check_path \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path "" ${app_id}__2 return $? } @@ -572,8 +394,8 @@ TEST_PORT_ALREADY_USED () { LXC_START "sudo systemctl enable netcat & sudo systemctl start netcat" # Install the application in a LXC container - INSTALL_APP "path=$check_path" "port=$check_port" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path + _INSTALL_APP "path=$check_path" "port=$check_port" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path return $? } @@ -590,7 +412,7 @@ TEST_BACKUP_RESTORE () { local check_path=$(default_install_path) # Install the application in a LXC container - LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" + _LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" local ret=$? @@ -609,7 +431,7 @@ TEST_BACKUP_RESTORE () { log_small_title "Backup of the application..." # Made a backup of the application - RUN_YUNOHOST_CMD "backup create -n Backup_test --apps $app_id" + _RUN_YUNOHOST_CMD "backup create -n Backup_test --apps $app_id" ret=$? fi @@ -627,7 +449,7 @@ TEST_BACKUP_RESTORE () { if [ $j -eq 0 ] then # Remove the application - REMOVE_APP + _REMOVE_APP log_small_title "Restore after removing the application..." @@ -635,7 +457,6 @@ TEST_BACKUP_RESTORE () { elif [ $j -eq 1 ] then - LXC_STOP LOAD_LXC_SNAPSHOT snap0 # Remove the previous residual backups @@ -648,15 +469,13 @@ TEST_BACKUP_RESTORE () { fi # Restore the application from the previous backup - RUN_YUNOHOST_CMD "backup restore Backup_test --force --apps $app_id" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path + _RUN_YUNOHOST_CMD "backup restore Backup_test --force --apps $app_id" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $SUBDOMAIN $check_path local ret=$? [ $ret -eq 0 ] || main_result=1 break_before_continue - - LXC_STOP done return $main_result @@ -670,73 +489,61 @@ TEST_CHANGE_URL () { # Check if an install have previously work at_least_one_install_succeeded || return 1 this_is_a_web_app || return 0 + + log_small_title "Preliminary install..." \ + && _LOAD_SNAPSHOT_OR_INSTALL_APP "/" + + local ret=$? + [ $ret -eq 0 ] || { return 1; } # Try in 6 times ! # Without modify the domain, root to path, path to path and path to root. # And then, same with a domain change - local main_result=0 local i=0 - for i in $(seq 1 7) + for i in $(seq 1 6) do # Same domain, root to path if [ $i -eq 1 ]; then - check_path=/ local new_path=/path local new_domain=$SUBDOMAIN # Same domain, path to path elif [ $i -eq 2 ]; then - check_path=/path local new_path=/path_2 local new_domain=$SUBDOMAIN # Same domain, path to root elif [ $i -eq 3 ]; then - check_path=/path local new_path=/ local new_domain=$SUBDOMAIN # Other domain, root to path elif [ $i -eq 4 ]; then - check_path=/ local new_path=/path local new_domain=$DOMAIN # Other domain, path to path elif [ $i -eq 5 ]; then - check_path=/path local new_path=/path_2 local new_domain=$DOMAIN # Other domain, path to root elif [ $i -eq 6 ]; then - check_path=/path - local new_path=/ - local new_domain=$DOMAIN - - # Other domain, root to root - elif [ $i -eq 7 ]; then - check_path=/ local new_path=/ local new_domain=$DOMAIN fi - # Install the application in a LXC container - log_small_title "Preliminary install..." \ - && LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" \ - && log_small_title "Change the url from $SUBDOMAIN$check_path to $new_domain$new_path..." \ - && RUN_YUNOHOST_CMD "app change-url $app_id -d '$new_domain' -p '$new_path'" \ - && VALIDATE_THAT_APP_CAN_BE_ACCESSED $new_domain $new_path + log_small_title "Changing the url to $new_domain$new_path..." \ + && _RUN_YUNOHOST_CMD "app change-url $app_id -d $new_domain -p $new_path" \ + && _VALIDATE_THAT_APP_CAN_BE_ACCESSED $new_domain $new_path local ret=$? - [ $ret -eq 0 ] || main_result=1 + [ $ret -eq 0 ] || { return 1; } break_before_continue - - LXC_STOP done - return $main_result + return 0 } @@ -809,7 +616,7 @@ ACTIONS_CONFIG_PANEL () { # Install the application in a LXC container log_small_title "Preliminary install..." local check_path=$(default_install_path) - LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" + _LOAD_SNAPSHOT_OR_INSTALL_APP "$check_path" local main_result=0 @@ -832,7 +639,7 @@ ACTIONS_CONFIG_PANEL () { log_info "> List the available actions..." # List the actions - RUN_YUNOHOST_CMD "app action list $app_id" + _RUN_YUNOHOST_CMD "app action list $app_id" local ret=$? [ $ret -eq 0 ] || main_result=1 @@ -843,7 +650,7 @@ ACTIONS_CONFIG_PANEL () { log_info "> Show the config panel..." # Show the config-panel - RUN_YUNOHOST_CMD "app config show-panel $app_id" + _RUN_YUNOHOST_CMD "app config show-panel $app_id" local ret=$? [ $ret -eq 0 ] || main_result=1 break_before_continue @@ -1010,12 +817,12 @@ ACTIONS_CONFIG_PANEL () { if [ "$test_type" == "config_panel" ] then # Aply a configuration - RUN_YUNOHOST_CMD "app config apply $app_id $action_config_action $action_config_argument_built" + _RUN_YUNOHOST_CMD "app config apply $app_id $action_config_action $action_config_argument_built" ret=$? elif [ "$test_type" == "actions" ] then # Execute an action - RUN_YUNOHOST_CMD "app action run $app_id $action_config_action $action_config_argument_built" + _RUN_YUNOHOST_CMD "app action run $app_id $action_config_action $action_config_argument_built" ret=$? fi [ $ret -eq 0 ] || main_result=1 @@ -1025,7 +832,6 @@ ACTIONS_CONFIG_PANEL () { fi done - LXC_STOP return $main_result } diff --git a/lib/tests_coordination.sh b/lib/tests_coordination.sh new file mode 100644 index 0000000..d6306fa --- /dev/null +++ b/lib/tests_coordination.sh @@ -0,0 +1,400 @@ +#!/bin/bash + +source lib/lxc.sh +source lib/tests.sh +source lib/witness.sh + +complete_log="./Complete.log" + +# Purge some log files +> "$complete_log" + +# Redirect fd 3 (=debug steam) to complete log +exec 3>>$complete_log + +#======================================================================= +# Parse the check_process and generate jsons that describe tests to run +#======================================================================= + +# Parse the check_process only if it's exist +check_process="$package_path/check_process" + +# Extract a section found between $1 and $2 from the file $3 +extract_check_process_section () { + local source_file="${3:-$check_process}" + local extract=0 + local line="" + while read line + do + # Extract the line + if [ $extract -eq 1 ] + then + # Check if the line is the second line to found + if echo $line | grep -q "$2"; then + # Break the loop to finish the extract process + break; + fi + # Copy the line in the partial check_process + echo "$line" + fi + + # Search for the first line + if echo $line | grep -q "$1"; then + # Activate the extract process + extract=1 + fi + done < "$source_file" +} + + +parse_check_process() { + + log_info "Parsing check_process file" + + # Remove all commented lines in the check_process + sed --in-place '/^#/d' "$check_process" + # Remove all spaces at the beginning of the lines + sed --in-place 's/^[ \t]*//g' "$check_process" + + # Extract the Upgrade infos + extract_check_process_section "^;;; Upgrade options" ";; " > $TEST_CONTEXT/check_process.upgrade_options + mkdir -p $TEST_CONTEXT/upgrades + local commit + for commit in $(cat $TEST_CONTEXT/check_process.upgrade_options | grep "^; commit=.*" | awk -F= '{print $2}') + do + cat $TEST_CONTEXT/check_process.upgrade_options | sed -n -e "/^;; $commit/,/^;;/ p" | grep -v "^;;" > $TEST_CONTEXT/upgrades/$commit + done + rm $TEST_CONTEXT/check_process.upgrade_options + + local test_serie_id="0" + + # Parse each tests serie + while read <&3 tests_serie + do + test_serie_id=$((test_serie_id+1)) + local test_id=$((test_serie_id * 100)) + local test_serie_rawconf=$TEST_CONTEXT/raw_test_serie_config + + # Extract the section of the current tests serie + extract_check_process_section "^$tests_serie" "^;;" > $test_serie_rawconf + # This is the arg list to be later fed to "yunohost app install" + # Looking like domain=foo.com&path=/bar&password=stuff + # "Standard" arguments like domain/path will later be overwritten + # during tests + local install_args=$( extract_check_process_section "^; Manifest" "^; " $test_serie_rawconf | awk '{print $1}' | tr -d '"' | tr '\n' '&') + local preinstall_template=$(extract_check_process_section "^; pre-install" "^; " $test_serie_rawconf) + local action_infos=$( extract_check_process_section "^; Actions" "^; " $test_serie_rawconf) + local configpanel_infos=$( extract_check_process_section "^; Config_panel" "^; " $test_serie_rawconf) + + # Add (empty) special args if they ain't provided in check_process + echo "$install_args" | tr '&' '\n' | grep -q "^domain=" ||install_args+="&domain=" + echo "$install_args" | tr '&' '\n' | grep -q "^path=" ||install_args+="&path=" + echo "$install_args" | tr '&' '\n' | grep -q "^admin=" ||install_args+="&admin=" + echo "$install_args" | tr '&' '\n' | grep -q "^is_public=" ||install_args+="&is_public=" + + extract_check_process_section "^; Checks" "^; " $test_serie_rawconf > $TEST_CONTEXT/check_process.tests_infos + + is_test_enabled () { + # Find the line for the given check option + local value=$(grep -m1 -o "^$1=." "$TEST_CONTEXT/check_process.tests_infos" | awk -F= '{print $2}') + # And return this value + [ "${value:0:1}" = "1" ] + } + + add_test() { + local test_type="$1" + local test_arg="$2" + test_id="$((test_id+1))" + local extra="{}" + local _install_args="$install_args" + + # Upgrades with a specific commit + if [[ "$test_type" == "TEST_UPGRADE" ]] && [[ -n "$test_arg" ]] + then + local specific_upgrade_install_args="$(grep "^manifest_arg=" "$TEST_CONTEXT/upgrades/$commit" | cut -d'=' -f2-)" + [[ -n "$specific_upgrade_install_args" ]] && _install_args="$specific_upgrade_install_args" + + local upgrade_name="$(grep "^name=" "$TEST_CONTEXT/upgrades/$commit" | cut -d'=' -f2)" + extra="$(jq -n --arg upgrade_name "$upgrade_name" '{ $upgrade_name }')" + elif [[ "$test_type" == "ACTIONS_CONFIG_PANEL" ]] && [[ "$test_arg" == "actions" ]] + then + extra="$(jq -n --arg actions "$action_infos" '{ $actions }')" + elif [[ "$test_type" == "ACTIONS_CONFIG_PANEL" ]] && [[ "$test_arg" == "actions" ]] + then + extra="$(jq -n --arg configpanel "$configpanel_infos" '{ $configpanel }')" + fi + + jq -n \ + --arg test_serie "$test_serie" \ + --arg test_type "$test_type" \ + --arg test_arg "$test_arg" \ + --arg preinstall_template "$preinstall_template" \ + --arg install_args "$_install_args" \ + --argjson extra "$extra" \ + '{ $test_serie, $test_type, $test_arg, $preinstall_template, $install_args, $extra }' \ + > "$TEST_CONTEXT/tests/$test_id.json" + } + + # For not-the-main-test-serie, we only consider testing the install and + # upgrade from previous commits + if [[ "$test_serie_id" != "1" ]] + then + is_test_enabled setup_root && add_test "TEST_INSTALL" "root" + is_test_enabled setup_sub_dir && add_test "TEST_INSTALL" "subdir" + is_test_enabled setup_nourl && add_test "TEST_INSTALL" "nourl" + grep "^upgrade=1" "$TEST_CONTEXT/check_process.tests_infos" | + while IFS= read -r LINE; + do + commit=$(echo $LINE | grep -o "from_commit=.*" | awk -F= '{print $2}') + [ -n "$commit" ] || continue + add_test "TEST_UPGRADE" "$commit" + done + + continue + else + test_serie="default" + fi + + is_test_enabled pkg_linter && add_test "PACKAGE_LINTER" + is_test_enabled setup_sub_dir && add_test "TEST_INSTALL" "subdir" + is_test_enabled setup_root && add_test "TEST_INSTALL" "root" + is_test_enabled setup_nourl && add_test "TEST_INSTALL" "nourl" + is_test_enabled setup_private && add_test "TEST_INSTALL" "private" + is_test_enabled multi_instance && add_test "TEST_MULTI_INSTANCE" + is_test_enabled backup_restore && add_test "TEST_BACKUP_RESTORE" + + # Upgrades + grep "^upgrade=1" "$TEST_CONTEXT/check_process.tests_infos" | + while IFS= read -r LINE; + do + commit=$(echo $LINE | grep -o "from_commit=.*" | awk -F= '{print $2}') + add_test "TEST_UPGRADE" "$commit" + done + + # "Advanced" features + + is_test_enabled change_url && add_test "TEST_CHANGE_URL" + is_test_enabled actions && add_test "ACTIONS_CONFIG_PANEL" "actions" + is_test_enabled config_panel && add_test "ACTIONS_CONFIG_PANEL" "config_panel" + + # Port already used ... do we really need this ... + + if grep -q -m1 "port_already_use=1" "$TEST_CONTEXT/check_process.tests_infos" + then + local check_port=$(grep -m1 "port_already_use=1" "$TEST_CONTEXT/check_process.tests_infos" | grep -o -E "\([0-9]+\)" | tr -d '()') + else + local check_port=6660 + fi + + is_test_enabled port_already_use && add_test "TEST_PORT_ALREADY_USED" "$check_port" + + done 3<<< "$(grep "^;; " "$check_process")" + + return 0 +} + +guess_test_configuration() { + + log_error "Not check_process file found." + log_warning "Package check will attempt to automatically guess what tests to run." + + local test_id=100 + + add_test() { + local test_type="$1" + local test_arg="$2" + test_id="$((test_id+1))" + local extra="{}" + + jq -n \ + --arg test_serie "default" \ + --arg test_type "$test_type" \ + --arg test_arg "$test_arg" \ + --arg preinstall_template "" \ + --arg install_args "$install_args" \ + --argjson extra "$extra" \ + '{ $test_serie, $test_type, $test_arg, $preinstall_template, $install_args, $extra }' \ + > "$TEST_CONTEXT/tests/$test_id.json" + } + + local install_args=$(python "./lib/manifest_parsing.py" "$package_path/manifest.json" | cut -d ':' -f1,2 | tr ':' '=' | tr '\n' '&') + + add_test "PACKAGE_LINTER" + add_test "TEST_INSTALL subdir" + add_test "TEST_INSTALL root" + if echo $install_args | grep -q "is_public=" + then + add_test "TEST_INSTALL" "private" + fi + if grep multi_instance "$package_path/manifest.json" | grep -q true + then + add_test "TEST_MULTI_INSTANCE" + fi + add_test "TEST_BACKUP_RESTORE" + add_test "TEST_UPGRADE" +} + +#================================================= +# Misc test helpers & coordination +#================================================= + +run_all_tests() { + + mkdir -p $TEST_CONTEXT/tests + mkdir -p $TEST_CONTEXT/results + + [ -e "$check_process" ] \ + && parse_check_process \ + || guess_test_configuration + + # Start the timer for this test + start_timer + # And keep this value separately + complete_start_timer=$starttime + + # Break after the first tests serie + if [ $interactive -eq 1 ]; then + read -p "Press a key to start the tests..." < /dev/tty + fi + + # Launch all tests successively + cat $TEST_CONTEXT/tests/*.json >&3 + + # Reset and create a fresh container to work with + check_lxd_setup + LXC_RESET + LXC_CREATE + # Be sure that the container is running + LXC_START "true" + + # Print the version of YunoHost from the LXC container + log_small_title "YunoHost versions" + LXC_START "sudo yunohost --version" + + # Init the value for the current test + current_test_number=1 + + # The list of test contains for example "TEST_UPGRADE some_commit_id + for testfile in $(ls $TEST_CONTEXT/tests/*.json); + do + TEST_LAUNCHER $testfile + done + + # Print the final results of the tests + # FIXME COMPUTE_RESULTS_SUMMARY $test_serie_id + + # Restore the started time for the timer + starttime=$complete_start_timer + # End the timer for the test + stop_timer 3 + + echo "You can find the complete log of these tests in $(realpath $complete_log)" + +} + +TEST_LAUNCHER () { + local testfile="$1" + + # Start the timer for this test + start_timer + # And keep this value separately + local global_start_timer=$starttime + + current_test_id=$(basename $testfile | cut -d. -f1) + current_test_infos="$TEST_CONTEXT/tests/$current_test_id.json" + current_test_results="$TEST_CONTEXT/results/$current_test_id.json" + echo "{}" > $current_test_results + + local test_type=$(jq -r '.test_type' $testfile) + local test_arg=$(jq -r '.test_arg' $testfile) + + # Execute the test + $test_type $test_arg + + [ $? -eq 0 ] && SET_RESULT "success" main_result || SET_RESULT "failure" main_result + + break_before_continue + + # Restore the started time for the timer + starttime=$global_start_timer + # End the timer for the test + stop_timer 2 + + LXC_STOP + + # Update the lock file with the date of the last finished test. + # $$ is the PID of package_check itself. + echo "$1 $2:$(date +%s):$$" > "$lock_file" +} + +SET_RESULT() { + local result=$1 + local name=$2 + [ "$result" == "success" ] && log_report_test_success || log_report_test_failed + local current_results="$(cat $current_test_results)" + echo "$current_results" | jq --arg result $result ".$name=\$result" > $current_test_results +} + +#================================================= + +at_least_one_install_succeeded () { + + for TEST in $(ls $TEST_CONTEXT/tests/*.json) + do + local test_id=$(basename $TEST | cut -d. -f1) + jq -e '. | select(.test_type == "TEST_INSTALL")' $TEST >/dev/null \ + && jq -e '. | select(.main_result == "success")' $TEST_CONTEXT/results/$test_id.json >/dev/null \ + && return 0 + done + + log_error "All installs failed, therefore the following tests cannot be performed..." + return 1 +} + +break_before_continue () { + + if [ $interactive -eq 1 ] + then + echo "To enter a shell on the lxc:" + echo " sudo lxc exec $LXC_NAME bash" + read -p "Press a key to delete the application and continue...." < /dev/tty + fi +} + +start_test () { + + total_number_of_test=$(ls $TEST_CONTEXT/tests/*.json | wc -l) + + log_title "$1 [Test $current_test_number/$total_number_of_test]" + + # Increment the value of the current test + current_test_number=$((current_test_number+1)) +} + +this_is_a_web_app () { + + # Usually the fact that we test "nourl" + # installs should be a good indicator for the fact that it's not a webapp + for TEST in $(ls $TEST_CONTEXT/tests/*.json) + do + jq -e '. | select(.test_type == "TEST_INSTALL") | select(.test_arg == "nourl")' $TEST \ + && return 1 + done + + return 0 +} + +default_install_path() { + # All webapps should be installable at the root of a domain ? + this_is_a_web_app && echo "/" || echo "" +} + +path_to_install_type() { + local check_path="$1" + + [ -z "$check_path" ] && echo "nourl" \ + || [ "$check_path" == "/" ] && echo "root" \ + || echo "subdir" + +} + diff --git a/sub_scripts/witness.sh b/lib/witness.sh similarity index 100% rename from sub_scripts/witness.sh rename to lib/witness.sh diff --git a/package_check.sh b/package_check.sh index 9d0ef7d..5d58b37 100755 --- a/package_check.sh +++ b/package_check.sh @@ -1,26 +1,8 @@ #!/bin/bash -cd $(dirname $(realpath $0) | sed 's@/sub_scripts$@@g') -source "./sub_scripts/common.sh" -source "./sub_scripts/lxc.sh" -source "./sub_scripts/testing_process.sh" - -complete_log="./Complete.log" - -# Purge some log files -> "$complete_log" -> "./lxc_boot.log" - -TEST_CONTEXT=$(mktemp -d /tmp/package_check.XXXXXX) - -# Redirect fd 3 (=debug steam) to complete log -exec 3>>$complete_log - -#================================================= -# Starting and checking -#================================================= -# Generic functions -#================================================= +cd $(dirname $(realpath $0)) +source "./lib/common.sh" +source "./lib/tests_coordination.sh" print_help() { cat << EOF @@ -38,16 +20,6 @@ exit 0 } -clean_exit () { - - LXC_RESET - - rm -rf "$TEST_CONTEXT" - rm -f "$lock_file" - - exit $1 -} - #================================================= # Pase CLI arguments #================================================= @@ -57,7 +29,6 @@ clean_exit () { [ "$#" -eq 0 ] && print_help gitbranch="" -force_install_ok=0 interactive=0 function parse_args() { @@ -155,21 +126,6 @@ fi # $$ is the PID of package_check itself. echo "start:$(date +%s):$$" > "$lock_file" -#================================================= -# Various logistic checks and upgrades... -#================================================= - -assert_we_are_connected_to_the_internets - -#self_upgrade # FIXME renenable this later -fetch_or_upgrade_package_linter - -# Reset and create a fresh container to work with -check_lxd_setup -lxc image list $LXC_BASE | grep -q -w $LXC_BASE || log_critical "The base image $LXC_BASE doesn't exist yet. Consider using the build_base_lxc.sh to create it first" -LXC_RESET -LXC_CREATE - #================================================= # Pick up the package #================================================= @@ -223,488 +179,19 @@ FETCH_PACKAGE_TO_TEST() { fi } +################################### +# Main code +################################### + +assert_we_are_connected_to_the_internets +#self_upgrade # FIXME renenable this later +fetch_or_upgrade_package_linter + +TEST_CONTEXT=$(mktemp -d /tmp/package_check.XXXXXX) + FETCH_PACKAGE_TO_TEST $path_to_package_to_test readonly app_id="$(cat $package_path/manifest.json | jq -r .id)" - -#================================================= -# Determine and print the results -#================================================= - -COMPUTE_RESULTS_SUMMARY () { - - return -# -# local test_serie_id=$1 -# source $TEST_CONTEXT/$test_serie_id/results -# -# # Print the test result -# print_result () { -# -# # Print the result of this test -# # we use printf to force the length to 30 (filled with space) -# testname=$(printf %-30.30s "$1:") -# if [ $2 -eq 1 ] -# then -# echo "$testname ${BOLD}${GREEN}SUCCESS${NORMAL}" -# elif [ $2 -eq -1 ] -# then -# echo "$testname ${BOLD}${RED}FAIL${NORMAL}" -# else -# echo "$testname Not evaluated." -# fi -# } -# -# # Print the result for each test -# echo -e "\n\n" -# print_result "Package linter" $RESULT_linter -# print_result "Install (root)" $RESULT_check_root -# print_result "Install (subpath)" $RESULT_check_subdir -# print_result "Install (no url)" $RESULT_check_nourl -# print_result "Install (private)" $RESULT_check_private -# print_result "Install (multi-instance)" $RESULT_check_multi_instance -# print_result "Upgrade" $RESULT_check_upgrade -# print_result "Backup" $RESULT_check_backup -# print_result "Restore" $RESULT_check_restore -# print_result "Change URL" $RESULT_change_url -# print_result "Port already used" $RESULT_check_port -# print_result "Actions and config-panel" $RESULT_action_config_panel -# -# # Determine the level for this app -# -# # Each level can has 5 different values -# # 0 -> If this level can't be validated -# # 1 -> If this level is forced. Even if the tests fails -# # 2 -> Indicates the tests had previously validated this level -# # auto -> This level has not a value yet. -# # na -> This level will not be checked, but it'll be ignored in the final sum -# -# # Set default values for level, if they're empty. -# test -n "${level[1]}" || level[1]=auto -# test -n "${level[2]}" || level[2]=auto -# test -n "${level[3]}" || level[3]=auto -# test -n "${level[4]}" || level[4]=auto -# test -n "${level[5]}" || level[5]=auto -# test -n "${level[5]}" || level[5]=auto -# test -n "${level[6]}" || level[6]=auto -# test -n "${level[7]}" || level[7]=auto -# test -n "${level[8]}" || level[8]=auto -# test -n "${level[9]}" || level[9]=0 -# test -n "${level[10]}" || level[10]=0 -# -# pass_level_1() { -# # FIXME FIXME #FIXME -# return 0 -# } -# -# pass_level_2() { -# # -> The package can be install and remove in all tested configurations. -# # Validated if none install failed -# [ $RESULT_check_subdir -ne -1 ] && \ -# [ $RESULT_check_root -ne -1 ] && \ -# [ $RESULT_check_private -ne -1 ] && \ -# [ $RESULT_check_multi_instance -ne -1 ] -# } -# -# pass_level_3() { -# # -> The package can be upgraded from the same version. -# # Validated if the upgrade is ok. Or if the upgrade has been not tested but already validated before. -# [ $RESULT_check_upgrade -eq 1 ] || \ -# ( [ $RESULT_check_upgrade -ne -1 ] && \ -# [ "${level[3]}" == "2" ] ) -# } -# -# pass_level_4() { -# # -> The package can be backup and restore without error -# # Validated if backup and restore are ok. Or if backup and restore have been not tested but already validated before. -# ( [ $RESULT_check_backup -eq 1 ] && \ -# [ $RESULT_check_restore -eq 1 ] ) || \ -# ( [ $RESULT_check_backup -ne -1 ] && \ -# [ $RESULT_check_restore -ne -1 ] && \ -# [ "${level[4]}" == "2" ] ) -# } -# -# pass_level_5() { -# # -> The package have no error with package linter -# # -> The package does not have any alias_traversal error -# # Validated if Linter is ok. Or if Linter has been not tested but already validated before. -# [ $RESULT_alias_traversal -ne 1 ] && \ -# ([ $RESULT_linter -ge 1 ] || \ -# ( [ $RESULT_linter -eq 0 ] && \ -# [ "${level[5]}" == "2" ] ) ) -# } -# -# pass_level_6() { -# # -> The package can be backup and restore without error -# # This is from the linter, tests if app is the Yunohost-apps organization -# [ $RESULT_linter_level_6 -eq 1 ] || \ -# ([ $RESULT_linter_level_6 -eq 0 ] && \ -# [ "${level[6]}" == "2" ] ) -# } -# -# pass_level_7() { -# # -> None errors in all tests performed -# # Validated if none errors is happened. -# [ $RESULT_check_subdir -ne -1 ] && \ -# [ $RESULT_check_upgrade -ne -1 ] && \ -# [ $RESULT_check_private -ne -1 ] && \ -# [ $RESULT_check_multi_instance -ne -1 ] && \ -# [ $RESULT_check_port -ne -1 ] && \ -# [ $RESULT_check_backup -ne -1 ] && \ -# [ $RESULT_check_restore -ne -1 ] && \ -# [ $RESULT_change_url -ne -1 ] && \ -# [ $RESULT_action_config_panel -ne -1 ] && \ -# ([ $RESULT_linter_level_7 -ge 1 ] || -# ([ $RESULT_linter_level_7 -eq 0 ] && \ -# [ "${level[8]}" == "2" ] )) -# } -# -# pass_level_8() { -# # This happens in the linter -# # When writing this, defined as app being maintained + long term quality (= -# # certain amount of time level 5+ in the last year) -# [ $RESULT_linter_level_8 -ge 1 ] || \ -# ([ $RESULT_linter_level_8 -eq 0 ] && \ -# [ "${level[8]}" == "2" ] ) -# } -# -# pass_level_9() { -# list_url="https://raw.githubusercontent.com/YunoHost/apps/master/apps.json" -# curl --silent $list_url | jq ".[\"$app_id\"].high_quality" | grep -q "true" -# } -# -# # Check if the level can be changed -# level_can_change () { -# # If the level is set at auto, it's waiting for a change -# # And if it's set at 2, its value can be modified by a new result -# [ "${level[$1]}" == "auto" ] || [ "${level[$1]}" -eq 2 ] -# } -# -# if level_can_change 1; then pass_level_1 && level[1]=2 || level[1]=0; fi -# if level_can_change 2; then pass_level_2 && level[2]=2 || level[2]=0; fi -# if level_can_change 3; then pass_level_3 && level[3]=2 || level[3]=0; fi -# if level_can_change 4; then pass_level_4 && level[4]=2 || level[4]=0; fi -# if level_can_change 5; then pass_level_5 && level[5]=2 || level[5]=0; fi -# if level_can_change 6; then pass_level_6 && level[6]=2 || level[6]=0; fi -# if level_can_change 7; then pass_level_7 && level[7]=2 || level[7]=0; fi -# if level_can_change 8; then pass_level_8 && level[8]=2 || level[8]=0; fi -# if level_can_change 9; then pass_level_9 && level[9]=2 || level[9]=0; fi -# -# # Level 10 has no definition yet -# level[10]=0 -# -# # Initialize the global level -# global_level=0 -# -# # Calculate the final level -# for i in `seq 1 10` -# do -# -# # If there is a level still at 'auto', it's a mistake. -# if [ "${level[i]}" == "auto" ] -# then -# # So this level will set at 0. -# level[i]=0 -# -# # If the level is at 1 or 2. The global level will be set at this level -# elif [ "${level[i]}" -ge 1 ] -# then -# global_level=$i -# -# # But, if the level is at 0, the loop stop here -# # Like that, the global level rise while none level have failed -# else -# break -# fi -# done -# -# # If some witness files was missing, it's a big error ! So, the level fall immediately at 0. -# if [ $RESULT_witness -eq 1 ] -# then -# log_error "Some witness files has been deleted during those tests ! It's a very bad thing !" -# global_level=0 -# fi -# -# # If the package linter returned a critical error, the app is flagged as broken / level 0 -# if [ $RESULT_linter_broken -eq 1 ] -# then -# log_error "The package linter reported a critical failure ! App is considered broken !" -# global_level=0 -# fi -# -# if [ $RESULT_alias_traversal -eq 1 ] -# then -# log_error "Issue alias_traversal was detected ! Please see here https://github.com/YunoHost/example_ynh/pull/45 to fix that." -# fi -# -# # Then, print the levels -# # Print the global level -# verbose_level=$(grep "^$global_level " "./levels.list" | cut -c4-) -# -# log_info "Level of this application: $global_level ($verbose_level)" -# -# # And print the value for each level -# for i in `seq 1 10` -# do -# display="0" -# if [ "${level[$i]}" -ge 1 ]; then -# display="1" -# fi -# echo -e "\t Level $i: $display" -# done -} - -#================================================= -# Parse the check_process -#================================================= - -# Parse the check_process only if it's exist -check_process="$package_path/check_process" - -# Extract a section found between $1 and $2 from the file $3 -extract_check_process_section () { - local source_file="${3:-$check_process}" - local extract=0 - local line="" - while read line - do - # Extract the line - if [ $extract -eq 1 ] - then - # Check if the line is the second line to found - if echo $line | grep -q "$2"; then - # Break the loop to finish the extract process - break; - fi - # Copy the line in the partial check_process - echo "$line" - fi - - # Search for the first line - if echo $line | grep -q "$1"; then - # Activate the extract process - extract=1 - fi - done < "$source_file" -} - - -parse_check_process() { - - log_info "Parsing check_process file" - - # Remove all commented lines in the check_process - sed --in-place '/^#/d' "$check_process" - # Remove all spaces at the beginning of the lines - sed --in-place 's/^[ \t]*//g' "$check_process" - - # Extract the Upgrade infos - extract_check_process_section "^;;; Upgrade options" ";; " > $TEST_CONTEXT/check_process.upgrade_options - mkdir -p $TEST_CONTEXT/upgrades - local commit - for commit in $(cat $TEST_CONTEXT/check_process.upgrade_options | grep "^; commit=.*" | awk -F= '{print $2}') - do - cat $TEST_CONTEXT/check_process.upgrade_options | sed -n -e "/^;; $commit/,/^;;/ p" | grep -v "^;;" > $TEST_CONTEXT/upgrades/$commit - done - rm $TEST_CONTEXT/check_process.upgrade_options - - local test_serie_id="0" - - # Parse each tests serie - while read <&3 tests_serie - do - test_serie_id=$((test_serie_id+1)) - local test_id=$((test_serie_id * 100)) - local test_serie_rawconf=$TEST_CONTEXT/raw_test_serie_config - - # Extract the section of the current tests serie - extract_check_process_section "^$tests_serie" "^;;" > $test_serie_rawconf - # This is the arg list to be later fed to "yunohost app install" - # Looking like domain=foo.com&path=/bar&password=stuff - # "Standard" arguments like domain/path will later be overwritten - # during tests - local install_args=$( extract_check_process_section "^; Manifest" "^; " $test_serie_rawconf | awk '{print $1}' | tr -d '"' | tr '\n' '&') - local preinstall_template=$(extract_check_process_section "^; pre-install" "^; " $test_serie_rawconf) - local action_infos=$( extract_check_process_section "^; Actions" "^; " $test_serie_rawconf) - local configpanel_infos=$( extract_check_process_section "^; Config_panel" "^; " $test_serie_rawconf) - - extract_check_process_section "^; Checks" "^; " $test_serie_rawconf > $TEST_CONTEXT/check_process.tests_infos - - is_test_enabled () { - # Find the line for the given check option - local value=$(grep -m1 -o "^$1=." "$TEST_CONTEXT/check_process.tests_infos" | awk -F= '{print $2}') - # And return this value - [ "${value:0:1}" = "1" ] - } - - add_test() { - local test_type="$1" - local test_arg="$2" - test_id="$((test_id+1))" - local extra="{}" - local _install_args="$install_args" - - # Upgrades with a specific commit - if [[ "$test_type" == "TEST_UPGRADE" ]] && [[ -n "$test_arg" ]] - then - local specific_upgrade_install_args="$(grep "^manifest_arg=" "$TEST_CONTEXT/upgrades/$commit" | cut -d'=' -f2-)" - [[ -n "$specific_upgrade_install_args" ]] && _install_args="$specific_upgrade_install_args" - - local upgrade_name="$(grep "^name=" "$TEST_CONTEXT/upgrades/$commit" | cut -d'=' -f2)" - extra="$(jq -n --arg upgrade_name "$upgrade_name" '{ $upgrade_name }')" - elif [[ "$test_type" == "ACTIONS_CONFIG_PANEL" ]] && [[ "$test_arg" == "actions" ]] - then - extra="$(jq -n --arg actions "$action_infos" '{ $actions }')" - elif [[ "$test_type" == "ACTIONS_CONFIG_PANEL" ]] && [[ "$test_arg" == "actions" ]] - then - extra="$(jq -n --arg configpanel "$configpanel_infos" '{ $configpanel }')" - fi - - jq -n \ - --arg test_serie "$test_serie" \ - --arg test_type "$test_type" \ - --arg test_arg "$test_arg" \ - --arg preinstall_template "$preinstall_template" \ - --arg install_args "$_install_args" \ - --argjson extra "$extra" \ - '{ $test_serie, $test_type, $test_arg, $preinstall_template, $install_args, $extra }' \ - > "$TEST_CONTEXT/tests/$test_id.json" - } - - # For not-the-main-test-serie, we only consider testing the install and - # upgrade from previous commits - if [[ "$test_serie_id" != "1" ]] - then - is_test_enabled setup_sub_dir && add_test "TEST_INSTALL" "subdir" - is_test_enabled setup_root && add_test "TEST_INSTALL" "root" - is_test_enabled setup_nourl && add_test "TEST_INSTALL" "nourl" - grep "^upgrade=1" "$TEST_CONTEXT/check_process.tests_infos" | - while IFS= read -r LINE; - do - commit=$(echo $LINE | grep -o "from_commit=.*" | awk -F= '{print $2}') - [ -n "$commit" ] || continue - add_test "TEST_UPGRADE" "$commit" - done - - continue - else - test_serie="default" - fi - - is_test_enabled pkg_linter && add_test "PACKAGE_LINTER" - is_test_enabled setup_sub_dir && add_test "TEST_INSTALL" "subdir" - is_test_enabled setup_root && add_test "TEST_INSTALL" "root" - is_test_enabled setup_nourl && add_test "TEST_INSTALL" "nourl" - is_test_enabled setup_private && add_test "TEST_INSTALL" "private" - is_test_enabled multi_instance && add_test "TEST_MULTI_INSTANCE" - is_test_enabled backup_restore && add_test "TEST_BACKUP_RESTORE" - - # Upgrades - grep "^upgrade=1" "$TEST_CONTEXT/check_process.tests_infos" | - while IFS= read -r LINE; - do - commit=$(echo $LINE | grep -o "from_commit=.*" | awk -F= '{print $2}') - add_test "TEST_UPGRADE" "$commit" - done - - # "Advanced" features - - is_test_enabled change_url && add_test "TEST_CHANGE_URL" - is_test_enabled actions && add_test "ACTIONS_CONFIG_PANEL" "actions" - is_test_enabled config_panel && add_test "ACTIONS_CONFIG_PANEL" "config_panel" - - # Port already used ... do we really need this ... - - if grep -q -m1 "port_already_use=1" "$TEST_CONTEXT/check_process.tests_infos" - then - local check_port=$(grep -m1 "port_already_use=1" "$TEST_CONTEXT/check_process.tests_infos" | grep -o -E "\([0-9]+\)" | tr -d '()') - else - local check_port=6660 - fi - - is_test_enabled port_already_use && add_test "TEST_PORT_ALREADY_USED" "$check_port" - - done 3<<< "$(grep "^;; " "$check_process")" - - return 0 -} - -guess_test_configuration() { - - log_error "Not check_process file found." - log_warning "Package check will attempt to automatically guess what tests to run." - - add_test() { - local test_type="$1" - local test_arg="$2" - test_id="$((test_id+1))" - local extra="{}" - - jq -n \ - --arg test_serie "default" \ - --arg test_type "$test_type" \ - --arg test_arg "$test_arg" \ - --arg preinstall_template "" \ - --arg install_args "$install_args" \ - --argjson extra "$extra" \ - '{ $test_serie, $test_type, $test_arg, $preinstall_template, $install_args, $extra }' \ - > "$TEST_CONTEXT/tests/$test_id.json" - } - - local install_args=$(python "./sub_scripts/manifest_parsing.py" "$package_path/manifest.json" | cut -d ':' -f1,2 | tr ':' '=' | tr '\n' '&') - - add_test "PACKAGE_LINTER" - add_test "TEST_INSTALL subdir" - add_test "TEST_INSTALL root" - if echo $install_args | grep -q "is_public=" - then - add_test "TEST_INSTALL" "private" - fi - if grep multi_instance "$package_path/manifest.json" | grep -q true - then - add_test "TEST_MULTI_INSTANCE" - fi - add_test "TEST_BACKUP_RESTORE" - add_test "TEST_UPGRADE" -} - -#================================================= - -run_all_tests() { - - # Start the timer for this test - start_timer - # And keep this value separately - complete_start_timer=$starttime - - # Break after the first tests serie - if [ $interactive -eq 1 ]; then - read -p "Press a key to start the tests..." < /dev/tty - fi - - # Launch all tests successively - cat $TEST_CONTEXT/tests/*.json >&3 - RUN_ALL_TESTS $TEST_CONTEXT/tests/ - - # Print the final results of the tests - COMPUTE_RESULTS_SUMMARY $test_serie_id - - # Restore the started time for the timer - starttime=$complete_start_timer - # End the timer for the test - stop_timer 3 - - echo "You can find the complete log of these tests in $(realpath $complete_log)" - -} - -mkdir -p $TEST_CONTEXT/tests -mkdir -p $TEST_CONTEXT/results - -[ -e "$check_process" ] \ - && parse_check_process \ - || guess_test_configuration - run_all_tests clean_exit 0