Merge pull request #1997 from Salamandar/refactor

Python cleanup

README.md

@@ -2,23 +2,23 @@
 
 <img src="https://avatars.githubusercontent.com/u/1519495?s=200&v=4" width=80><img src="https://yunohost.org/user/images/yunohost_package.png" width=80>
 
-Here you will find the repositories and versions of every apps available in YunoHost's default catalog.
+This repository contains the default YunoHost app catalog, as well as tools
+that can be run manually or automatically.
 
-It is browsable here: https://yunohost.org/apps
+The catalog is stored in [**`apps.toml`**](./apps.toml) and is browsable here:
+<https://yunohost.org/apps>
 
-The main file of the catalog is [**apps.toml**](./apps.toml) which contains
-references to the corresponding Git repositories for each application, along
-with a few metadata about them such as its category or maintenance state. This
-file regularly read by `list_builder.py` which publish the results on
-https://app.yunohost.org/default/.
+It contains refences to the apps' repositories, along with a few metadata about
+them such as its category or maintenance state. This file is regularly read by
+`tools/list_builder.py` which publish the results on <https://app.yunohost.org/default>.
 
-### Where can I learn about app packaging in YunoHost?
+## Where can I learn about app packaging in YunoHost?
 
-- You can browse the contributor documentation : https://yunohost.org/contributordoc
+- You can browse [the contributor documentation](https://yunohost.org/contributordoc)
 - If you are not familiar with Git/GitHub, you can have a look at our [homemade guide](https://yunohost.org/#/packaging_apps_git)
 - Don't hesitate to reach for help on the dedicated [application packaging chatroom](https://yunohost.org/chat_rooms) ... we can even schedule an audio meeting to help you get started!
 
-### How to add your app to the application catalog
+## How to add your app to the application catalog
 
 > **Note**
 > The YunoHost project will **NOT** integrate in its catalog applications that are not
@@ -30,16 +30,20 @@ https://app.yunohost.org/default/.
 > with keeping your app working and up to date with packaging evolutions on the long run.
 
 To add your application to the catalog:
-* Fork this repository and edit the [apps.toml](https://github.com/YunoHost/apps/tree/master/apps.toml) file
-* Add your app's ID and git information at the right alphabetical place
-* Indicate the app's functioning state: `notworking`, `inprogress`, or `working`
-* Indicate the app category, which you can pick from `categories.toml`
-* Indicate any anti-feature that your app may be subject to, see `antifeatures.toml` (or remove the `antifeatures` key if there's none)
-* Indicate if your app can be thought of as an alternative to popular proprietary services (or remove the `potential_alternative_to` key if there's none)
-* *Do not* add the `level` entry by yourself. Our automatic test suite ("the CI") will handle it.
+
+* Fork [this repository](https://github.com/YunoHost/apps)
+* Edit the [`apps.toml`](/apps.toml) file
+  * Add your app's ID and git information at the right alphabetical place
+  * Indicate the app's functioning state: `notworking`, `inprogress`, or `working`
+  * Indicate the app category, which you can pick from `categories.toml`
+  * Indicate any anti-feature that your app may be subject to, see `antifeatures.toml` (or remove the `antifeatures` key if there's none)
+  * Indicate if your app can be thought of as an alternative to popular proprietary services (or remove the `potential_alternative_to` key if there's none)
+  * *Do not* add the `level` entry by yourself. Our automatic test suite ("the CI") will handle it.
+* Commit and push your modifications to your repository
 * Create a [Pull Request](https://github.com/YunoHost/apps/pulls/)
 
 App example addition:
+
 ```toml
 [your_app]
 antifeatures = [ "deprecated-software" ]   # Remove if no relevant antifeature applies
@@ -58,17 +62,23 @@ url = "https://github.com/YunoHost-Apps/your_app_ynh"
 > obtain an access to the developer CI where you'll be able to test your app
 > easily.
 
-### Updating apps levels in the catalog
+## Updating apps levels in the catalog
 
-App packagers should *not* manually set their apps' level. The levels of all the apps are automatically updated once per week on Friday, according to the results from the official app CI.
+App packagers should *not* manually set their apps' level. The levels of all
+the apps are automatically updated once per week on Friday, according to the
+results from the official app CI.
 
-### Apps flagged as not-maintained
+## Apps flagged as not-maintained
 
-Applications with no recent activity and no active sign from maintainer may be flagged in `apps.toml` with the `package-not-maintained` antifeature tag to signify that the app is inactive and may slowly become outdated with respect to the upstream, or with respect to good packaging practices. It does **not** mean that the app is not working anymore.
+Applications with no recent activity and no active sign from maintainer may be
+flagged in `apps.toml` with the `package-not-maintained` antifeature tag to
+signify that the app is inactive and may slowly become outdated with respect to
+the upstream, or with respect to good packaging practices. It does **not** mean
+that the app is not working anymore.
 
-Feel free to contact the app group if you feel like taking over the maintenance of a currently unmaintained app!
+Feel free to contact the app group if you feel like taking over the maintenance
+of a currently unmaintained app!
 
-### `graveyard.toml`
-
-This file is for apps that are long-term not-working and unlikely to be ever revived
+## `graveyard.toml`
 
+This file is for apps that are long-term not-working and unlikely to be ever revived.

rebuild.sh

@@ -7,4 +7,4 @@ cd $workdir
 date >> $log
 git pull &>/dev/null
 
-./list_builder.py &>> $log || sendxmpppy "[listbuilder] Rebuilding the application list failed miserably"
+./tools/list_builder.py &>> $log || sendxmpppy "[listbuilder] Rebuilding the application list failed miserably"

store/README.md

@@ -4,7 +4,7 @@ This is a Flask app interfacing with YunoHost's app catalog for a cool browsing
 
 ## Developement
 
-```
+```bash
 python3 -m venv venv
 source venv/bin/activate
 pip3 install -r requirements.txt
@@ -19,22 +19,22 @@ curl https://app.yunohost.org/default/v3/apps.json > ../builds/default/v3/apps.j
 
 # You will also want to run list_builder.py to initialize the .apps_cache (at least for a few apps, you can Ctrl+C after a while)
 pushd ..
-    python3 list_builder.py
+    ./tools/list_builder.py
 popd
 ```
 
 And then start the dev server:
 
-```
+```bash
 source venv/bin/activate
 FLASK_APP=app.py FLASK_ENV=development flask run
 ```
 
 ## Translation
 
-It's based on Flask-Babel : https://python-babel.github.io/
+It's based on Flask-Babel : <https://python-babel.github.io>
 
-```
+```bash
 source venv/bin/activate
 pybabel extract --ignore-dirs venv -F babel.cfg -o messages.pot .
 

tools/README-generator/__init__.py

@@ -0,0 +1 @@
+#!/usr/bin/env python3

tools/README-generator/make_readme.py

@@ -2,12 +2,13 @@
 
 import argparse
 import json
-import toml
 import os
 from pathlib import Path
 
+import toml
 from jinja2 import Environment, FileSystemLoader
 
+
 def value_for_lang(values, lang):
      if not isinstance(values, dict):
          return values

tools/README-generator/webhook.py

@@ -1,15 +1,16 @@
-import os
-import hmac
-import shlex
-import hashlib
+#!/usr/bin/env python3
+
 import asyncio
+import hashlib
+import hmac
+import os
+import shlex
 import tempfile
 
+from make_readme import generate_READMEs
 from sanic import Sanic, response
 from sanic.response import text
 
-from make_readme import generate_READMEs
-
 app = Sanic(__name__)
 
 github_webhook_secret = open("github_webhook_secret", "r").read().strip()

tools/__init__.py

@@ -0,0 +1 @@
+#!/usr/bin/env python3

tools/app_caches.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+
+import argparse
+import shutil
+import logging
+from multiprocessing import Pool
+from pathlib import Path
+from typing import Any
+
+import tqdm
+
+from appslib.utils import (REPO_APPS_ROOT,  # pylint: disable=import-error
+                           get_catalog, git_repo_age)
+from git import Repo
+
+
+def app_cache_folder(app: str) -> Path:
+    return REPO_APPS_ROOT / ".apps_cache" / app
+
+
+def app_cache_clone(app: str, infos: dict[str, str]) -> None:
+    logging.info("Cloning %s...", app)
+    git_depths = {
+        "notworking": 5,
+        "inprogress": 20,
+        "default": 40,
+    }
+    if app_cache_folder(app).exists():
+        shutil.rmtree(app_cache_folder(app))
+    Repo.clone_from(
+        infos["url"],
+        to_path=app_cache_folder(app),
+        depth=git_depths.get(infos["state"], git_depths["default"]),
+        single_branch=True, branch=infos.get("branch", "master"),
+    )
+
+
+def app_cache_clone_or_update(app: str, infos: dict[str, str]) -> None:
+    app_path = app_cache_folder(app)
+
+    # Don't refresh if already refreshed during last hour
+    age = git_repo_age(app_path)
+    if age is False:
+        app_cache_clone(app, infos)
+        return
+
+    # if age < 3600:
+    #     logging.info(f"Skipping {app}, it's been updated recently.")
+    #     return
+
+    logging.info("Updating %s...", app)
+    repo = Repo(app_path)
+    repo.remote("origin").set_url(infos["url"])
+
+    branch = infos.get("branch", "master")
+    if repo.active_branch != branch:
+        all_branches = [str(b) for b in repo.branches]
+        if branch in all_branches:
+            repo.git.checkout(branch, "--force")
+        else:
+            repo.git.remote("set-branches", "--add", "origin", branch)
+            repo.remote("origin").fetch(f"{branch}:{branch}")
+
+    repo.remote("origin").fetch(refspec=branch, force=True)
+    repo.git.reset("--hard", f"origin/{branch}")
+
+
+def __app_cache_clone_or_update_mapped(data):
+    name, info = data
+    try:
+        app_cache_clone_or_update(name, info)
+    except Exception as err:
+        logging.error("Error while updating %s: %s", name, err)
+
+
+def apps_cache_update_all(apps: dict[str, dict[str, Any]], parallel: int = 8) -> None:
+    with Pool(processes=parallel) as pool:
+        tasks = pool.imap_unordered(__app_cache_clone_or_update_mapped, apps.items())
+        for _ in tqdm.tqdm(tasks, total=len(apps.keys())):
+            pass
+
+
+def __run_for_catalog():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-v", "--verbose", action="store_true")
+    parser.add_argument("-j", "--processes", type=int, default=8)
+    args = parser.parse_args()
+    if args.verbose:
+        logging.getLogger().setLevel(logging.INFO)
+
+    apps_cache_update_all(get_catalog(), parallel=args.processes)
+
+
+if __name__ == "__main__":
+    __run_for_catalog()

tools/appslib/apps_cache.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+
+import logging
+from pathlib import Path
+
+import utils
+from git import Repo
+
+
+def apps_cache_path() -> Path:
+    path = apps_repo_root() / ".apps_cache"
+    path.mkdir()
+    return path
+
+
+def app_cache_path(app: str) -> Path:
+    path = apps_cache_path() / app
+    path.mkdir()
+    return path
+
+
+# def refresh_all_caches(catalog: dict[str, dict[str, str]]):
+#     for app, infos
+#     pass
+
+
+def app_cache_clone(app: str, infos: dict[str, str]) -> None:
+    git_depths = {
+        "notworking": 5,
+        "inprogress": 20,
+        "default": 40,
+    }
+
+    Repo.clone_from(
+        infos["url"],
+        to_path=app_cache_path(app),
+        depth=git_depths.get(infos["state"], git_depths["default"]),
+        single_branch=True, branch=infos.get("branch", "master"),
+    )
+
+
+def app_cache_update(app: str, infos: dict[str, str]) -> None:
+    app_path = app_cache_path(app)
+    age = utils.git_repo_age(app_path)
+    if age is False:
+        return app_cache_clone(app, infos)
+
+    if age < 3600:
+        logging.info(f"Skipping {app}, it's been updated recently.")
+        return
+
+    repo = Repo(app_path)
+    repo.remote("origin").set_url(infos["url"])
+
+    branch = infos.get("branch", "master")
+    if repo.active_branch != branch:
+        all_branches = [str(b) for b in repo.branches]
+        if branch in all_branches:
+            repo.git.checkout(branch, "--force")
+        else:
+            repo.git.remote("set-branches", "--add", "origin", branch)
+            repo.remote("origin").fetch(f"{branch}:{branch}")
+
+    repo.remote("origin").fetch(refspec=branch, force=True)
+    repo.git.reset("--hard", f"origin/{branch}")
+
+
+def cache_all_apps(catalog: dict[str, dict[str, str]]) -> None:

tools/appslib/utils.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+
+import sys
+import subprocess
+from typing import Any, TextIO, Generator
+import time
+from functools import cache
+from pathlib import Path
+from git import Repo
+
+import toml
+
+REPO_APPS_ROOT = Path(Repo(__file__, search_parent_directories=True).working_dir)
+
+
+@cache
+def apps_repo_root() -> Path:
+    return Path(__file__).parent.parent.parent
+
+
+def git(cmd: list[str], cwd: Path | None = None) -> str:
+    full_cmd = ["git"]
+    if cwd:
+        full_cmd.extend(["-C", str(cwd)])
+    full_cmd.extend(cmd)
+    return subprocess.check_output(
+        full_cmd,
+        # env=my_env,
+    ).strip().decode("utf-8")
+
+
+def git_repo_age(path: Path) -> bool | int:
+    for file in [path / ".git" / "FETCH_HEAD", path / ".git" / "HEAD"]:
+        if file.exists():
+            return int(time.time() - file.stat().st_mtime)
+    return False
+
+
+# Progress bar helper, stolen from https://stackoverflow.com/a/34482761
+def progressbar(
+        it: list[Any],
+        prefix: str = "",
+        size: int = 60,
+        file: TextIO = sys.stdout) -> Generator[Any, None, None]:
+    count = len(it)
+
+    def show(j, name=""):
+        name += "          "
+        x = int(size * j / count)
+        file.write(
+            "%s[%s%s] %i/%i %s\r" % (prefix, "#" * x, "." * (size - x), j, count, name)
+        )
+        file.flush()
+
+    show(0)
+    for i, item in enumerate(it):
+        yield item
+        show(i + 1, item[0])
+    file.write("\n")
+    file.flush()
+
+
+@cache
+def get_catalog(working_only=False):
+    """Load the app catalog and filter out the non-working ones"""
+    catalog = toml.load((REPO_APPS_ROOT / "apps.toml").open("r", encoding="utf-8"))
+    if working_only:
+        catalog = {
+            app: infos for app, infos in catalog.items()
+            if infos.get("state") != "notworking"
+        }
+    return catalog

tools/autopatches/autopatch.py

@@ -1,9 +1,11 @@
 #!/usr/bin/python3
+
 import json
-import sys
-import requests
 import os
 import subprocess
+import sys
+
+import requests
 
 catalog = requests.get("https://raw.githubusercontent.com/YunoHost/apps/master/apps.json").json()
 

tools/autopatches/patches/add-cpe/patch.sh

@@ -1,7 +1,8 @@
 #!/usr/bin/python3
 
-import json
 import csv
+import json
+
 
 def find_cpe(app_id):
     with open("../../patches/add-cpe/cpe.csv", newline='') as f:

tools/autoupdate_app_sources/autoupdate_app_sources.py

@@ -1,13 +1,15 @@
-import time
+#!/usr/bin/env python3
+
+import glob
 import hashlib
+import os
 import re
 import sys
-import requests
-import toml
-import os
-import glob
+import time
 from datetime import datetime
 
+import requests
+import toml
 from rest_api import GithubAPI, GitlabAPI, RefType
 
 STRATEGIES = [

tools/autoupdate_app_sources/rest_api.py

@@ -1,8 +1,11 @@
-from enum import Enum
+#!/usr/bin/env python3
+
 import re
-import requests
+from enum import Enum
 from typing import List
 
+import requests
+
 
 class RefType(Enum):
     tags = 1

tools/autoupdater-upgrader/autoupdater-upgrader.py

@@ -1,14 +1,17 @@
-#!venv/bin/python3
+#!/usr/bin/env python3
 
-import sys, os, time
-import urllib.request, json
+import json
+import os
 import re
+import sys
+import time
+import urllib.request
 
-from github import Github
 import github
-
+from github import Github
 # Debug
 from rich.traceback import install
+
 install(width=150, show_locals=True, locals_max_length=None, locals_max_string=None)
 
 #####

tools/bot-repo-cleanup/cleanup.py

@@ -1,4 +1,4 @@
-#!venv/bin/python3
+#!/usr/bin/env python3
 
 # Obtained with `pip install PyGithub`, better within a venv
 from github import Github

tools/catalog_linter.py

@@ -2,10 +2,10 @@
 
 import json
 import sys
+from difflib import SequenceMatcher
 from functools import cache
 from pathlib import Path
 from typing import Any, Dict, Generator, List, Tuple
-from difflib import SequenceMatcher
 
 import jsonschema
 import toml

tools/list_builder.py

@@ -1,24 +1,26 @@
 #!/usr/bin/python3
 
 import copy
-import sys
+import json
 import os
 import re
-import json
-from shutil import which
-import toml
 import subprocess
+import sys
 import time
-from typing import TextIO, Generator, Any
+from collections import OrderedDict
 from pathlib import Path
+from shutil import which
+from typing import Any, Generator, TextIO
+
+import toml
 from git import Repo
 
-from collections import OrderedDict
-from tools.packaging_v2.convert_v1_manifest_to_v2_for_catalog import convert_v1_manifest_to_v2_for_catalog
+from packaging_v2.convert_v1_manifest_to_v2_for_catalog import \
+    convert_v1_manifest_to_v2_for_catalog  # pylint: disable=import-error
 
 now = time.time()
 
-REPO_APPS_PATH = Path(__file__).parent
+REPO_APPS_PATH = Path(__file__).parent.parent
 
 # Load categories and reformat the structure to have a list with an "id" key
 categories = toml.load((REPO_APPS_PATH / "categories.toml").open("r", encoding="utf-8"))

tools/packaging_v2/__init__.py

@@ -0,0 +1 @@
+#!/usr/bin/env python3

tools/packaging_v2/convert_app_to_packaging_v2.py

@@ -1,7 +1,9 @@
+#!/usr/bin/env python3
+
 import argparse
+import json
 import os
 import re
-import json
 import subprocess
 from glob import glob
 
@@ -226,7 +228,8 @@ def _convert_v1_manifest_to_v2(app_path):
 def _dump_v2_manifest_as_toml(manifest):
 
     import re
-    from tomlkit import document, nl, table, dumps, comment
+
+    from tomlkit import comment, document, dumps, nl, table
 
     toml_manifest = document()
     toml_manifest.add("packaging_format", 2)

tools/packaging_v2/convert_v1_manifest_to_v2_for_catalog.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 import copy
 
 

tools/update_app_levels/update_app_levels.py

@@ -1,106 +1,226 @@
-import time
-import toml
-import requests
+#!/usr/bin/env python3
+"""
+Update app catalog: commit, and create a merge request
+"""
+
+import argparse
+import logging
 import tempfile
-import os
-import sys
-import json
+import textwrap
+import time
 from collections import OrderedDict
+from typing import Any
 
-token = open(os.path.dirname(__file__) + "/../../.github_token").read().strip()
+from pathlib import Path
+import jinja2
+import requests
+import toml
+from git import Repo
 
-tmpdir = tempfile.mkdtemp(prefix="update_app_levels_")
-os.system(f"git clone 'https://oauth2:{token}@github.com/yunohost/apps' {tmpdir}")
-os.system(f"git -C {tmpdir} checkout -b update_app_levels")
+# APPS_REPO = "YunoHost/apps"
+APPS_REPO = "Salamandar/apps"
 
-# Load the app catalog and filter out the non-working ones
-catalog = toml.load(open(f"{tmpdir}/apps.toml"))
-
-# Fetch results from the CI
 CI_RESULTS_URL = "https://ci-apps.yunohost.org/ci/api/results"
-ci_results = requests.get(CI_RESULTS_URL).json()
 
-comment = {
-    "major_regressions": [],
-    "minor_regressions": [],
-    "improvements": [],
-    "outdated": [],
-    "missing": [],
-}
+REPO_APPS_ROOT = Path(Repo(__file__, search_parent_directories=True).working_dir)
 
-for app, infos in catalog.items():
 
-    if infos.get("state") != "working":
-        continue
+def github_token() -> str | None:
+    github_token_path = REPO_APPS_ROOT.parent / ".github_token"
+    if github_token_path.exists():
+        return github_token_path.open("r", encoding="utf-8").read().strip()
+    return None
 
-    if app not in ci_results:
-        comment["missing"].append(app)
-        continue
 
+def get_ci_results() -> dict[str, dict[str, Any]]:
+    return requests.get(CI_RESULTS_URL, timeout=10).json()
+
+
+def ci_result_is_outdated(result) -> bool:
     # 3600 * 24 * 60 = ~2 months
-    if (int(time.time()) - ci_results[app].get("timestamp", 0)) > 3600 * 24 * 60:
-        comment["outdated"].append(app)
-        continue
+    return (int(time.time()) - result.get("timestamp", 0)) > 3600 * 24 * 60
 
-    ci_level = ci_results[app]["level"]
-    current_level = infos.get("level")
 
-    if ci_level == current_level:
-        continue
-    elif current_level is None or ci_level > current_level:
-        comment["improvements"].append((app, current_level, ci_level))
-    elif ci_level < current_level:
-        if ci_level <= 4 and current_level > 4:
-            comment["major_regressions"].append((app, current_level, ci_level))
+def update_catalog(catalog, ci_results) -> dict:
+    """
+    Actually change the catalog data
+    """
+    # Re-sort the catalog keys / subkeys
+    for app, infos in catalog.items():
+        catalog[app] = OrderedDict(sorted(infos.items()))
+    catalog = OrderedDict(sorted(catalog.items()))
+
+    def app_level(app):
+        if app not in ci_results:
+            return 0
+        if ci_result_is_outdated(ci_results[app]):
+            return 0
+        return ci_results[app]["level"]
+
+    for app, info in catalog.items():
+        info["level"] = app_level(app)
+
+    return catalog
+
+
+def list_changes(catalog, ci_results) -> dict[str, list[tuple[str, int, int]]]:
+    """
+    Lists changes for a pull request
+    """
+
+    changes = {
+        "major_regressions": [],
+        "minor_regressions": [],
+        "improvements": [],
+        "outdated": [],
+        "missing": [],
+    }
+
+    for app, infos in catalog.items():
+        if infos.get("state") != "working":
+            continue
+
+        if app not in ci_results:
+            changes["missing"].append(app)
+            continue
+
+        if ci_result_is_outdated(ci_results[app]):
+            changes["outdated"].append(app)
+            continue
+
+        ci_level = ci_results[app]["level"]
+        current_level = infos.get("level")
+
+        if ci_level == current_level:
+            continue
+
+        if current_level is None or ci_level > current_level:
+            changes["improvements"].append((app, current_level, ci_level))
+            continue
+
+        if ci_level < current_level:
+            if ci_level <= 4 < current_level:
+                changes["major_regressions"].append((app, current_level, ci_level))
+            else:
+                changes["minor_regressions"].append((app, current_level, ci_level))
+
+    return changes
+
+
+def pretty_changes(changes: dict[str, list[tuple[str, int, int]]]) -> str:
+    pr_body_template = textwrap.dedent("""
+        {%- if changes["major_regressions"] %}
+        ### Major regressions 😭
+        {% for app in changes["major_regressions"] %}
+        - [ ] [{{app.0}}: {{app.1}} → {{app.2}}](https://ci-apps.yunohost.org/ci/apps/{{app.0}}/latestjob)
+        {%- endfor %}
+        {% endif %}
+        {%- if changes["minor_regressions"] %}
+        ### Minor regressions 😬
+        {% for app in changes["minor_regressions"] %}
+        - [ ] [{{app.0}}: {{app.1}} → {{app.2}}](https://ci-apps.yunohost.org/ci/apps/{{app.0}}/latestjob)
+        {%- endfor %}
+        {% endif %}
+        {%- if changes["improvements"] %}
+        ### Improvements 🥳
+        {% for app in changes["improvements"] %}
+        - [{{app.0}}: {{app.1}} → {{app.2}}](https://ci-apps.yunohost.org/ci/apps/{{app.0}}/latestjob)
+        {%- endfor %}
+        {% endif %}
+        {%- if changes["missing"] %}
+        ### Missing 🫠
+        {% for app in changes["missing"] %}
+        - [{{app}} (See latest job if it exists)](https://ci-apps.yunohost.org/ci/apps/{{app.0}}/latestjob)
+        {%- endfor %}
+        {% endif %}
+        {%- if changes["outdated"] %}
+        ### Outdated ⏰
+        {% for app in changes["outdated"] %}
+        - [ ] [{{app}} (See latest job if it exists)](https://ci-apps.yunohost.org/ci/apps/{{app.0}}/latestjob)
+        {%- endfor %}
+        {% endif %}
+    """)
+
+    return jinja2.Environment().from_string(pr_body_template).render(changes=changes)
+
+
+def make_pull_request(pr_body: str) -> None:
+    pr_data = {
+        "title": "Update app levels according to CI results",
+        "body": pr_body,
+        "head": "update_app_levels",
+        "base": "master"
+    }
+
+    with requests.Session() as s:
+        s.headers.update({"Authorization": f"token {github_token()}"})
+        response = s.post(f"https://api.github.com/repos/{APPS_REPO}/pulls", json=pr_data)
+
+        if response.status_code == 422:
+            response = s.get(f"https://api.github.com/repos/{APPS_REPO}/pulls", data={"head": "update_app_levels"})
+            response.raise_for_status()
+            pr_number = response.json()[0]["number"]
+
+            # head can't be updated
+            del pr_data["head"]
+            response = s.patch(f"https://api.github.com/repos/{APPS_REPO}/pulls/{pr_number}", json=pr_data)
+            response.raise_for_status()
+            existing_url = response.json()["html_url"]
+            logging.warning(f"An existing Pull Request has been updated at {existing_url} !")
         else:
-            comment["minor_regressions"].append((app, current_level, ci_level))
+            response.raise_for_status()
 
-    infos["level"] = ci_level
+            new_url = response.json()["html_url"]
+            logging.info(f"Opened a Pull Request at {new_url} !")
 
-# Also re-sort the catalog keys / subkeys
-for app, infos in catalog.items():
-    catalog[app] = OrderedDict(sorted(infos.items()))
-catalog = OrderedDict(sorted(catalog.items()))
 
-updated_catalog = toml.dumps(catalog)
-updated_catalog = updated_catalog.replace(",]", " ]")
-open(f"{tmpdir}/apps.toml", "w").write(updated_catalog)
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--commit", action=argparse.BooleanOptionalAction, default=True)
+    parser.add_argument("--pr", action=argparse.BooleanOptionalAction, default=True)
+    parser.add_argument("-v", "--verbose", action=argparse.BooleanOptionalAction)
+    args = parser.parse_args()
 
-os.system(f"git -C {tmpdir} commit apps.toml -m 'Update app levels according to CI results'")
-os.system(f"git -C {tmpdir} push origin update_app_levels --force")
-os.system(f"rm -rf {tmpdir}")
+    logging.getLogger().setLevel(logging.INFO)
+    if args.verbose:
+        logging.getLogger().setLevel(logging.DEBUG)
 
-PR_body = ""
-if comment["major_regressions"]:
-    PR_body += "\n### Major regressions\n\n"
-    for app, current_level, new_level in comment['major_regressions']:
-        PR_body += f"- [ ] {app} | {current_level} -> {new_level} | https://ci-apps.yunohost.org/ci/apps/{app}/latestjob\n"
-if comment["minor_regressions"]:
-    PR_body += "\n### Minor regressions\n\n"
-    for app, current_level, new_level in comment['minor_regressions']:
-        PR_body += f"- [ ] {app} | {current_level} -> {new_level} | https://ci-apps.yunohost.org/ci/apps/{app}/latestjob\n"
-if comment["improvements"]:
-    PR_body += "\n### Improvements\n\n"
-    for app, current_level, new_level in comment['improvements']:
-        PR_body += f"- {app} | {current_level} -> {new_level} | https://ci-apps.yunohost.org/ci/apps/{app}/latestjob\n"
-if comment["missing"]:
-    PR_body += "\n### Missing results\n\n"
-    for app in comment['missing']:
-        PR_body += f"- {app} | https://ci-apps.yunohost.org/ci/apps/{app}/latestjob\n"
-if comment["outdated"]:
-    PR_body += "\n### Outdated results\n\n"
-    for app in comment['outdated']:
-        PR_body += f"- [ ] {app} | https://ci-apps.yunohost.org/ci/apps/{app}/latestjob\n"
+    with tempfile.TemporaryDirectory(prefix="update_app_levels_") as tmpdir:
+        logging.info("Cloning the repository...")
+        apps_repo = Repo.clone_from(f"git@github.com:{APPS_REPO}", to_path=tmpdir)
 
-PR = {"title": "Update app levels according to CI results",
-      "body": PR_body,
-      "head": "update_app_levels",
-      "base": "master"}
+        # Load the app catalog and filter out the non-working ones
+        catalog = toml.load((Path(apps_repo.working_tree_dir) / "apps.toml").open("r", encoding="utf-8"))
 
-with requests.Session() as s:
-    s.headers.update({"Authorization": f"token {token}"})
-r = s.post("https://api.github.com/repos/yunohost/apps/pulls", json.dumps(PR))
+        new_branch = apps_repo.create_head("update_app_levels", apps_repo.refs.master)
+        apps_repo.head.reference = new_branch
 
-if r.status_code != 200:
-    print(r.text)
-    sys.exit(1)
+        logging.info("Retrieving the CI results...")
+        ci_results = get_ci_results()
+
+        # Now compute changes, then update the catalog
+        changes = list_changes(catalog, ci_results)
+        pr_body = pretty_changes(changes)
+        catalog = update_catalog(catalog, ci_results)
+
+        # Save the new catalog
+        updated_catalog = toml.dumps(catalog)
+        updated_catalog = updated_catalog.replace(",]", " ]")
+        (Path(apps_repo.working_tree_dir) / "apps.toml").open("w", encoding="utf-8").write(updated_catalog)
+
+        if args.commit:
+            logging.info("Committing and pushing the new catalog...")
+            apps_repo.index.add("apps.toml")
+            apps_repo.index.commit("Update app levels according to CI results")
+            apps_repo.remote().push(force=True)
+
+        if args.verbose:
+            print(pr_body)
+
+        if args.pr:
+            logging.info("Opening a pull request...")
+            make_pull_request(pr_body)
+
+
+if __name__ == "__main__":
+    main()