Merge pull request #1 from YunoHost/master

fetch

README.md

@@ -1,43 +1,57 @@
-# YunoHost apps directory
+# YunoHost application catalog
 
-<img src="https://yunohost.org/logo.png" width=80>
+<img src="https://yunohost.org/logo.png" width=80><img src="https://yunohost.org/user/images/yunohost_package.png" width=80>
 
-Here you will find the repositories and versions of every apps integrated in YunoHost.
+Here you will find the repositories and versions of every apps available in YunoHost's default catalog.
 
-https://yunohost.org/apps
+It is browsable here: https://yunohost.org/apps
 
-The application list corresponds to **apps.json** which contains references to known "free-software" YunoHost packages along with a few metadata about them.
+The main file of the catalog is [**apps.json**](./apps.json) which contains
+references to the corresponding git repositories for each application, along
+with a few metadata about them such as its category or maintenance state. This
+file regularly read by `list_builder.py` which publish the results on
+https://app.yunohost.org/default/.
 
-## Contributing
+### Where can I learn about app packaging in Yunohost ?
 
-### How to add your app to the apps list
+- You can browse the contributor documentation : https://yunohost.org/contributordoc
+- If you are not familiar with Git/Github, you can have a look at our [homemade guide](https://yunohost.org/#/packaging_apps_git)
+- Don't hesitate to reach for help on the dedicated [application packaging chatroom](https://yunohost.org/chat_rooms) ... we can even schedule an audio meeting to help you get started !
 
-**If** your app is under a free-software licence : 
+### How to add your app to the application catalog
-* Fork and edit the [apps list](https://github.com/YunoHost/apps/tree/master/apps.json)
+
+N.B. : the Yunohost project will **NOT** integrate in its catalog applications that are not
+based on free-software upstreams.
+
+To add your application to the catalog:
+* Fork this repository and edit the [apps.json](https://github.com/YunoHost/apps/tree/master/apps.json) file
 * Add your app's ID and git information at the right alphabetical place
 * Indicate the app's functioning state: `notworking`, `inprogress`, or `working`
-* Do not add the level yourself. The CI will do it.
+* *Do not* add the level entry by yourself. Our automatic test suite ("the CI") will handle it.
-* Send a [Pull Request](https://github.com/YunoHost/apps/pulls/)
+* Create a [Pull Request](https://github.com/YunoHost/apps/pulls/)
 
 App example addition:
 ```json
     "wallabag": {
         "branch": "master",
-        "revision": "c2fc62438ac5c9503e3f4ebfdc425ec03a0ec0c0",
+        "revision": "HEAD",
         "url": "https://github.com/abeudin/wallabag_ynh",
         "state": "working"
     }
 ```
 
-N.B. : You can now put `HEAD` as `revision`. This way, you won't have to come and update this file each time you change things in your app. *But* this also means that any change to your `master` branch will be made available to everybody. Hence, when developing things which are not production-ready, if you use `HEAD` we strongly recommend that you develop in a `testing` branch (for instance) until you consider things stable enough to be merged in `master`.
+N.B. : We strongly encourage you to transfer the ownership of your repository to
+the Yunohost-Apps organization on Github, such that the community will help you
+with keeping your app working and up to date with packaging evolutions.
 
-N.B. 2 : Organization is still debating about what to do with non-free apps listing (cf. [this thread](https://forum.yunohost.org/t/about-community-and-official-apps/6372/25). Such a list is unlikely to be maintained by the YunoHost project officially. However, it could be created and maintained by member of the community. Check out [the forum](https://forum.yunohost.org) about this.
+N.B.2 : If `"revision": "HEAD"` is used in `apps.json`, any commit to the
+`master` branch on your app will automatically be published to the catalog.
+Therefore we strongly encourage you to develop in separate branches, and only
+merge changes that were carefully tested. Get in touch with the Apps group to
+obtain an access to the developer CI where you'll be able to test your app
+easily.
 
-### How to help translating
+#### Helper script
-
-We invite you to use [translate.yunohost.org](https://translate.yunohost.org/) instead of doing Pull Request for files in `locales` folder.
-
-### Helper script
 
 You can use the <code>add_or_update.py</code> python script to add or update
 your app from one of the 2 json files.
@@ -48,70 +62,24 @@ Usage:
 ./add_or_update.py apps.json [github/gitlab url OR app name [github/gitlab url OR app name [github/gitlab url OR app name ...]]]
 ```
 
-### How to make my app a High Quality app ?
+### How to help translating
 
-A High Quality app will be highlighted in the app list and marked as a level 8 app.  
+Update on Nov. 2020 : this part is broken / not maintained anymore for the
-To become a High Quality app, a package has to follow the following rules:
+moment...
 
-* The app should already have been in the community list for 2 months.
+We invite you to use [translate.yunohost.org](https://translate.yunohost.org/)
-* The app should be kept up to date, regarding the upstream source (if it’s possible with our current YunoHost version).
+instead of doing Pull Request for files in `locales` folder.
-* The package itself should be up to date regarding the packaging recommendations and helpers.
-* The package should be level 7 for at least 1 month.
-* The repository should have testing and master branches, at least. The list should point to HEAD, so the list stays up to date.
-* Any modification should be done to the testing branch, and wait at least for one approval of one member of the Apps group so that we can ensure that there’s nothing in opposition to those criteria, nor any changes that would harm servers.
-* The package should comply with the [requirements of the level 8](https://github.com/YunoHost/doc/blob/master/packaging_apps_levels.md#level-8).
 
-You can find the validation form used by Apps group [here](https://github.com/YunoHost/apps/blob/master/hq_validation_template.md).
+### How to make my app flagged as High Quality ?
 
-If the app is already tagged as High Quality and one of those criteria isn't respected anymore: after a warning, the tag will be removed until the criterion is again validated.
+A High Quality app will be highlighted in the app list and marked as a level 9 app.  
+To become a High Quality app, a package has to follow the criterias listed [here](hq_validation_template.md).
 
-To make an app a High Quality app, technically, you have to add the tag ```"high_quality": true```.
+Once the app is validated is "high quality", the tag `"high_quality": true`
-```json
+shall be added to the app infos inside the catalog (`apps.json`).
-    "wallabag": {
-        "branch": "master",
-        "high_quality": true,
-        "revision": "HEAD",
-        "url": "https://github.com/abeudin/wallabag_ynh.git",
-        "state": "working"
-    }
-```
 
-### How to make my app a Featured app ?
+### Apps flagged as not-maintained
 
-A Featured app is highlighted in the app list and shown before any others.  
+Applications with no recent activity and no active sign from maintainer may be flagged in `apps.json` with `"maintained": false` to signify that the app is inactive and may slowly become outdated with respect to the upstream, or with respect to good packaging practices. It does **not** mean that the app is not working anymore.
-To become a Featured app, a package has to follow the following rules:
 
-* The app should already be a High Quality app.
+Feel free to contact the app group if you feel like taking over the maintenance of a currently unmaintained app!
-* The upstream app should be accessible and well made.
-* The app should be interesting and demanded by the community.
-* The app should fit the spirit of YunoHost.
-
-**Please note that the exact process to decide which apps are going to be Featured, and for how many time, isn't yet defined...**
-
-To make an app a Featured app, technically, you have to add the tag ```"featured": true```.
-```json
-    "wallabag": {
-        "branch": "master",
-        "high_quality": true,
-        "featured": true,
-        "revision": "HEAD",
-        "url": "https://github.com/abeudin/wallabag_ynh.git",
-        "state": "working"
-    }
-```
-
-### What to do if I can't maintain my app anymore ?
-
-If you don't have time anymore to maintain an app, you can update its status to inform users and packagers that you will not maintain it anymore.  
-In order to do so, use the tag `"maintained":`.  
-This tag can have 5 different values:
-- `"maintained": true` That's the default value if the tag isn't present for your app. That simply means that this app is maintained.
-- `"maintained": "request_help"` Use that value to inform other packagers that you need help to maintain this app. You'll then be more than one maintainer for this apps.
-- `"maintained": "request_adoption"` Use that value to inform other packagers, as well as users, that you're going to give up that app. So that you would like another maintainer to take care of it.
-- `"maintained": false` or `"maintained": "orphaned"` This value means that this app is no longer maintained... That means also that a packager can declare himself/herself as its new maintainer.  
-Please contact the Apps group if you want to take care of an unmaintained app.
-
-If you want to modify the status of one of your apps, for any reason, please consider informing the community via the forum. Users would probably be glad to be informed that an app they use will become unmaintained.
-
-#### More information
-See [yunohost.org/packaging_apps](https://yunohost.org/packaging_apps)

autopatches/autopatch.py

@@ -0,0 +1,188 @@
+#!/usr/bin/python3
+import json
+import sys
+import requests
+import os
+import subprocess
+
+catalog = requests.get("https://raw.githubusercontent.com/YunoHost/apps/master/apps.json").json()
+
+my_env = os.environ.copy()
+my_env["GIT_TERMINAL_PROMPT"] = "0"
+os.makedirs(".apps_cache", exist_ok=True)
+
+login = open("login").read().strip()
+token = open("token").read().strip()
+github_api = "https://api.github.com"
+
+
+def apps(min_level=4):
+
+    for app, infos in catalog.items():
+        if infos.get("state") == "working" and infos.get("level", -1) > min_level:
+            infos["id"] = app
+            yield infos
+
+
+def app_cache_folder(app):
+    return os.path.join(".apps_cache", app)
+
+
+def git(cmd, in_folder=None):
+
+    if not isinstance(cmd, list):
+        cmd = cmd.split()
+    if in_folder:
+        cmd = ["-C", in_folder] + cmd
+    cmd = ["git"] + cmd
+    return subprocess.check_output(cmd, env=my_env).strip().decode("utf-8")
+
+
+# Progress bar helper, stolen from https://stackoverflow.com/a/34482761
+def progressbar(it, prefix="", size=60, file=sys.stdout):
+    it = list(it)
+    count = len(it)
+    def show(j, name=""):
+        name += "          "
+        x = int(size*j/count)
+        file.write("%s[%s%s] %i/%i %s\r" % (prefix, "#"*x, "."*(size-x), j,  count, name))
+        file.flush()
+    show(0)
+    for i, item in enumerate(it):
+        yield item
+        show(i+1, item["id"])
+    file.write("\n")
+    file.flush()
+
+
+def build_cache():
+
+    for app in progressbar(apps(), "Git cloning: ", 40):
+        folder = os.path.join(".apps_cache", app["id"])
+        reponame = app["url"].rsplit("/", 1)[-1]
+        git(f"clone --quiet --depth 1 --single-branch {app['url']} {folder}")
+        git(f"remote add fork https://{login}:{token}@github.com/{login}/{reponame}", in_folder=folder)
+
+
+def apply(patch):
+
+    patch_path = os.path.abspath(os.path.join("patches", patch, "patch.sh"))
+
+    for app in progressbar(apps(), "Apply to: ", 40):
+        folder = os.path.join(".apps_cache", app["id"])
+        current_branch = git(f"symbolic-ref --short HEAD", in_folder=folder)
+        git(f"reset --hard origin/{current_branch}", in_folder=folder)
+        os.system(f"cd {folder} && bash {patch_path}")
+
+
+def diff():
+
+    for app in apps():
+        folder = os.path.join(".apps_cache", app["id"])
+        if bool(subprocess.check_output(f"cd {folder} && git diff", shell=True).strip().decode("utf-8")):
+            print("\n\n\n")
+            print("=================================")
+            print("Changes in : " + app["id"])
+            print("=================================")
+            print("\n")
+            os.system(f"cd {folder} && git --no-pager diff")
+
+
+def push(patch):
+
+    title = "[autopatch] " + open(os.path.join("patches", patch, "pr_title.md")).read().strip()
+
+    def diff_not_empty(app):
+        folder = os.path.join(".apps_cache", app["id"])
+        return bool(subprocess.check_output(f"cd {folder} && git diff", shell=True).strip().decode("utf-8"))
+
+    def app_is_on_github(app):
+        return "github.com" in app["url"]
+
+    apps_to_push = [app for app in apps() if diff_not_empty(app) and app_is_on_github(app)]
+
+    with requests.Session() as s:
+        s.headers.update({"Authorization": f"token {token}"})
+        for app in progressbar(apps_to_push, "Forking: ", 40):
+            app["repo"] = app["url"][len("https://github.com/"):].strip("/")
+            fork_if_needed(app["repo"], s)
+
+        for app in progressbar(apps_to_push, "Pushing: ", 40):
+            app["repo"] = app["url"][len("https://github.com/"):].strip("/")
+            app_repo_name = app["url"].rsplit("/", 1)[-1]
+            folder = os.path.join(".apps_cache", app["id"])
+            current_branch = git(f"symbolic-ref --short HEAD", in_folder=folder)
+            git(f"reset origin/{current_branch}", in_folder=folder)
+            git(["commit", "-a", "-m", title, "--author='Yunohost-Bot <>'"], in_folder=folder)
+            try:
+                git(f"remote remove fork", in_folder=folder)
+            except Exception:
+                pass
+            git(f"remote add fork https://{login}:{token}@github.com/{login}/{app_repo_name}", in_folder=folder)
+            git(f"push fork {current_branch}:{patch} --quiet --force", in_folder=folder)
+            create_pull_request(app["repo"], patch, current_branch, s)
+
+
+def fork_if_needed(repo, s):
+
+    repo_name = repo.split("/")[-1]
+    r = s.get(github_api + f"/repos/{login}/{repo_name}")
+
+    if r.status_code == 200:
+        return
+
+    r = s.post(github_api + f"/repos/{repo}/forks")
+
+    if r.status_code != 200:
+        print(r.text)
+
+
+def create_pull_request(repo, patch, base_branch, s):
+
+    PR = {"title": "[autopatch] " + open(os.path.join("patches", patch, "pr_title.md")).read().strip(),
+          "body": "This is an automatic PR\n\n" + open(os.path.join("patches", patch, "pr_body.md")).read().strip(),
+          "head": login + ":" + patch,
+          "base": base_branch,
+          "maintainer_can_modify": True}
+
+    r = s.post(github_api + f"/repos/{repo}/pulls", json.dumps(PR))
+
+    if r.status_code != 200:
+        print(r.text)
+    else:
+        json.loads(r.text)["html_url"]
+
+
+def main():
+
+    action = sys.argv[1]
+    if action == "--help":
+        print("""
+    Example usage:
+
+# Init local git clone for all apps
+./autopatch --build-cache
+
+# Apply patch in all local clones
+./autopatch --apply explicit-php-version-in-deps
+
+# Inspect diff for all apps
+./autopatch --diff
+
+# Push and create pull requests on all apps with non-empty diff
+./autopatch --push explicit-php-version-in-deps
+""")
+
+    elif action == "--build-cache":
+        build_cache()
+    elif action == "--apply":
+        apply(sys.argv[2])
+    elif action == "--diff":
+        diff()
+    elif action == "--push":
+        push(sys.argv[2])
+    else:
+        print("Unknown action %s" % action)
+
+
+main()

autopatches/patches/missing-seteu-in-change_url/patch.sh

@@ -0,0 +1,10 @@
+
+cd scripts/
+
+if [ ! -e change_url ] || grep -q 'ynh_abort_if_errors' change_url
+then
+    # The app doesn't has any change url script or already has ynh_abort_if_error
+    exit 0
+fi
+
+sed 's@\(source /usr/share/yunohost/helpers\)@\1\nynh_abort_if_errors@g' -i change_url

autopatches/patches/missing-seteu-in-change_url/pr_body.md

@@ -0,0 +1,2 @@
+
+This is an ***automated*** patch to fix the lack of `ynh_abort_if_errors` in change_url script

autopatches/patches/missing-seteu-in-change_url/pr_title.md

@@ -0,0 +1 @@
+Missing ynh_abort_if_errors in change_url scripts

autopatches/patches/new-permission-system/patch.sh

@@ -0,0 +1,66 @@
+
+cd scripts/
+
+if grep -q 'ynh_legacy_permissions' upgrade || grep -q 'ynh_permission_' install
+then
+    # App already using the new permission system - not patching anything
+    exit 0
+fi
+
+if ! grep -q "protected_\|skipped_" install
+then
+    # App doesn't has any (un)protected / skipped setting ?
+    # Probably not a webapp or permission ain't relevant for it ?
+    exit 0
+fi
+
+CONFIGURE_PERMISSION_DURING_INSTALL='
+# Make app public if necessary
+if [ \"\$is_public\" -eq 1 ]
+then
+	ynh_permission_update --permission=\"main\" --add=\"visitors\"
+fi
+'
+
+MIGRATE_LEGACY_PERMISSIONS='
+#=================================================
+# Migrate legacy permissions to new system
+#=================================================
+if ynh_legacy_permissions_exists
+then
+	ynh_legacy_permissions_delete_all
+
+	ynh_app_setting_delete --app=\$app --key=is_public
+fi'
+
+for SCRIPT in "remove upgrade backup restore change_url"
+do
+    [[ -e $SCRIPT ]] || continue
+
+    perl -p0e 's@.*ynh_app_setting_.*protected_.*@@g' -i $SCRIPT
+    perl -p0e 's@.*ynh_app_setting_.*skipped_.*@@g' -i $SCRIPT
+    perl -p0e 's@\s*if.*-z.*is_public.*(.|\n)*?fi\s@\n@g' -i $SCRIPT
+    perl -p0e 's@\s*if.*is_public.*(-eq|=).*(.|\n)*?fi\s@\n@g' -i $SCRIPT
+    perl -p0e 's@is_public=.*\n@@g' -i $SCRIPT
+    perl -p0e 's@ynh_app_setting_.*is_public.*@@g' -i $SCRIPT
+    perl -p0e 's@.*# Make app .*@@g' -i $SCRIPT
+    perl -p0e 's@.*# Fix is_public as a boolean.*@@g' -i $SCRIPT
+    perl -p0e 's@.*# If app is public.*@@g' -i $SCRIPT
+    perl -p0e 's@.*# .*allow.*credentials.*anyway.*@@g' -i $SCRIPT
+    perl -p0e 's@.*ynh_script_progression.*SSOwat.*@@g' -i $SCRIPT
+    perl -p0e 's@#=*\s#.*SETUP SSOWAT.*\s#=*\s@@g' -i $SCRIPT
+done
+
+
+perl -p0e 's@.*domain_regex.*@@g' -i install
+perl -p0e 's@.*# If app is public.*@@g' -i install
+perl -p0e 's@.*# Make app .*@@g' -i install
+perl -p0e 's@.*# .*allow.*credentials.*anyway.*@@g' -i install
+perl -p0e "s@if.*is_public.*(-eq|=)(.|\n){0,100}setting(.|\n)*?fi\n@$CONFIGURE_PERMISSION_DURING_INSTALL@g" -i install
+perl -p0e 's@.*ynh_app_setting_.*is_public.*\s@@g' -i install
+perl -p0e 's@.*ynh_app_setting_.*protected_.*@@g' -i install
+perl -p0e 's@.*ynh_app_setting_.*skipped_.*@@g' -i install
+
+grep -q 'is_public=' install || perl -p0e 's@(.*Configuring SSOwat.*)@\1\nynh_permission_update --permission=\"main\" --add=\"visitors\"@g' -i install
+
+perl -p0e "s@ynh_abort_if_errors@ynh_abort_if_errors\n$MIGRATE_LEGACY_PERMISSIONS@g" -i upgrade

autopatches/patches/new-permission-system/pr_body.md

@@ -0,0 +1,11 @@
+
+NB. : this is an ***automated*** attempt to migrate the app to the new permission system
+
+You should ***not*** blindly trust the proposed changes. In particular, the auto-patch will not handle:
+- situations which are more complex than "if is_public is true, allow visitors"
+- situations where the app needs to be temporarily public (then possible private) during initial configuration
+- apps that need to define extra permission for specific section of the app (such as admin interface)
+- apps using non-standard syntax
+- other specific use cases
+
+***PLEASE*** carefully review, test and amend the proposed changes if you find that the autopatch did not do a proper job.

autopatches/patches/new-permission-system/pr_title.md

@@ -0,0 +1 @@
+Autopatch to migrate to new permission system

hq_validation_template.md

@@ -2,25 +2,21 @@
 
 Package URL:
 
-This template is designed to be used as it is by Apps group to validate requests from packagers for the tag High Quality.
+This template is designed to be used by the Apps group to validate requests from packagers for the tag High Quality.
 
-Mandatory check boxes:
+- [ ] The package is level 8.
-- [ ] The package is level 7.
+- [ ] The app is reasonably up to date with the upstream version.
-- [ ] The package has been level 7 for at least 1 month.
+- [ ] The maintainers intend to maintain the app, and will communicate with the Apps group if they intend to stop maintaining the app.
-- [ ] The package has been in the list for at least 2 months.
+- [ ] The package **supports all recommended integrations with Yunohost**, in particular:
-- [ ] The package is up to date regarding the packaging recommendations and helpers.
+    - [ ] Architectures: The package has been tested and validated for other architectures it's supposed to work on (in particular ARM or 32bit), or properly handles the detection of unsupported architectures at the beginning of the install script.
-- [ ] The repository has a testing branch.
+    - [ ] Yunohost tile integration: The package integrates the YunoHost tile `yunohost_panel.conf.inc` in its nginx configuration.
-- [ ] All commits are made in testing branch before being merged into master.
+    - [ ] LDAP/SSO integration *(if relevant)*: The package supports LDAP authentication **and** automatic login through Yunohost's SSO.
-- [ ] The list points to HEAD, not to a specific commit.
+    - [ ] Fail2ban integration *(if relevant)*: The package provides rules to block brute force attempts on the app
-- [ ] The repository has a [`pull_request_template.md`](https://github.com/YunoHost/apps/blob/master/pull_request_template-HQ-apps.md)
+- [ ] The package has been **reviewed by members of the Apps group** to validate that:
-- [ ] The package shows the YunoHost tile `yunohost_panel.conf.inc`
+   - [ ] It is up to date with the recommended packaging practices.
-
+   - [ ] There are no obvious security issues or borderline practices.
-Optional check boxes:
+- [ ] The maintainers agree to follow the **recommended development workflow**:
-- [ ] The package is level 7 for ARM as well.
+   - [ ] The `revision` field in the app catalog (`apps.json`) points to `HEAD`
-*If the app is really important for the community, we can accept it with a broken ARM support. But this should be clearly explained and managed.*
+   - [ ] All pull requests should target the `testing` branch before being merged into `master`.
-- [ ] The app is up to date with the upstream version.  
+   - [ ] All pull requests should be reviewed and validated by another member of the app group before merging.
-*If this is possible with the last YunoHost version.*
+   - [ ] The repository has a [`pull_request_template.md`](https://github.com/YunoHost/apps/blob/master/pull_request_template-HQ-apps.md).
-- [ ] The package supports LDAP  
-*If the app upstream supports it*
-- [ ] The package supports HTTP authentication  
-*If the app upstream supports it*

list_builder.py

@@ -178,6 +178,32 @@ def build_catalog():
     with open("./builds/default/v0/community.json", 'w') as f:
         f.write(json.dumps(community_apps_dict, sort_keys=True))
 
+    ##############################
+    # Version for catalog in doc #
+    ##############################
+    categories = yaml.load(open("categories.yml").read())
+    os.system("mkdir -p ./builds/default/doc_catalog")
+    def infos_for_doc_catalog(infos):
+        level = infos.get("level")
+        if not isinstance(level, int):
+            level = -1
+        return {
+            "id": infos["id"],
+            "category": infos["category"],
+            "url": infos["git"]["url"],
+            "name": infos["manifest"]["name"],
+            "description": infos["manifest"]["description"],
+            "state": infos["state"],
+            "level": level,
+            "broken": level <= 0,
+            "good_quality": level >= 8,
+            "bad_quality":  level <= 5,
+        }
+    result_dict_doc = {k: infos_for_doc_catalog(v) for k, v in result_dict.items() if v["state"] in ["working", "validated"]}
+    with open("builds/default/doc_catalog/apps.json", 'w') as f:
+        f.write(json.dumps({"apps": result_dict_doc, "categories": categories}, sort_keys=True))
+
+
 
 def build_app_dict(app, infos):
 
@@ -242,7 +268,7 @@ def include_translations_in_manifest(manifest):
             for question in questions:
                 key = "%s_manifest_arguments_%s_%s" % (app_name, category, question["name"])
                 # don't overwrite already existing translation in manifests for now
-                if translations.get(key) and not current_lang not in question["ask"]:
+                if translations.get(key) and "ask" in question and not current_lang not in question["ask"]:
                     #print("[ask]", current_lang, key)
                     question["ask"][current_lang] = translations[key]