Merge branch 'stretch-unstable' into patch-3

.github/FUNDING.yml

@@ -0,0 +1,4 @@
+# These are supported funding model platforms
+
+custom: https://donate.yunohost.org
+liberapay: YunoHost

README.md

@@ -1,48 +1,43 @@
+<p align="center">
+    <img alt="YunoHost" src="https://raw.githubusercontent.com/YunoHost/doc/master/images/logo_roundcorner.png" width="100px" />
+</p>
+
+<h1 align="center">YunoHost</h1>
+
+<div align="center">
+
 [![Build status](https://travis-ci.org/YunoHost/yunohost.svg?branch=stretch-unstable)](https://travis-ci.org/YunoHost/yunohost)
 [![GitHub license](https://img.shields.io/github/license/YunoHost/yunohost)](https://github.com/YunoHost/yunohost/blob/stretch-unstable/LICENSE)
+[![Mastodon Follow](https://img.shields.io/mastodon/follow/28084)](https://mastodon.social/@yunohost)
 
-# YunoHost core
+</div>
 
-This repository is the core of YunoHost code.
+YunoHost is an operating system aiming to simplify as much as possible the administration of a server.
 
+This repository corresponds to the core code of YunoHost, mainly written in Python and Bash.
+
+- [Project features](https://yunohost.org/#/whatsyunohost)
 - [Project website](https://yunohost.org)
-- [Bugtracker](https://github.com/YunoHost/issues).
+- [Install documentation](https://yunohost.org/install)
+- [Issue tracker](https://github.com/YunoHost/issues)
+
+# Screenshots
+
+Webadmin ([Yunohost-Admin](https://github.com/YunoHost/yunohost-admin)) | Single sign-on user portal ([SSOwat](https://github.com/YunoHost/ssowat))
+--- |  ---
+![](https://raw.githubusercontent.com/YunoHost/doc/master/images/webadmin.png) | ![](https://raw.githubusercontent.com/YunoHost/doc/master/images/user_panel.png)
+
 
 ## Contributing
 
-- You can develop on this repository using [ynh-dev](https://github.com/YunoHost/ynh-dev) with `use-git` sub-command.
+- You can learn how to get started with developing on YunoHost by reading [this piece of documentation](https://yunohost.org/dev).
-- On this repository we are [following this workflow](https://yunohost.org/#/build_system_en): `stable ← testing ← unstable ← your_branch`.
+- Come chat with us on the [dev chatroom](https://yunohost.org/#/chat_rooms) !
-- Note: If you modify Python scripts, you will have to modifiy the actions map.
 - You can help translate YunoHost on our [translation platform](https://translate.yunohost.org/engage/yunohost/?utm_source=widget)
 
+<p align="center">
 <img src="https://translate.yunohost.org/widgets/yunohost/-/multi-auto.svg" alt="Translation status" />
-
+</p>
-
-## Repository content
-
-- [YunoHost core Python 2.7 scripts](./src/yunohost).
-- [An actionsmap](./data/actionsmap/yunohost.yml) used by moulinette.
-- [Services configuration templates](./data/templates).
-- [Hooks](./data/hooks).
-- [Locales](./locales) for translations of `yunohost` command.
-- [Shell helpers](./helpers.d) for [application packaging](https://yunohost.org/#/packaging_apps_helpers_en).
-- [Modules for the XMPP server Metronome](./lib/metronome/modules).
-- [Debian files](./debian) for package creation.
-
-## How does it work?
-
-- Python core scripts are accessible through two interfaces thanks to the [moulinette framework](https://github.com/YunoHost/moulinette):
- - [CLI](https://en.wikipedia.org/wiki/Command-line_interface) for `yunohost` command.
- - [API](https://en.wikipedia.org/wiki/Application_programming_interface) for [web administration module](https://github.com/YunoHost/yunohost-admin) (other modules could be implemented).
-- You can find more details about how YunoHost works on this [documentation (in French)](https://yunohost.org/#/package_list_fr).
-
-## Dependencies
-
-- [Python 2.7](https://www.python.org/download/releases/2.7)
-- [Moulinette](https://github.com/YunoHost/moulinette)
-- [Bash](https://www.gnu.org/software/bash/bash.html)
-- [Debian Stretch](https://www.debian.org/releases/stretch)
 
 ## License
 
-As [other components of YunoHost core code](https://yunohost.org/#/faq_en), this repository is licensed GNU AGPL v3.
+As [other components of YunoHost](https://yunohost.org/#/faq_en), this repository is licensed under GNU AGPL v3.

data/actionsmap/yunohost.yml

@@ -285,7 +285,7 @@ user:
                 ### user_permission_list()
                 list:
                     action_help: List permissions and corresponding accesses
-                    api: GET /users/permissions/<permission>
+                    api: GET /users/permissions
                     arguments:
                         -s:
                             full: --short
@@ -300,7 +300,7 @@ user:
                 ### user_permission_update()
                 update:
                     action_help: Manage group or user permissions
-                    api: POST /users/permissions/<permission>
+                    api: PUT /users/permissions/<permission>
                     arguments:
                         permission:
                             help: Permission to manage (e.g. mail or nextcloud or wordpress.editors)

data/helpers.d/setting

@@ -159,7 +159,7 @@ ynh_add_protected_uris() {
 ynh_app_setting()
 {
     ACTION="$1" APP="$2" KEY="$3" VALUE="${4:-}" python - <<EOF
-import os, yaml
+import os, yaml, sys
 app, action = os.environ['APP'], os.environ['ACTION'].lower()
 key, value = os.environ['KEY'], os.environ.get('VALUE', None)
 setting_file = "/etc/yunohost/apps/%s/settings.yml" % app
@@ -177,7 +177,7 @@ else:
         if key in ['redirected_urls', 'redirected_regex']:
             value = yaml.load(value)
         if key in ["unprotected_uris", "unprotected_regex", "protected_uris", "protected_regex"]:
-           logger.warning("/!\\ Packagers! This app is using the legacy permission system. Please delete these legacy settings and use the new helpers 'ynh_permission_{create,urls,update,delete}' and the 'visitors' group to manage public/private access.")
+           sys.stderr.write("/!\\ Packagers! This app is using the legacy permission system. Please delete these legacy settings and use the new helpers 'ynh_permission_{create,urls,update,delete}' and the 'visitors' group to manage public/private access.\n")
         settings[key] = value
     else:
         raise ValueError("action should either be get, set or delete")

locales/en.json

@@ -23,7 +23,7 @@
     "app_id_invalid": "Invalid app ID",
     "app_incompatible": "The app {app} is incompatible with your YunoHost version",
     "app_install_files_invalid": "These files cannot be installed",
-    "app_install_failed": "Could not install {app}",
+    "app_install_failed": "Could not install {app}: {error}",
     "app_install_script_failed": "An error occurred inside the app installation script",
     "app_location_already_used": "The app '{app}' is already installed in ({path})",
     "app_make_default_location_already_used": "Can't make the app '{app}' the default on the domain, '{domain}' is already in use by the other app '{other_app}'",
@@ -292,7 +292,7 @@
     "ldap_initialized": "LDAP initialized",
     "license_undefined": "undefined",
     "mail_alias_remove_failed": "Could not remove e-mail alias '{mail:s}'",
-    "mail_domain_unknown": "Unknown e-mail address for domain '{domain:s}'",
+    "mail_domain_unknown": "Invalid e-mail address for domain '{domain:s}'. Please, use a domain administrated by this server.",
     "mail_forward_remove_failed": "Could not remove e-mail forwarding '{mail:s}'",
     "mailbox_disabled": "E-mail turned off for user {user:s}",
     "mailbox_used_space_dovecot_down": "The Dovecot mailbox service needs to be up, if you want to fetch used mailbox space",

locales/fr.json

@@ -120,7 +120,7 @@
     "ldap_initialized": "L’annuaire LDAP initialisé",
     "license_undefined": "indéfinie",
     "mail_alias_remove_failed": "Impossible de supprimer l’alias de courriel '{mail:s}'",
-    "mail_domain_unknown": "Le domaine '{domain:s}' pour l'adresse de courriel est inconnu",
+    "mail_domain_unknown": "Le domaine '{domain:s}' de cette adress de courriel n'est pas valide. Merci d'utiliser un domain administré par ce serveur.",
     "mail_forward_remove_failed": "Impossible de supprimer le courriel de transfert '{mail:s}'",
     "maindomain_change_failed": "Impossible de modifier le domaine principal",
     "maindomain_changed": "Le domaine principal modifié",

locales/pt.json

@@ -72,7 +72,7 @@
     "ldap_initialized": "LDAP inicializada com êxito",
     "license_undefined": "indefinido",
     "mail_alias_remove_failed": "Não foi possível remover a etiqueta de correio '{mail:s}'",
-    "mail_domain_unknown": "Domínio de endereço de correio desconhecido '{domain:s}'",
+    "mail_domain_unknown": "Domínio de endereço de correio '{domain:s}' inválido. Por favor, usa um domínio administrado per esse servidor.",
     "mail_forward_remove_failed": "Não foi possível remover o reencaminhamento de correio '{mail:s}'",
     "maindomain_change_failed": "Incapaz alterar o domínio raiz",
     "maindomain_changed": "Domínio raiz alterado com êxito",

src/yunohost/app.py

@@ -994,19 +994,19 @@ def app_install(operation_logger, app, label=None, args=None, no_remove_on_failu
         install_failed = True if install_retcode != 0 else False
         if install_failed:
             error = m18n.n('app_install_script_failed')
-            logger.exception(error)
+            logger.exception(m18n.n("app_install_failed", app=app_id, error=error))
-            operation_logger.error(error)
+            failure_message_with_debug_instructions = operation_logger.error(error)
     # Script got manually interrupted ... N.B. : KeyboardInterrupt does not inherit from Exception
     except (KeyboardInterrupt, EOFError):
         error = m18n.n('operation_interrupted')
-        logger.exception(error)
+        logger.exception(m18n.n("app_install_failed", app=app_id, error=error))
-        operation_logger.error(error)
+        failure_message_with_debug_instructions = operation_logger.error(error)
     # Something wrong happened in Yunohost's code (most probably hook_exec)
-    except Exception as e :
+    except Exception as e:
         import traceback
         error = m18n.n('unexpected_error', error=u"\n" + traceback.format_exc())
-        logger.exception(error)
+        logger.exception(m18n.n("app_install_failed", app=app_id, error=error))
-        operation_logger.error(error)
+        failure_message_with_debug_instructions = operation_logger.error(error)
     finally:
         # Whatever happened (install success or failure) we check if it broke the system
         # and warn the user about it
@@ -1015,8 +1015,8 @@ def app_install(operation_logger, app, label=None, args=None, no_remove_on_failu
             _assert_system_is_sane_for_app(manifest, "post")
         except Exception as e:
             broke_the_system = True
-            logger.exception(str(e))
+            logger.exception(m18n.n("app_install_failed", app=app_id, error=str(e)))
-            operation_logger.error(str(e))
+            failure_message_with_debug_instructions = operation_logger.error(str(e))
 
         # If the install failed or broke the system, we remove it
         if install_failed or broke_the_system:
@@ -1055,7 +1055,7 @@ def app_install(operation_logger, app, label=None, args=None, no_remove_on_failu
             # Remove all permission in LDAP
             for permission_name in user_permission_list()["permissions"].keys():
                 if permission_name.startswith(app_instance_name+"."):
-                    permission_delete(permission_name, force=True)
+                    permission_delete(permission_name, force=True, sync_perm=False)
 
             if remove_retcode != 0:
                 msg = m18n.n('app_not_properly_removed',
@@ -1074,9 +1074,9 @@ def app_install(operation_logger, app, label=None, args=None, no_remove_on_failu
             shutil.rmtree(app_setting_path)
             shutil.rmtree(extracted_app_folder)
 
-            app_ssowatconf()
+            permission_sync_to_user()
 
-            raise YunohostError("app_install_failed", app=app_id)
+            raise YunohostError(failure_message_with_debug_instructions, raw_msg=True)
 
     # Clean hooks and add new ones
     hook_remove(app_instance_name)
@@ -1102,7 +1102,7 @@ def app_install(operation_logger, app, label=None, args=None, no_remove_on_failu
         permission_url(app_instance_name + ".main", url=None, sync_perm=False)
 
     # Migrate classic public app still using the legacy unprotected_uris
-    if app_settings.get("unprotected_uris", None) == "/":
+    if app_settings.get("unprotected_uris", None) == "/" or app_settings.get("skipped_uris", None) == "/":
         user_permission_update(app_instance_name + ".main", remove="all_users", add="visitors", sync_perm=False)
 
     permission_sync_to_user()

src/yunohost/data_migrations/0011_setup_group_permission.py

@@ -117,7 +117,7 @@ class MyMigration(Migration):
             app_setting(app, 'allowed_users', delete=True)
 
             # Migrate classic public app still using the legacy unprotected_uris
-            if app_setting(app, "unprotected_uris") == "/":
+            if app_setting(app, "unprotected_uris") == "/" or app_setting(app, "skipped_uris") == "/":
                 user_permission_update(app+".main", remove="all_users", add="visitors", sync_perm=False)
 
             permission_sync_to_user()

src/yunohost/hook.py

@@ -196,7 +196,7 @@ def hook_list(action, list_by='name', show_info=False):
         else:
             _append_folder(result, HOOK_FOLDER)
     except OSError:
-        logger.debug("system hook folder not found for action '%s' in %s",
+        logger.debug("No default hook for action '%s' in %s",
                      action, HOOK_FOLDER)
 
     try:
@@ -207,7 +207,7 @@ def hook_list(action, list_by='name', show_info=False):
         else:
             _append_folder(result, CUSTOM_HOOK_FOLDER)
     except OSError:
-        logger.debug("custom hook folder not found for action '%s' in %s",
+        logger.debug("No custom hook for action '%s' in %s",
                      action, CUSTOM_HOOK_FOLDER)
 
     return {'hooks': result}

src/yunohost/log.py

@@ -315,7 +315,8 @@ class RedactingFormatter(Formatter):
         try:
             # This matches stuff like db_pwd=the_secret or admin_password=other_secret
             # (the secret part being at least 3 chars to avoid catching some lines like just "db_pwd=")
-            match = re.search(r'(pwd|pass|password|secret|key|token)=(\S{3,})$', record.strip())
+            # For 'key', we require to at least have one word char [a-zA-Z0-9_] before it to avoid catching "--key" used in many helpers
+            match = re.search(r'(pwd|pass|password|secret|\wkey|token)=(\S{3,})$', record.strip())
             if match and match.group(2) not in self.data_to_redact:
                 self.data_to_redact.append(match.group(2))
         except Exception as e:

src/yunohost/permission.py

@@ -150,7 +150,7 @@ def user_permission_update(operation_logger, permission, add=None, remove=None,
 
     # Don't update LDAP if we update exactly the same values
     if set(new_allowed_groups) == set(current_allowed_groups):
-        logger.warning("permission_already_up_to_date")
+        logger.warning(m18n.n("permission_already_up_to_date"))
         return
 
     # Commit the new allowed group list