From 3755c5754a01cab4a7578c3f0df172848aff26fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=89mile?= Date: Thu, 28 Oct 2021 15:32:57 +0200 Subject: [PATCH 1/6] update readme for borg client configuration --- README.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 924cb84..4e1cc8e 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # Borg Backup for YunoHost -[![Integration level](https://dash.yunohost.org/integration/borg.svg)](https://dash.yunohost.org/appci/app/borg) ![](https://ci-apps.yunohost.org/ci/badges/borg.status.svg) ![](https://ci-apps.yunohost.org/ci/badges/borg.maintain.svg) +[![Integration level](https://dash.yunohost.org/integration/borg.svg)](https://dash.yunohost.org/appci/app/borg) ![](https://ci-apps.yunohost.org/ci/badges/borg.status.svg) ![](https://ci-apps.yunohost.org/ci/badges/borg.maintain.svg) [![Install Borg with YunoHost](https://install-app.yunohost.org/install-with-yunohost.svg)](https://install-app.yunohost.org/?app=borg) -A [Borg](https://borgbackup.readthedocs.io/en/stable/index.html#what-is-borgbackup) implementation to backup a YunoHost server. This is the Borg Backup App to be installed on a server to backup. It works together with a [Borg Server App](https://github.com/YunoHost-Apps/borgserver_ynh) installed on a host server. +A [Borg](https://borgbackup.readthedocs.io/en/stable/index.html#what-is-borgbackup) implementation to backup a YunoHost server. This is the Borg Backup App to be installed on a server to backup. It works together with a [Borg Server App](https://github.com/YunoHost-Apps/borgserver_ynh) installed on a host server. ## :warning: NB. : This doc is partially obsolete and should be reworked! :warning: @@ -22,15 +22,16 @@ You should received an email after the first backup succeeded. ### Set up Borg Backup App on guest Server A Firstly, set up the Borg Backup App (`borg`) on the guest Server A you want to backup: + ``` $ yunohost app install borg -Indicate the domain name of server B where to upload backups: host.serverb -Indicate the ssh user to use to connect on this server: servera -Indicate a strong passphrase, that you will keep preciously if you want to be able to use your backups: N0tAW3akp4ssw0rdYoloMacN!guets -Would you like to backup your YunoHost configuration ? [0 | 1] (default: 1): -Would you like to backup mails and user home directory ? [0 | 1] (default: 1): -Which apps would you backup (list separated by comma or 'all') ? (default: all): -Indicate the backup frequency (see systemd OnCalendar format) (default: Daily): +In which borg repository location do you want to backup your files ?: user@host.serverb:/remote/repository +Provide a strong passphrase to encrypt your backups. No blank space: +Should Borg backup your YunoHost configuration? [yes | no] (default: yes): +Should Borg backup emails and user home directory? [yes | no] (default: yes): +Which apps should Borg backup ? (default: all): +With which regular time schedule should the backups be performed? (see systemd OnCalendar format) (default: Daily): +Do you want admin to receive mail notifications on backups ? [always | errors_only | never]: never ``` #### Syntax to define a backup time schedule @@ -44,7 +45,7 @@ You can schedule regular backups at specific time. Only one regular time schedul * 5,17:00 : Every day at 5 AM and at 5 PM See here for more info : https://wiki.archlinux.org/index.php/Systemd/Timers#Realtime_timer -#### Information generated by Borg Backup +#### Information generated by Borg Backup At the end of the installation, the Borg Backup App (``borg``) displays the SSH public key and the SSH user to give to the person who has access to the host Server B and will set up Borg Server App. ``` You should now install the "Borg Server" app on host.serverb and fill questions like this: @@ -93,9 +94,9 @@ borg list ./::ARCHIVE_NAME | grep db.sql borg list ./::ARCHIVE_NAME | grep dump.sql ``` * Be sure to have your passphrase available even if your server is completely broken - + ## How to restore a complete system - + *For infos on restoring process, check [this yunohost forum thread](https://forum.yunohost.org/t/restoring-whole-yunohost-from-borg-backups/12705/3) and [that one](https://forum.yunohost.org/t/how-to-properly-backup-and-restore/12583/3), also [using Borg with sshkeys](https://thisiscasperslife.wordpress.com/2017/11/28/using-borg-backup-across-ssh-with-sshkeys/), the [`borg extract` documentation](https://borgbackup.readthedocs.io/en/stable/usage/extract.html), and this [general tutorial on Borg Backup](https://practical-admin.com/blog/backups-using-borg/).* In the following explanations: @@ -211,7 +212,7 @@ sudo yunohost backup restore auto_borg_XX_XX_XX_XX:XX --apps [Get the storage space used by the backup repository on the host server](https://borgbackup.readthedocs.io/en/stable/usage/info.html) ``borg info /home/servera/backup`` -### Backup YunoHost apps with different criticallity levels +### Backup YunoHost apps with different criticallity levels If you want to backup your guest server: * with different YunoHost apps From ef97208252a89095e999315faec7fee76246ab94 Mon Sep 17 00:00:00 2001 From: ljf Date: Tue, 30 Nov 2021 04:13:07 +0100 Subject: [PATCH 2/6] [enh] Add config panel --- conf/backup-with-borg | 8 +++++ config_panel.toml | 78 +++++++++++++++++++++++++++++++++++++++++++ scripts/config | 46 +++++++++++++++++++++++++ scripts/install | 5 +-- scripts/upgrade | 2 ++ 5 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 config_panel.toml create mode 100644 scripts/config diff --git a/conf/backup-with-borg b/conf/backup-with-borg index d5d70c2..33aaf1f 100644 --- a/conf/backup-with-borg +++ b/conf/backup-with-borg @@ -22,6 +22,8 @@ filter_hooks() { fail_if_partially_failed() { grep Skipped|Error } +sudo yunohost app setting ${borg_id} last_run -v "${current_date}" +sudo yunohost app setting ${borg_id} state -v "ongoing" # Backup system part conf conf=$(sudo yunohost app setting ${borg_id} conf) @@ -74,6 +76,12 @@ fi domain=$(hostname) repository="$(sudo yunohost app setting ${borg_id} repository)" mailalert="$(sudo yunohost app setting ${borg_id} mailalert)" +if [[ ! -z "$errors" ]]; then + sudo yunohost app setting ${borg_id} state -v "failed" +else + sudo yunohost app setting ${borg_id} state -v "successful" +fi + if [[ ! -z "$errors" && $mailalert != "never" ]]; then cat <(echo -e "$errors\n\n\n") "$log_file" "$err_file" | mail -s "[borg] Backup failed from $domain onto $repository" root exit 1 diff --git a/config_panel.toml b/config_panel.toml new file mode 100644 index 0000000..b42e708 --- /dev/null +++ b/config_panel.toml @@ -0,0 +1,78 @@ +version = "1.0" + +[main] +services = [] + + [main.settings] + name = "" + visible = "false" + + [main.settings.repository] + type = "string" + + [main.settings.ssh_public_key] + type = "text" + visible = "false" + bind = "/root/.ssh/id___APP___ed25519.pub" + + [main.settings.state] + type = "string" + visible = "false" + + [main.settings.last_run] + type = "string" + visible = "false" + + [main.general] + [main.general.info] + ask = """\ + Repository : `{repository}`\ + SSH public key : `{ssh_public_key}`\ + Backup state : {state} + Last run: {last_run} + """ + type = "alert" + style = "info" + + [main.general.on_calendar] + ask.en = "Frequency" + type = "string" + help = "With which regular time schedule should the backups be performed? (see systemd OnCalendar format)" + bind = "OnCalendar:/etc/systemd/system/__APP__.timer" + + [main.general.mailalert] + ask.en = "Mail alert" + type = "select" + choices.always = "Always" + choices.errors_only = "Only if an error occured" + choices.never = "Never alert me" + help = "Alerts are sent to the first user of this server" + + [main.content] + name = "What should be backuped ?" + optional = false + + [main.content.conf] + ask.en = "Configuration" + type = "boolean" + + [main.content.data] + ask.en = "Data" + type = "boolean" + + [main.content.apps] + ask.en = "Apps" + type = "tags" + help = "App list separated by comma. You can write 'all' to select all apps, even those installed after this borg app. You can also select all apps but some apps by writing 'exclude:' following by an app list separated by comma." + +[advanced] +name = "Advanced" +services = [] + [advanced.list] + name = "Last backups list" + [advanced.list.last_backups] + ask.en = """\ + {last_backups}\ + """ + type = "markdown" + diff --git a/scripts/config b/scripts/config new file mode 100644 index 0000000..5e93896 --- /dev/null +++ b/scripts/config @@ -0,0 +1,46 @@ +#!/bin/bash + +source /usr/share/yunohost/helpers + +ynh_abort_if_errors + +#================================================= +# SPECIFIC GETTERS FOR TOML SHORT KEY +#================================================= + +get__info() { + if [ "$state" == "failed" ]; then + cat << EOF +style: "danger" +EOF + elif [ "$state" == "successful" ]; then + cat << EOF +style: "success" +EOF + else + cat << EOF +style: "info" +EOF + fi +} + +get__last_backups() { + cat << EOF +value: + $(borg list --short --last 50 $repository) +EOF +} + +#================================================= +# SPECIFIC VALIDATORS FOR TOML SHORT KEYS +#================================================= +validate__on_calendar() { + + (! systemd-analyze calendar $on_calendar > /dev/null) && + echo 'Please follow systemd OnCalendar format: https://man.archlinux.org/man/systemd.time.7#CALENDAR_EVENTS' +} + +#================================================= +# GENERIC FINALIZATION +#================================================= +ynh_app_config_run $1 diff --git a/scripts/install b/scripts/install index 6aefffe..e87500e 100755 --- a/scripts/install +++ b/scripts/install @@ -21,7 +21,6 @@ ynh_abort_if_errors #================================================= export app=$YNH_APP_INSTANCE_NAME - # Retrieve arguments ynh_export repository passphrase on_calendar conf data apps mailalert @@ -36,7 +35,9 @@ if [[ $repository == *"@"* ]]; then fi ssh_user=$(echo "$repository" | cut -d"@" -f1 | cut -d"/" -f2) fi -ynh_save_args repository server passphrase on_calendar conf data apps mailalert +state="repository uncreated" +last_run="-" +ynh_save_args repository server passphrase on_calendar conf data apps mailalert state last_run #================================================= # INSTALL DEPENDENCIES diff --git a/scripts/upgrade b/scripts/upgrade index 6a59011..9c95d17 100755 --- a/scripts/upgrade +++ b/scripts/upgrade @@ -28,6 +28,8 @@ if [[ $mailalert != "always" && $mailalert != "errors_only" && $mailalert != "ne ynh_app_setting_set --app=$app --key="mailalert" --value="errors_only" export mailalert="errors_only" fi +ynh_app_setting_set --app=$app --key="state" --value="not run since last update" +ynh_app_setting_set --app=$app --key="last_run" --value="-" #================================================= # CHECK IF AN UPGRADE IS NEEDED From d8a95811e20aac4ff1fb84d1804fe1b72193893a Mon Sep 17 00:00:00 2001 From: ljf Date: Tue, 30 Nov 2021 14:00:36 +0100 Subject: [PATCH 3/6] [enh] Config panel to change repo address --- config_panel.toml | 22 ++++++++++------------ manifest.json | 4 ++-- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/config_panel.toml b/config_panel.toml index b42e708..7cdcfdf 100644 --- a/config_panel.toml +++ b/config_panel.toml @@ -7,33 +7,31 @@ services = [] name = "" visible = "false" - [main.settings.repository] - type = "string" - - [main.settings.ssh_public_key] - type = "text" - visible = "false" - bind = "/root/.ssh/id___APP___ed25519.pub" - [main.settings.state] type = "string" - visible = "false" [main.settings.last_run] type = "string" - visible = "false" [main.general] [main.general.info] ask = """\ - Repository : `{repository}`\ - SSH public key : `{ssh_public_key}`\ Backup state : {state} Last run: {last_run} """ type = "alert" style = "info" + [main.settings.repository] + ask.en = "Repository" + type = "string" + help = "Specify a local repository like /mount/my_external_harddrive/backups or a remote repository using this format: ssh://USER@DOMAIN.TLD:PORT/~/backup . If you plan to use borgserver_ynh app : 'USER' is *not* meant to be an existing user on the guest server, instead, it will be created *on the host server* during the installation of the Borg Server App. With borgserver_ynh apps you can't specify another repo path than ~/backup." + + [main.settings.ssh_public_key] + ask.en = "Public key: {ssh_public_key}" + type = "display_text" + bind = "/root/.ssh/id___APP___ed25519.pub" + [main.general.on_calendar] ask.en = "Frequency" type = "string" diff --git a/manifest.json b/manifest.json index 8b55114..a6e2b81 100644 --- a/manifest.json +++ b/manifest.json @@ -6,7 +6,7 @@ "en": "Backup your server on a host server using Borg.", "fr": "Sauvegardez votre serveur sur un serveur distant avec Borg." }, - "version": "1.1.16~ynh24", + "version": "1.1.16~ynh25", "url": "https://borgbackup.readthedocs.io", "license": "BSD-3-Clause", "maintainer": { @@ -15,7 +15,7 @@ "url": "https://reflexlibre.net" }, "requirements": { - "yunohost": ">= 4.1.0" + "yunohost": ">= 4.3.0" }, "multi_instance": true, "services": [], From 88c040b101e8c77828f3d475a32484daff3b1748 Mon Sep 17 00:00:00 2001 From: ljf Date: Tue, 30 Nov 2021 19:00:25 +0100 Subject: [PATCH 4/6] [fix] Bad yaml returned by config script --- config_panel.toml | 19 ++++++++++--------- manifest.json | 2 +- scripts/config | 42 +++++++++++++++++++++++++++++++++++++----- 3 files changed, 48 insertions(+), 15 deletions(-) diff --git a/config_panel.toml b/config_panel.toml index 7cdcfdf..b5b4d41 100644 --- a/config_panel.toml +++ b/config_panel.toml @@ -16,20 +16,20 @@ services = [] [main.general] [main.general.info] ask = """\ - Backup state : {state} - Last run: {last_run} + Backup state : {{state}} + Last run: {{last_run}} """ type = "alert" style = "info" - [main.settings.repository] + [main.general.repository] ask.en = "Repository" type = "string" help = "Specify a local repository like /mount/my_external_harddrive/backups or a remote repository using this format: ssh://USER@DOMAIN.TLD:PORT/~/backup . If you plan to use borgserver_ynh app : 'USER' is *not* meant to be an existing user on the guest server, instead, it will be created *on the host server* during the installation of the Borg Server App. With borgserver_ynh apps you can't specify another repo path than ~/backup." - [main.settings.ssh_public_key] + [main.general.ssh_public_key] ask.en = "Public key: {ssh_public_key}" - type = "display_text" + type = "markdown" bind = "/root/.ssh/id___APP___ed25519.pub" [main.general.on_calendar] @@ -58,6 +58,10 @@ services = [] ask.en = "Data" type = "boolean" + [main.content.data_multimedia] + ask.en = "Data multimedia" + type = "boolean" + [main.content.apps] ask.en = "Apps" type = "tags" @@ -69,8 +73,5 @@ services = [] [advanced.list] name = "Last backups list" [advanced.list.last_backups] - ask.en = """\ - {last_backups}\ - """ + ask.en = "" type = "markdown" - diff --git a/manifest.json b/manifest.json index a6e2b81..d0883ed 100644 --- a/manifest.json +++ b/manifest.json @@ -6,7 +6,7 @@ "en": "Backup your server on a host server using Borg.", "fr": "Sauvegardez votre serveur sur un serveur distant avec Borg." }, - "version": "1.1.16~ynh25", + "version": "1.1.16~ynh26", "url": "https://borgbackup.readthedocs.io", "license": "BSD-3-Clause", "maintainer": { diff --git a/scripts/config b/scripts/config index 5e93896..4f0b799 100644 --- a/scripts/config +++ b/scripts/config @@ -9,11 +9,16 @@ ynh_abort_if_errors #================================================= get__info() { - if [ "$state" == "failed" ]; then cat << EOF +ask: + en: "**Backup state**: ${old[state]} + + **Last run**: ${old[last_run]}" +EOF + if [ "${old[state]}" == "failed" ]; then style: "danger" EOF - elif [ "$state" == "successful" ]; then + elif [ "${old[state]}" == "successful" ]; then cat << EOF style: "success" EOF @@ -24,10 +29,24 @@ EOF fi } +get__ssh_public_key() { + cat << EOF +ask: + en: "**Public key**: \`$(cat /root/.ssh/id_${app}_ed25519.pub || echo '')\`" +EOF +} + +get__data_multimedia() { + if [ -e /home/yunohost.multimedia/.nobackup ]; then + echo "value: false" + else + echo "value: true" + fi +} get__last_backups() { cat << EOF -value: - $(borg list --short --last 50 $repository) +ask: |- +$(BORG_PASSPHRASE="$(yunohost app setting $app passphrase)" BORG_RSH="ssh -i /root/.ssh/id_${app}_ed25519 -oStrictHostKeyChecking=yes " borg list --short --last 50 ${old[repository]} | sed 's/^/ /g' 2> /dev/null) EOF } @@ -36,11 +55,24 @@ EOF #================================================= validate__on_calendar() { - (! systemd-analyze calendar $on_calendar > /dev/null) && + (systemd-analyze calendar $on_calendar > /dev/null) || echo 'Please follow systemd OnCalendar format: https://man.archlinux.org/man/systemd.time.7#CALENDAR_EVENTS' } +#================================================= +# SPECIFIC SETTERS FOR TOML SHORT KEYS +#================================================= + +set__data_multimedia() { + if [ "$data_multimedia" == "false" ]; then + mkdir -p /home/yunohost.multimedia/ + touch /home/yunohost.multimedia/.nobackup + else + ynh_secure_remove /home/yunohost.multimedia/.nobackup + fi +} #================================================= # GENERIC FINALIZATION #================================================= ynh_app_config_run $1 + From 7aa383d41c53faa2422601aacbd6103fe63e0d14 Mon Sep 17 00:00:00 2001 From: ljf Date: Tue, 30 Nov 2021 19:04:08 +0100 Subject: [PATCH 5/6] [enh] Avoid to rebuild borg at every upgrade --- scripts/_common.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/_common.sh b/scripts/_common.sh index c77a961..699abd3 100644 --- a/scripts/_common.sh +++ b/scripts/_common.sh @@ -12,7 +12,7 @@ pkg_dependencies="python3-pip python3-dev libacl1-dev libssl-dev liblz4-dev pyth # Install borg with pip if borg is not here install_borg_with_pip () { if [ -d /opt/borg-env ]; then - /opt/borg-env/bin/python /opt/borg-env/bin/pip list | grep "Version: $BORG_VERSION" || ynh_secure_remove /opt/borg-env + /opt/borg-env/bin/python /opt/borg-env/bin/pip list | grep "borgbackup *$BORG_VERSION" || ynh_secure_remove /opt/borg-env fi if [ ! -d /opt/borg-env ]; then python3 -m venv /opt/borg-env From de3c8d6b0a8e188e8473a904a866bb8fd5b7cb9d Mon Sep 17 00:00:00 2001 From: ljf Date: Tue, 30 Nov 2021 19:18:58 +0100 Subject: [PATCH 6/6] [fix] Exclude multimedia data --- scripts/config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/config b/scripts/config index 4f0b799..5ab0857 100644 --- a/scripts/config +++ b/scripts/config @@ -64,7 +64,7 @@ validate__on_calendar() { #================================================= set__data_multimedia() { - if [ "$data_multimedia" == "false" ]; then + if [ "$data_multimedia" == "0" ]; then mkdir -p /home/yunohost.multimedia/ touch /home/yunohost.multimedia/.nobackup else