diff --git a/README.md b/README.md index f8a380d..a8579e7 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,30 @@ -# Borg Backup App for YunoHost +# Borg Backup for YunoHost -[![Latest Version](https://img.shields.io/badge/version-1.0.3-green.svg?style=flat)](https://github.com/YunoHost-Apps/borg_ynh/releases) +[![Latest Version](https://img.shields.io/badge/version-1.1.15-green.svg?style=flat)](https://github.com/YunoHost-Apps/borg_ynh/releases) [![Status](https://img.shields.io/badge/status-testing-yellow.svg?style=flat)](https://github.com/YunoHost-Apps/borg_ynh/milestones) [![Integration level](https://dash.yunohost.org/integration/borg.svg)](https://dash.yunohost.org/appci/app/borg) -[![GitHub license](https://img.shields.io/badge/license-GPLv3-blue.svg?style=flat)](https://raw.githubusercontent.com/YunoHost-Apps/borg_ynh/master/LICENSE) -[![GitHub issues](https://img.shields.io/github/issues/YunoHost-Apps/borg_ynh.svg?style=flat)](https://github.com/YunoHost-Apps/borg_ynh/issues) - -[![Install Borg with YunoHost](https://install-app.yunohost.org/install-with-yunohost.png)](https://install-app.yunohost.org/?app=borg) +[![GitHub license](https://img.shields.io/badge/license-AGPLv3-blue.svg?style=flat)](https://raw.githubusercontent.com/YunoHost-Apps/borg_ynh/master/LICENSE) +[![GitHub issues](https://img.shields.io/github/issues/YunoHost-Apps/borg_ynh.svg?style=flat)](https://github.com/YunoHost-Apps/borg_ynh/issues) +[![Install Borg with YunoHost](https://install-app.yunohost.org/install-with-yunohost.svg)](https://install-app.yunohost.org/?app=borg) A [Borg](https://borgbackup.readthedocs.io/en/stable/index.html#what-is-borgbackup) implementation to backup a YunoHost server. This is the Borg Backup App to be installed on a server to backup. It works together with a [Borg Server App](https://github.com/YunoHost-Apps/borgserver_ynh) installed on a host server. -## How to backup your server with this app ? +## How to backup your server with this app? You want to backup a critical "guest" Server A onto a remote "host" Server B, you need: -* Domain name of server B: ``host.serverb`` -* Name of the server B SSH user (to be created by ``borgserver``) for connection from Server A: ``borgservera`` +* Domain name of server B: `host.serverb` +* Name of the server B SSH user (to be created by `borgserver`) for connection from Server A: `borgservera` * **Strong passphrase** to encrypt your backups on host Server B. And to **restore your backups**!! * IDs of YunoHost apps you want to backup * Regular time schedule for your backups, see below -* Install Borg Backup App (``borg``) on guest Server A -* Install Borg Server App (``borgserver``) on host Server B +* Install Borg Backup App (`borg`) on guest Server A +* Install Borg Server App (`borgserver`) on host Server B * Save the passphrase in another place than your server. Without the passphrase, you won't be able to restore data. You should received an email after the first backup succeeded. ### Set up Borg Backup App on guest Server A -Firstly, set up the Borg Backup App (``borg``) on the guest Server A you want to backup: +Firstly, set up the Borg Backup App (`borg`) on the guest Server A you want to backup: ``` $ yunohost app install borg Indicate the domain name of server B where to upload backups: host.serverb @@ -38,7 +37,7 @@ Indicate the backup frequency (see systemd OnCalendar format) (default: Daily): ``` #### Syntax to define a backup time schedule -You can schedule regular backups at specific time. Only one regular time schedule is possible for one ``borg`` instance, see below for workaround. Some examples: +You can schedule regular backups at specific time. Only one regular time schedule is possible for one `borg` instance, see below for workaround. Some examples: * Monthly : * Weekly : * Daily : Daily at midnight @@ -56,7 +55,7 @@ User: servera Public key: ssh-ed25519 AAAA[...] root@guest.servera ``` This information is also sent by email to the admin of guest Server A. -If you don't find the mail and you don't see the message in the log bar you can find the SSH public key with this command: +If you don't find the email and you don't see the message in the log bar you can find the SSH public key with this command: ``` $ cat /root/.ssh/id_borg_ed25519.pub ssh-ed25519 AAAA[...] root@guest.servera @@ -90,7 +89,7 @@ If you want to be sure to be able to restore your server, you should try to rest You should at least: * Keep your apps up to date (if apps are too old, they could be difficult to restore on a more recent recent version) - * Check regularly the presence of info.json and db.sql or dump.sql in your apps archives + * Check regularly the presence of `info.json` and `db.sql` or `dump.sql` in your apps archives ``` borg list ./::ARCHIVE_NAME | grep info.json borg list ./::ARCHIVE_NAME | grep db.sql @@ -100,13 +99,13 @@ borg list ./::ARCHIVE_NAME | grep dump.sql ## How to restore a complete system -*For infos on restoring process, check [this yunohost forum thread](https://forum.yunohost.org/t/restoring-whole-yunohost-from-borg-backups/12705/3) and [that one](https://forum.yunohost.org/t/how-to-properly-backup-and-restore/12583/3), also [using borg with sshkeys](https://thisiscasperslife.wordpress.com/2017/11/28/using-borg-backup-across-ssh-with-sshkeys/), the [`borg extract` documentation](https://borgbackup.readthedocs.io/en/stable/usage/extract.html), and this [general tutorial on borg backup](https://practical-admin.com/blog/backups-using-borg/).* +*For infos on restoring process, check [this yunohost forum thread](https://forum.yunohost.org/t/restoring-whole-yunohost-from-borg-backups/12705/3) and [that one](https://forum.yunohost.org/t/how-to-properly-backup-and-restore/12583/3), also [using Borg with sshkeys](https://thisiscasperslife.wordpress.com/2017/11/28/using-borg-backup-across-ssh-with-sshkeys/), the [`borg extract` documentation](https://borgbackup.readthedocs.io/en/stable/usage/extract.html), and this [general tutorial on Borg Backup](https://practical-admin.com/blog/backups-using-borg/).* In the following explanations: - the server to backup/restore will be called: `yuno` - the remote server that receives and store the back will be called: `rem` - `rem` is accessible at the domain `rem.tld` -- the remote user on `rem` which owns the borg backups will be called `yurem` +- the remote user on `rem` which owns the Borg backups will be called `yurem` - backup files will be stored in `rem` in the directory: `/home/yurem/backup` @@ -114,24 +113,24 @@ In the following explanations: The idea here, if you need to restore a whole yunohost system is: -1. Install a new debian VM -2. Install yunohost in it the usual way -3. Go through yunohost postinstall (parameters you will supply are not crucial, as they will be replaced by the restore) -4. Install borg +1. Install a new Debian VM +2. Install YunoHost in it the usual way +3. Go through YunoHost postinstall (parameters you will supply are not crucial, as they will be replaced by the restore) +4. Install Borg 5. Setup `rem` to accept ssh connections from `yuno` -6. Use borg to import backups from `rem` to `yuno` -7. Restore borg backups with the `yunohost backup restore` command, first config, then data, then each app one at a time -8. Remove the borg app and restore it +6. Use Borg to import backups from `rem` to `yuno` +7. Restore Borg backups with the `yunohost backup restore` command, first config, then data, then each app one at a time +8. Remove the Borg app and restore it -### Make it possible for `yuno` to connect to `rem` with borg +### Make it possible for `yuno` to connect to `rem` with Borg -At this stage, we will assume that `yuno` is a freshly installed yunohost (based on buster in my case). You should also have performed the yunohost postinstall. +At this stage, we will assume that `yuno` is a freshly installed YunoHost (based on Buster in my case). You should also have performed the YunoHost postinstall. If you don't want to restore the whole system, just some apps, you can skip some of the steps below. -#### Install the borg yunohost app in `yuno` +#### Install the Borg YunoHost app in `yuno` -The idea here is just to install borg, not in order to create backups, but only to use borg commands to import remote backups. +The idea here is just to install Borg, not in order to create backups, but only to use Borg commands to import remote backups. So for example, you can install it doing the following: ```bash @@ -142,9 +141,9 @@ sudo yunohost app install borg -a "server=rem.tld&ssh_user=yurem&conf=0&data=0&a In `yuno` you will need to get the ssh key that borg just created while installing: `sudo cat /root/.ssh/id_borg_ed25519.pub`, copy it to clipboard. -Connect via ssh to `rem`, go to `/home/yurem/.ssh/authorized_keys`, and past the borg public key you got at previous step. +Connect via ssh to `rem`, go to `/home/yurem/.ssh/authorized_keys`, and past the Borg public key you got at previous step. -Now to make sure this worked, you can try to ssh from `yuno` to `rem`. +Now to make sure this worked, you can try to SSH from `yuno` to `rem`. In `yuno` : `ssh -i /root/.ssh/id_borg_ed25519 yurem@rem.tld` . If you can get into `rem` , without it prompting for a password, then you're good to continue :) ### Restore backups to `yuno` @@ -173,9 +172,9 @@ yunohost backup restore auto_BACKUP_NAME --system # for config and data backups yunohost backup restore auto_BACKUP_NAME --apps # for other backups (=apps) ``` -### And nextcloud? It's super heavy!! +### And Nextcloud? It's super heavy!! -For nextcloud, the best is probably to reimport the backup without the data. And to import the data manually. +For Nextcloud, the best is probably to reimport the backup without the data. And to import the data manually. For that, you can do the following (as root): @@ -195,11 +194,11 @@ rm -r apps yunohost backup restore auto_nextcloud_XX_XX_XX_XX:XX --apps ``` -### Restore borg +### Restore Borg -Once you've restored the whole system, you will probably want to restore the borg app as well. +Once you've restored the whole system, you will probably want to restore the Borg app as well. -For that, remove the "dummy" borg you installed to do the restoration, and restore borg the same ways as for other apps: +For that, remove the "dummy" Borg you installed to do the restoration, and restore Borg the same ways as for other apps: ```bash sudo yunohost app remove borg @@ -215,7 +214,7 @@ sudo yunohost backup restore auto_borg_XX_XX_XX_XX:XX --apps [Get the storage space used by the backup repository on the host server](https://borgbackup.readthedocs.io/en/stable/usage/info.html) ``borg info /home/servera/backup`` -### Backup Yunohost apps with different criticallity levels +### Backup YunoHost apps with different criticallity levels If you want to backup your guest server: * with different YunoHost apps diff --git a/check_process b/check_process index 84a56da..5fa3e9b 100644 --- a/check_process +++ b/check_process @@ -1,11 +1,10 @@ ;; Test complet ; Manifest - server="domain.tld:22" - ssh_user="sam" + repository="ssh://sam@domain.tld:22/~/backup" passphrase="APassphrase" conf=1 data=1 - app="all" + apps="all" on_calendar="Daily" ; Checks pkg_linter=1 @@ -15,22 +14,15 @@ setup_private=0 setup_public=0 upgrade=1 + upgrade=1 from_commit=c1524dd8e37cc671c01f5b5363901dd6a01e6fc1 backup_restore=1 multi_instance=1 - incorrect_path=0 port_already_use=0 change_url=0 -;;; Levels - Level 1=auto - Level 2=auto - Level 3=auto - Level 4=na - Level 5=auto - Level 6=auto - Level 7=auto - Level 8=0 - Level 9=0 - Level 10=0 ;;; Options Email=ljf+borg_ynh@reflexlibre.net -Notification=down +Notificatio +;;; Upgrade options + ; commit=c1524dd8e37cc671c01f5b5363901dd6a01e6fc1 + name=Merge pull request #63 from YunoHost-Apps/testing + manifest_arg=server=domain.tld:22&ssh_user=package_checker&passphrase=APassphrase&conf=1&data=1&apps=all&on_calendar=Daily diff --git a/conf/backup-with-borg b/conf/backup-with-borg new file mode 100644 index 0000000..f153033 --- /dev/null +++ b/conf/backup-with-borg @@ -0,0 +1,81 @@ +#!/bin/bash + +# We don't stop the script on errors cause we want to backup all data we could backuped +#set -eu + +borg_id=$1 +errors="" +current_date=$(date +"%y%m%d_%H%M") +log_file="/var/log/${borg_id}/${current_date}.log" +err_file="/var/log/${borg_id}/${current_date}.err" +mkdir -p "/var/log/${borg_id}" +if [ -z "$borg_id" ] +then + echo "This script expects a borg app id as first argument" >&2 + exit 1 +fi + +filter_hooks() { + sudo ls /usr/share/yunohost/hooks/backup/ /etc/yunohost/hooks.d/backup/ | grep "\-$1_" | cut -d"-" -f2 | uniq 2>> $err_file +} + +fail_if_partially_failed() { + grep Skipped|Error +} + +# Backup system part conf +conf=$(sudo yunohost app setting ${borg_id} conf) +if [[ "$conf" = "1" ]] +then + if ! sudo yunohost backup create -n auto_conf --method ${borg_id}_app --system $(filter_hooks conf) 2>> $err_file >> $log_file ; then + errors+="\nThe backup miserably failed to backup system configurations." + fi +fi + +# Backup system data +data=$(sudo yunohost app setting ${borg_id} data) +if [[ "$data" = "1" ]] +then + if ! sudo yunohost backup create -n auto_data --method ${borg_id}_app --system $(filter_hooks data) 2>> $err_file >> $log_file ; then + errors+="\nThe backup miserably failed to backup system data." + fi +fi + +# Backup all apps independently +apps=$(sudo yunohost app setting ${borg_id} apps | tr -d ' ') +for application in $(sudo ls /etc/yunohost/apps/); do + + if ( [[ "$apps" =~ ^exclude: ]] && grep -wq "$application" <<< "$apps" ) || + ( [[ "$apps" != "all" ]] && [[ ! "$apps" =~ ^exclude: ]] && ! grep -wq "$application" <<< "$apps" ); + then + continue + fi + + if sudo test ! -f /etc/yunohost/apps/$application/scripts/backup ; then + errors+="\nWarning: The application $application has no backup script. This app won't be backuped." + continue + fi + + if ! sudo yunohost backup create -n auto_$application --method ${borg_id}_app --apps $application 2>> $err_file >> $log_file ; then + errors+="\nThe backup miserably failed to backup $application application." + fi +done + +#========================================================= +# SEND MAIL TO NOTIFY SUCCED OR FAILED OPERATIONS +#========================================================= + +partial_errors="$(cat $log_file | grep -E "Error|Skipped")" +if [ ! -z "$partial_errors" ]; then + errors+="\nSome backup partially failed:\n$partial_errors" +fi + +# Send mail on backup (partially) failed +domain=$(hostname) +if [ ! -z "$errors" ]; then + cat <(echo -e "$errors\n\n\n") "$log_file" "$err_file" | mail -s "[borg] Backup failed from $domain onto $repo" root + exit 1 +else + cat $log_file | mail -s "[borg] Backup succeed from $domain onto $repo" root + exit 0 +fi diff --git a/conf/backup-with-borg.j2 b/conf/backup-with-borg.j2 deleted file mode 100644 index 68df3b1..0000000 --- a/conf/backup-with-borg.j2 +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -# Adapt this script to different API 2.x vs 3.x -if yunohost -v | grep "version: 2." > /dev/null; then - ignore_apps="--ignore-apps" - ignore_system="--ignore-system" -else - ignore_apps="" - ignore_system="" -fi - -filter_hooks() { - ls /usr/share/yunohost/hooks/backup/ /etc/yunohost/hooks.d/backup/ | grep "\-$1_" | cut -d"-" -f2 | uniq -} - -# Backup system part conf -conf=$(yunohost app setting {{ app }} conf) -if [ $conf -eq 1 ] -then - yunohost backup create $ignore_apps -n auto_conf --method {{ app }}_app --system $(filter_hooks conf) -fi - -# Backup system data -data=$(yunohost app setting {{ app }} data) -if [ $data -eq 1 ] -then - yunohost backup create $ignore_apps -n auto_data --method {{ app }}_app --system $(filter_hooks data) -fi - -# Backup all apps independently -apps=$(yunohost app setting {{ app }} apps) -for app in $(ls /etc/yunohost/apps/*/scripts/backup | cut -d / -f 5); do - backup_app=false - if [[ "$apps" = "all" ]]; then - backup_app=true - else - for selected_app in $(echo $apps | tr "," " ");do - if [[ "$selected_app" == "$app" ]]; then - backup_app=true - break - fi - done - fi - if [ "$backup_app" == "true" ];then - yunohost backup create $ignore_system -n auto_$app --method {{ app }}_app --apps $app - fi -done diff --git a/conf/backup_method b/conf/backup_method new file mode 100644 index 0000000..0b3c1ed --- /dev/null +++ b/conf/backup_method @@ -0,0 +1,90 @@ +#!/bin/bash + +set -e +app="${0#"./05-"}" +app="${app%"_app"}" + +BORG_PASSPHRASE="$(yunohost app setting $app passphrase)" +repo="$(yunohost app setting $app repository)" #$4 +if ssh-keygen -F "__SERVER__" >/dev/null ; then + BORG_RSH="ssh -i /root/.ssh/id_${app}_ed25519 -oStrictHostKeyChecking=yes " +else + BORG_RSH="ssh -i /root/.ssh/id_${app}_ed25519 -oStrictHostKeyChecking=no " +fi + +do_need_mount() { + true +} + +LOGFILE=/var/log/backup_borg.err +log_with_timestamp() { + sed -e "s/^/[$(date +"%Y-%m-%d_%H:%M:%S")] /" >> $LOGFILE +} + +do_backup() { + + export BORG_PASSPHRASE + export BORG_RSH + work_dir="$1" + name="$2" + repo="$3" + size="$4" + description="$5" + current_date=$(date +"%d_%m_%y_%H:%M") + pushd "$work_dir" + set +e + if borg init -e repokey "$repo" ; then + #human_size=`echo $size | awk '{ suffix=" KMGT"; for(i=1; $1>1024 && i < length(suffix); i++) $1/=1024; print int($1) substr(suffix, i, 1), $3; }'` + # Speed in Kbps + #speed=1000 + #evaluated_time=$(($size / ($speed * 1000 / 8) / 3600)) + echo "Hello, + +Your first backup on $repo is starting. + +This is an automated message from your beloved YunoHost server." | /usr/bin/mail.mailutils -a "Content-Type: text/plain; charset=UTF-8" -s "[YNH] First backup is starting" "root" + fi + set -e + + borg create "$repo::${name}_${current_date}" ./ 2>&1 >/dev/null | log_with_timestamp + popd + + # About thi _20 it's a crazy fix to avoid pruning wordpress__2 + # if you prune wordpress + borg prune "$repo" -P ${name}_20 --keep-hourly 2 --keep-daily=7 --keep-weekly=8 --keep-monthly=12 2>&1 >/dev/null | log_with_timestamp +} + +do_mount() { + export BORG_PASSPHRASE + export BORG_RSH + work_dir="$1" + name="$2" + repo="$3" + size="$4" + description="$5" + borg mount "$repo::$name" "$work_dir" 2>&1 >/dev/null | log_with_timestamp +} + +work_dir="$2" +name="$3" + +size="$5" +description="$6" + +case "$1" in + need_mount) + do_need_mount "$work_dir" "$name" "$repo" "$size" "$description" + ;; + backup) + do_backup "$work_dir" "$name" "$repo" "$size" "$description" + ;; + mount) + do_mount + ;; + *) + echo "hook called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +exit 0 diff --git a/conf/backup_method.j2 b/conf/backup_method.j2 deleted file mode 100644 index 35ed6a1..0000000 --- a/conf/backup_method.j2 +++ /dev/null @@ -1,86 +0,0 @@ -#!/bin/bash - -set -e - -BORG_PASSPHRASE="{{ passphrase }}" -if ssh-keygen -F "{{ server }}" >/dev/null ; then - BORG_RSH="ssh -i /root/.ssh/id_{{ app }}_ed25519 -oStrictHostKeyChecking=yes " -else - BORG_RSH="ssh -i /root/.ssh/id_{{ app }}_ed25519 -oStrictHostKeyChecking=no " -fi -repo=ssh://{{ ssh_user }}@{{ server }}/~/backup #$4 - -do_need_mount() { - true -} - -LOGFILE=/var/log/backup_borg.err -log_with_timestamp() { - sed -e "s/^/[$(date +"%Y-%m-%d_%H:%M:%S")] /" >> $LOGFILE -} - -do_backup() { - - export BORG_PASSPHRASE - export BORG_RSH - work_dir=$1 - name=$2 - repo=$3 - size=$4 - description=$5 - current_date=$(date +"%d_%m_%y_%H:%M") - pushd $work_dir - set +e - if borg init -e repokey $repo ; then - #human_size=`echo $size | awk '{ suffix=" KMGT"; for(i=1; $1>1024 && i < length(suffix); i++) $1/=1024; print int($1) substr(suffix, i, 1), $3; }'` - # Speed in Kbps - #speed=1000 - #evaluated_time=$(($size / ($speed * 1000 / 8) / 3600)) - echo "Hello, - -Your first backup on $repo is starting. - -This is an automated message from your beloved YunoHost server." | /usr/bin/mail.mailutils -a "Content-Type: text/plain; charset=UTF-8" -s "[YNH] First backup is starting" "root" - fi - set -e - - borg create $repo::${name}_${current_date} ./ 2>&1 >/dev/null | log_with_timestamp - popd - - borg prune $repo -P ${name} --keep-hourly 2 --keep-daily=7 --keep-weekly=8 --keep-monthly=12 2>&1 >/dev/null | log_with_timestamp -} - -do_mount() { - export BORG_PASSPHRASE - export BORG_RSH - work_dir=$1 - name=$2 - repo=$3 - size=$4 - description=$5 - borg mount $repo::$name $work_dir 2>&1 >/dev/null | log_with_timestamp -} - -work_dir=$2 -name=$3 - -size=$5 -description=$6 - -case "$1" in - need_mount) - do_need_mount $work_dir $name $repo $size $description - ;; - backup) - do_backup $work_dir $name $repo $size $description - ;; - mount) - do_mount - ;; - *) - echo "hook called with unknown argument \`$1'" >&2 - exit 1 - ;; -esac - -exit 0 diff --git a/conf/sudoer b/conf/sudoer new file mode 100644 index 0000000..6a11357 --- /dev/null +++ b/conf/sudoer @@ -0,0 +1 @@ +__APP__ ALL=(root) NOPASSWD: /usr/local/bin/backup-with-__APP__, /usr/local/bin/borg, /usr/bin/yunohost backup create *, /usr/bin/yunohost app setting *, /bin/ls, /usr/bin/test diff --git a/conf/systemd.service b/conf/systemd.service index 19866e4..b2d0080 100644 --- a/conf/systemd.service +++ b/conf/systemd.service @@ -4,9 +4,9 @@ After=network.target [Service] Type=oneshot -ExecStart=/usr/local/bin/backup-with-__APP__ -User=root -Group=root +ExecStart=/usr/bin/sudo /usr/local/bin/backup-with-__APP__ __APP__ +User=__APP__ +Group=__APP__ -[Install] -WantedBy=multi-user.target +#[Install] +#WantedBy=multi-user.target diff --git a/conf/systemd.timer b/conf/systemd.timer new file mode 100644 index 0000000..6434e4e --- /dev/null +++ b/conf/systemd.timer @@ -0,0 +1,8 @@ +[Unit] +Description=Run backup __APP__ regularly + +[Timer] +OnCalendar=__ON_CALENDAR__ + +[Install] +WantedBy=timers.target diff --git a/conf/systemd.timer.j2 b/conf/systemd.timer.j2 deleted file mode 100644 index f21a951..0000000 --- a/conf/systemd.timer.j2 +++ /dev/null @@ -1,8 +0,0 @@ -[Unit] -Description=Run backup {{ app }} regularly - -[Timer] -OnCalendar={{ on_calendar }} - -[Install] -WantedBy=timers.target diff --git a/manifest.json b/manifest.json index af09fb3..5744207 100644 --- a/manifest.json +++ b/manifest.json @@ -1,12 +1,12 @@ { - "name": "Borg Backup App", + "name": "Borg Backup", "id": "borg", "packaging_format": 1, "description": { "en": "Backup your server on a host server using Borg.", "fr": "Sauvegardez votre serveur sur un serveur distant avec Borg." }, - "version": "1.1.13~ynh2", + "version": "1.1.16~ynh16", "url": "https://borgbackup.readthedocs.io", "license": "BSD-3-Clause", "maintainer": { @@ -15,35 +15,23 @@ "url": "https://reflexlibre.net" }, "requirements": { - "yunohost": ">= 3.5.0" + "yunohost": ">= 3.8.1" }, "multi_instance": true, "services": [], "arguments": { "install" : [ { - "name": "server", + "name": "repository", + "type": "string", "ask": { - "en": "What is the domain name of the host server where backups will be sent?", - "fr": "Quel est le nom de domaine du serveur distant sur lequel seront envoyés les sauvegardes ?" + "en": "In which borg repository location do you want to backup your files ?", + "fr": "Dans quel repo borg souhaitez-vous sauvegarder vos fichiers ?" }, "help":{ - "en": "If this host server does use a custom SSH port (different from 22), you can specify it with DOMAIN:PORT", - "fr": "Si le serveur distant utilise un port particulier pour SSH (différent de 22), vous pouvez le spécifier avec DOMAIN:PORT" + "en": "Specify a remote repository using this format: ssh://USER@DOMAIN.TLD:PORT/~/backup . 'USER' is *not* meant to be an existing user on the guest server. Instead, it will be created *on the host server* during the installation of the Borg Server App. It's also possible to use a local repository using a syntax such as /mount/my_external_harddrive/backups" }, - "example": "host.serverb:22" - }, - { - "name": "ssh_user", - "ask": { - "en": "Which SSH username should be used to connect to the host server?", - "fr": "Quel utilisateur faut-il utiliser pour se connecter au serveur distant ?" - }, - "help":{ - "en": "It is not meant to be an existing user on this guest server. Instead, it will be created *on the host server* during the installation of the Borg Server App.", - "fr": "Cet utilisateur n'est pas censé exister sur ce serveur. Il sera créé *sur le serveur distant* lors de l'installation de l'App Borg Server sur celui-ci." - }, - "example": "borgservera" + "example": "ssh://john@serverb.tld:22/~/backup" }, { "name": "passphrase", @@ -53,7 +41,7 @@ "fr": "Indiquez une phrase de passe forte pour chiffrer vos sauvegardes. Sans espaces" }, "help":{ - "en": "Keep it safe! ...if you want to be able to restore. Do not communicate it to host Server holder or anyone else. ", + "en": "Keep it safe! ...if you want to be able to restore. Do not communicate it to host Server holder or anyone else.", "fr": "Gardez-la précieusement! ...si vous voulez pouvoir restaurer. Ne donnez pas la clé au possesseur du Serveur distant, ni personne." } }, @@ -61,7 +49,7 @@ "name": "conf", "type": "boolean", "ask": { - "en": "Should Borg backup your YunoHost configuration ?", + "en": "Should Borg backup your YunoHost configuration?", "fr": "Borg doit-il sauvegarder la configuration système YunoHost ?" }, "default": true @@ -70,23 +58,29 @@ "name": "data", "type": "boolean", "ask": { - "en": "Should Borg backup emails and user home directory ?", + "en": "Should Borg backup emails and user home directory?", "fr": "Borg doit-elle sauvegarder les mails et les répertoires des utilisateurs ?" }, "default": true }, { "name": "apps", + "type": "string", "ask": { - "en": "Which apps should Borg backup (list separated by comma or 'all') ?", - "fr": "Quelles applications doivent être sauvegardées par Borg (liste séparée par virgule ou 'all' ?" + "en": "Which apps should Borg backup ?", + "fr": "Quelles applications doivent être sauvegardées par Borg ?" }, + "help":{ + "en": "App list separated by comma. You can write 'all' to select all apps, even those installed after this borg app. You can also select all apps but some apps by writing 'exclude:' following by an app list separated by comma.", + "fr": "Liste d'applications séparées par des virgules. Vous pouvez écrire 'all' pour sélectionner toutes les apps, même celles installées après cette application borg. Vous pouvez aussi sélectionner toutes les apps sauf certaines en écrivant 'exclude:' suivi d'une liste d'applications séparées par des virgules." + }, "default": "all" }, { "name": "on_calendar", + "type": "string", "ask": { - "en": "With which regular time schedule should the backups be performed ? (see systemd OnCalendar format)", + "en": "With which regular time schedule should the backups be performed? (see systemd OnCalendar format)", "fr": "À quelle fréquence les sauvegardes doivent-elles être effectuées ? (voir le format OnCalendar de systemd)" }, "example": "Monthly or Weekly or Daily or Hourly or 4:00 or 5,17:00 or Sat --1..7 18:00:00", diff --git a/scripts/_common.sh b/scripts/_common.sh index 59992d6..90945f3 100644 --- a/scripts/_common.sh +++ b/scripts/_common.sh @@ -5,18 +5,25 @@ #================================================= # App package root directory should be the parent folder PKG_DIR=$(cd ../; pwd) +BORG_VERSION=1.1.16 -pkg_dependencies="python3-pip python3-dev libacl1-dev libssl-dev liblz4-dev python3-jinja2 python3-setuptools python-virtualenv virtualenv" +pkg_dependencies="python3-pip python3-dev libacl1-dev libssl-dev liblz4-dev python3-jinja2 python3-setuptools python3-venv python-virtualenv virtualenv libfuse-dev pkg-config" # Install borg with pip if borg is not here install_borg_with_pip () { + if [ -d /opt/borg-env ]; then + /opt/borg-env/bin/python /opt/borg-env/bin/pip list | grep "Version: $BORG_VERSION" || ynh_secure_remove /opt/borg-env + fi if [ ! -d /opt/borg-env ]; then - virtualenv --python=python3 /opt/borg-env - /opt/borg-env/bin/python /opt/borg-env/bin/pip install borgbackup==1.1.13 + python3 -m venv /opt/borg-env + /opt/borg-env/bin/python /opt/borg-env/bin/pip install wheel + /opt/borg-env/bin/python /opt/borg-env/bin/pip install borgbackup[fuse]==$BORG_VERSION echo "#!/bin/bash /opt/borg-env/bin/python /opt/borg-env/bin/borg \"\$@\"" > /usr/local/bin/borg - chmod u+x /usr/local/bin/borg + touch "/opt/borg-env/$(ynh_get_debian_release)" fi + # We need this to be executable by other borg apps + chmod a+x /usr/local/bin/borg } #================================================= @@ -43,19 +50,20 @@ ynh_save_args () { if [ "$var" == "path_url" ]; then setting_var="path" fi - ynh_app_setting_set $app $setting_var ${!var} + ynh_app_setting_set $app $setting_var "${!var}" done } +# Need also the helper https://github.com/YunoHost-Apps/Experimental_helpers/blob/master/ynh_handle_getopts_args/ynh_handle_getopts_args - -ynh_configure () { - ynh_backup_if_checksum_is_different $2 - ynh_render_template "${PKG_DIR}/conf/$1.j2" "$2" - ynh_store_file_checksum $2 -} - - +# Send an email to inform the administrator +# +# usage: ynh_send_readme_to_admin app_message [recipients] +# | arg: -m --app_message= - The message to send to the administrator. +# | arg: -r, --recipients= - The recipients of this email. Use spaces to separate multiples recipients. - default: root +# example: "root admin@domain" +# If you give the name of a YunoHost user, ynh_send_readme_to_admin will find its email adress for you +# example: "root admin@domain user1 user2" # Send an email to inform the administrator # @@ -113,36 +121,3 @@ $(yunohost tools diagnosis | grep -B 100 "services:" | sed '/services:/d')" # Send the email to the recipients echo "$mail_message" | $mail_bin -a "Content-Type: text/plain; charset=UTF-8" -s "$mail_subject" "$recipients" } - - - -ynh_debian_release () { - lsb_release --codename --short -} - -is_buster () { - if [ "$(ynh_debian_release)" == "buster" ] - then - return 0 - else - return 1 - fi -} - -is_stretch () { - if [ "$(ynh_debian_release)" == "stretch" ] - then - return 0 - else - return 1 - fi -} - -is_jessie () { - if [ "$(ynh_debian_release)" == "jessie" ] - then - return 0 - else - return 1 - fi -} diff --git a/scripts/backup b/scripts/backup index 6775113..61e6920 100755 --- a/scripts/backup +++ b/scripts/backup @@ -18,13 +18,25 @@ ynh_abort_if_errors #================================================= # LOAD SETTINGS #================================================= +ynh_print_info --message="Loading installation settings..." app=$YNH_APP_INSTANCE_NAME +#================================================= +# DECLARE DATA AND CONF FILES TO BACKUP +#================================================= +ynh_print_info --message="Declaring files to be backed up..." ynh_backup "/usr/local/bin/backup-with-$app" +ynh_backup "/etc/sudoers.d/$app" ynh_backup "/etc/systemd/system/$app.service" ynh_backup "/etc/systemd/system/$app.timer" ynh_backup "/etc/yunohost/hooks.d/backup_method/05-${app}_app" ynh_backup "/root/.ssh/id_${app}_ed25519" ynh_backup "/root/.ssh/id_${app}_ed25519.pub" + +#================================================= +# END OF SCRIPT +#================================================= + +ynh_print_info --message="Backup script completed for $app. (YunoHost will then actually copy those files to the archive)." diff --git a/scripts/install b/scripts/install index b80bca6..72ad751 100755 --- a/scripts/install +++ b/scripts/install @@ -19,66 +19,89 @@ ynh_abort_if_errors #================================================= # RETRIEVE ARGUMENTS FROM THE MANIFEST #================================================= + export app=$YNH_APP_INSTANCE_NAME # Retrieve arguments -ynh_export server ssh_user passphrase on_calendar conf data apps +ynh_export repository passphrase on_calendar conf data apps #================================================= # STORE SETTINGS FROM MANIFEST #================================================= -ynh_save_args server ssh_user passphrase on_calendar conf data apps +server="" +if [[ $repository == *"@"* ]]; then + server=$(echo "$repository" | cut -d"@" -f2 | cut -d"/" -f1) + if [[ $server == *":"* ]]; then + server="[$(echo "$server" | cut -d":" -f1)]:$(echo "$server" | cut -d":" -f2)" + fi + ssh_user=$(echo "$repository" | cut -d"@" -f1 | cut -d"/" -f2) +fi +ynh_save_args repository server passphrase on_calendar conf data apps #================================================= # INSTALL DEPENDENCIES #================================================= +ynh_script_progression --message="Installing dependencies..." + ynh_install_app_dependencies $pkg_dependencies install_borg_with_pip +#================================================= +# CREATE DEDICATED USER +#================================================= +ynh_script_progression --message="Configuring system user..." --weight=1 + +# Create a system user +ynh_system_user_create --username=$app + +#================================================= +# SPECIFIC SETUP #================================================= # ACTIVATE BACKUP METHODS #================================================= + +mkdir -p /etc/yunohost/hooks.d/backup mkdir -p /etc/yunohost/hooks.d/backup_method mkdir -p /usr/share/yunohost/backup_method +mkdir -p /var/log/${app} +chown -R $app:$app /var/log/${app} +chmod u+w /var/log/${app} #================================================= # SETUP THE BACKUP METHOD #================================================= -ynh_configure backup_method "/etc/yunohost/hooks.d/backup_method/05-${app}_app" +ynh_add_config --template="backup_method" --destination="/etc/yunohost/hooks.d/backup_method/05-${app}_app" +chmod go=--- "/etc/yunohost/hooks.d/backup_method/05-${app}_app" -#================================================= -# CONFIGURE CRON -#================================================= -ynh_configure backup-with-borg "/usr/local/bin/backup-with-$app" + +ynh_add_config --template="backup-with-borg" --destination="/usr/local/bin/backup-with-$app" chmod u+x "/usr/local/bin/backup-with-$app" -ynh_add_systemd_config -ynh_configure systemd.timer "/etc/systemd/system/$app.timer" -systemctl enable $app.timer -systemctl start $app.timer -#yunohost service add $app.timer -#yunohost service enable $app.timer -#yunohost service start $app.timer +chown $app:$app "/usr/local/bin/backup-with-$app" -#================================================= -# GENERATE SSH KEY -#================================================= -private_key="/root/.ssh/id_${app}_ed25519" -test -f $private_key || ssh-keygen -q -t ed25519 -N "" -f $private_key +ynh_add_config --template="sudoer" --destination="/etc/sudoers.d/$app" -#================================================= -# Display key -#================================================= +if [ ! -z "$server" ]; then + #================================================= + # GENERATE SSH KEY + #================================================= -echo "You should now install the \"Borg Server\" app on $server and fill questions like this: -User: ${ssh_user} -Public key: $(cat ${private_key}.pub)" + private_key="/root/.ssh/id_${app}_ed25519" + test -f $private_key || ssh-keygen -q -t ed25519 -N "" -f $private_key + #================================================= + # Display key + #================================================= -#================================================= -# SEND A README FOR THE ADMIN -#================================================= -ynh_print_OFF -message="You should now install the \"Borg Server\" app on $server and fill questions like this: + echo "You should now install the \"Borg Server\" app on $server and with the following credentials: + User: ${ssh_user} + Public key: $(cat ${private_key}.pub)" + + #================================================= + # SEND A README FOR THE ADMIN + #================================================= + + ynh_print_OFF + message="You should now install the \"Borg Server\" app on $server and with the following credentials: User: ${ssh_user} Public key: $(cat ${private_key}.pub) @@ -88,6 +111,35 @@ yunohost app install https://github.com/YunoHost-Apps/borgserver_ynh -a \"ssh_us If you facing an issue or want to improve this app, please open a new issue in this project: https://github.com/YunoHost-Apps/borg_ynh" -ynh_send_readme_to_admin "$message" "root" -ynh_print_ON + ynh_send_readme_to_admin "$message" + ynh_print_ON +fi +#================================================= +# SETUP SYSTEMD +#================================================= +ynh_script_progression --message="Configuring a systemd service..." --weight=1 + +# Create a dedicated systemd config +ynh_add_systemd_config + +#================================================= +# CONFIGURE SYSTEMD TIMER +#================================================= +ynh_add_config --template="systemd.timer" --destination="/etc/systemd/system/$app.timer" +systemctl enable $app.timer --quiet +systemctl start $app.timer + +#================================================= +# INTEGRATE SERVICE IN YUNOHOST +#================================================= +ynh_script_progression --message="Integrating service in YunoHost..." --weight=1 + +yunohost service add $app --description="Deduplicating backup program" + + +#================================================= +# END OF SCRIPT +#================================================= + +ynh_script_progression --message="Installation of $app completed" --last diff --git a/scripts/remove b/scripts/remove index 3463708..e3da1d5 100755 --- a/scripts/remove +++ b/scripts/remove @@ -15,17 +15,42 @@ source /usr/share/yunohost/helpers app=$YNH_APP_INSTANCE_NAME +#================================================= +# REMOVE SERVICE INTEGRATION IN YUNOHOST +#================================================= + +# Remove the service from the list of services known by YunoHost (added from `yunohost service add`) +if ynh_exec_warn_less yunohost service status $app >/dev/null +then + ynh_script_progression --message="Removing $app service integration..." + yunohost service remove $app +fi + #================================================= # REMOVE DEPENDENCIES #================================================= +ynh_script_progression --message="Removing dependencies..." + ynh_remove_app_dependencies #================================================= # REMOVE FILES #================================================= systemctl stop $app.timer -systemctl disable $app.timer +systemctl disable $app.timer --quiet ynh_remove_systemd_config ynh_secure_remove "/etc/systemd/system/$app.timer" ynh_secure_remove "/usr/local/bin/backup-with-$app" ynh_secure_remove "/etc/yunohost/hooks.d/backup_method/05-${app}_app" + +# Remove borg if we are removing the last borg app on the system +if [ "$(yunohost app list | grep "id: borg" | wc -l)" == "1" ] ; then + ynh_secure_remove "/opt/borg-env" + ynh_secure_remove "/usr/local/bin/borg" +fi + +#================================================= +# END OF SCRIPT +#================================================= + +ynh_script_progression --message="Removal of $app completed" --last diff --git a/scripts/restore b/scripts/restore index df9443e..6a70a59 100755 --- a/scripts/restore +++ b/scripts/restore @@ -19,41 +19,66 @@ ynh_abort_if_errors #================================================= # LOAD SETTINGS #================================================= +ynh_script_progression --message="Loading installation settings..." app=$YNH_APP_INSTANCE_NAME -server=$(ynh_app_setting_get $app server) -ssh_user=$(ynh_app_setting_get $app ssh_user) - #================================================= # INSTALL DEPENDENCIES #================================================= +ynh_script_progression --message="Reinstalling dependencies..." + ynh_install_app_dependencies $pkg_dependencies install_borg_with_pip +#================================================= +# CREATE DEDICATED USER +#================================================= +ynh_script_progression --message="Configuring system user..." --weight=1 + +# Create a system user +ynh_system_user_create --username=$app + #================================================= # ACTIVATE BACKUP METHODS #================================================= + mkdir -p /etc/yunohost/hooks.d/backup_method mkdir -p /usr/share/yunohost/backup_method +mkdir -p /etc/yunohost/hooks.d/backup +mkdir -p /var/log/${app} +chown -R $app:$app /var/log/${app} +chmod u+w /var/log/${app} #================================================= # RESTORE FILES #================================================= + ynh_restore +chmod go=--- "/etc/yunohost/hooks.d/backup_method/05-${app}_app" +chmod u+x "/usr/local/bin/backup-with-$app" +chown $app:$app "/usr/local/bin/backup-with-$app" #================================================= # ADVERTISE SERVICE IN ADMIN PANEL #================================================= -yunohost service add $app -systemctl enable $app.timer +systemctl enable $app.timer --quiet systemctl start $app.timer -#yunohost service add $app.timer -#yunohost service enable $app.timer -#yunohost service start $app.timer #================================================= # RESTORE SYSTEMD #================================================= -systemctl enable $app.service +systemctl enable $app.service --quiet +#================================================= +# INTEGRATE SERVICE IN YUNOHOST +#================================================= +ynh_script_progression --message="Integrating service in YunoHost..." --weight=1 + +yunohost service add $app --description="Deduplicating backup program" + +#================================================= +# END OF SCRIPT +#================================================= + +ynh_script_progression --message="Restoration completed for $app" --last diff --git a/scripts/upgrade b/scripts/upgrade index 139e4ee..8e0c8a1 100755 --- a/scripts/upgrade +++ b/scripts/upgrade @@ -15,21 +15,14 @@ source /usr/share/yunohost/helpers export app=$YNH_APP_INSTANCE_NAME -export server=$(ynh_app_setting_get $app server) -export ssh_user=$(ynh_app_setting_get $app ssh_user) -export passphrase=$(ynh_app_setting_get $app passphrase) -export on_calendar=$(ynh_app_setting_get $app on_calendar) -export conf=$(ynh_app_setting_get $app conf) -export data=$(ynh_app_setting_get $app data) -export apps=$(ynh_app_setting_get $app apps) - -export server=$(ynh_app_setting_get $app server) -export ssh_user=$(ynh_app_setting_get $app ssh_user) -export passphrase=$(ynh_app_setting_get $app passphrase) -export on_calendar=$(ynh_app_setting_get $app on_calendar) -export conf=$(ynh_app_setting_get $app conf) -export data=$(ynh_app_setting_get $app data) -export apps=$(ynh_app_setting_get $app apps) +export repository="$(ynh_app_setting_get $app repository)" +export server="$(ynh_app_setting_get $app server)" +export ssh_user="$(ynh_app_setting_get $app ssh_user)" +export passphrase="$(ynh_app_setting_get $app passphrase)" +export on_calendar="$(ynh_app_setting_get $app on_calendar)" +export conf="$(ynh_app_setting_get $app conf)" +export data="$(ynh_app_setting_get $app data)" +export apps="$(ynh_app_setting_get $app apps)" #================================================= # CHECK IF AN UPGRADE IS NEEDED @@ -39,6 +32,7 @@ ynh_check_app_version_changed #================================================= # BACKUP BEFORE UPGRADE THEN ACTIVE TRAP #================================================= +ynh_script_progression --message="Backing up the app before upgrading (may take a while)..." # Backup the current version of the app ynh_backup_before_upgrade @@ -46,38 +40,111 @@ ynh_clean_setup () { # restore it if the upgrade fails ynh_restore_upgradebackup } + # Exit if an error occurs during the execution of the script ynh_abort_if_errors +#================================================= +# ENSURE DOWNWARD COMPATIBILITY +#================================================= +ynh_script_progression --message="Ensuring downward compatibility..." --weight=1 + +if [ -z "$repository" ]; then + repository="ssh://$ssh_user@$server/~/backup" + server=$(echo "$repository" | cut -d"@" -f2 | cut -d"/" -f1) + if [[ $server == *":"* ]]; then + server="[$(echo "$server" | cut -d":" -f1)]:$(echo "$server" | cut -d":" -f2)" + fi + ynh_app_setting_set $app repository "$repository" + ynh_app_setting_set $app server "$server" + ynh_app_setting_delete $app ssh_user +fi if grep "borg.timer" /etc/yunohost/services.yml > /dev/null ; then yunohost service remove $app.timer - systemctl enable $app.timer + systemctl enable $app.timer --quiet systemctl start $app.timer fi # Replace backports with pip -if is_buster; then - if [ ! -f /opt/borg-env/buster ] ; then - rm -f /etc/apt/sources.list.d/$app-stretch-backports.list +[ ! -e " /etc/apt/sources.list.d/$app-stretch-backports.list" ] || rm -f /etc/apt/sources.list.d/$app-stretch-backports.list + +# Reinstall borg if debian change of major version +if [ ! -f "/opt/borg-env/$(ynh_get_debian_release)" ] ; then ynh_secure_remove /opt/borg-env - install_borg_with_pip - touch /opt/borg-env/buster - fi - fi +#================================================= +# UPGRADE DEPENDENCIES +#================================================= +ynh_script_progression --message="Upgrading dependencies..." --weight=1 + +ynh_install_app_dependencies $pkg_dependencies + +#================================================= +# CREATE DEDICATED USER +#================================================= +ynh_script_progression --message="Configuring system user..." --weight=1 + +# Create a system user +ynh_system_user_create --username=$app + + +#================================================= +# SPECIFIC UPGRADE +#================================================= +# Upgrade borgbackup +#================================================= +ynh_script_progression --message="Upgrading borgbackup..." --weight=1 + +install_borg_with_pip + #================================================= # SETUP THE BACKUP METHOD #================================================= -ynh_configure backup_method "/etc/yunohost/hooks.d/backup_method/05-${app}_app" +ynh_script_progression --message="Setting up backup method..." --weight=1 +ynh_add_config --template="backup_method" --destination="/etc/yunohost/hooks.d/backup_method/05-${app}_app" +chmod go=--- "/etc/yunohost/hooks.d/backup_method/05-${app}_app" + +ynh_add_config --template="backup-with-borg" --destination="/usr/local/bin/backup-with-$app" +chmod u+x "/usr/local/bin/backup-with-$app" +chown $app:$app "/usr/local/bin/backup-with-$app" + +ynh_add_config --template="sudoer" --destination="/etc/sudoers.d/$app" #================================================= -# CONFIGURE CRON +# SETUP SYSTEMD #================================================= -ynh_configure backup-with-borg "/usr/local/bin/backup-with-$app" -chmod u+x "/usr/local/bin/backup-with-$app" +ynh_script_progression --message="Upgrading systemd configuration..." --weight=1 + +# Create a dedicated systemd config ynh_add_systemd_config -ynh_configure systemd.timer "/etc/systemd/system/$app.timer" -systemctl enable $app.timer + +#================================================= +# CONFIGURE SYSTEMD TIMER +#================================================= +ynh_add_config --template="systemd.timer" --destination="/etc/systemd/system/$app.timer" +systemctl enable $app.timer --quiet systemctl start $app.timer + +mkdir -p /etc/yunohost/hooks.d/backup +mkdir -p /var/log/${app} +chown -R $app:$app /var/log/${app} +chmod u+w /var/log/${app} + +#================================================= +# GENERIC FINALIZATION +#================================================= + +#================================================= +# INTEGRATE SERVICE IN YUNOHOST +#================================================= +ynh_script_progression --message="Integrating service in YunoHost..." --weight=1 + +yunohost service add $app --description="Deduplicating backup program" + +#================================================= +# END OF SCRIPT +#================================================= + +ynh_script_progression --message="Upgrade of $app completed" --last