YunoHost-Apps/borg_ynh
1
0
Fork 0
mirror of https://github.com/YunoHost-Apps/borg_ynh.git synced 2024-09-03 18:16:05 +02:00
Code Issues Activity Actions

Merge pull request #61 from YunoHost-Apps/testing

Browse source 
Testing
This commit is contained in:
Alexandre Aubin 2020-11-28 20:56:36 +01:00 • committed by GitHub
commit 8efa276680
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
4 changed files with 150 additions and 112 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

151
README.md
View file

@ -10,7 +10,8 @@
A [Borg](https://borgbackup.readthedocs.io/en/stable/index.html#what-is-borgbackup) implementation to backup a YunoHost server. This is the Borg Backup App to be installed on a server to backup. It works together with a [Borg Server App](https://github.com/YunoHost-Apps/borgserver_ynh) installed on a host server.
# How does it work? Set up Borg Apps
## How to backup your server with this app ?
You want to backup a critical "guest" Server A onto a remote "host" Server B, you need:
* Domain name of server B: ``host.serverb``
* Name of the server B SSH user (to be created by ``borgserver``) for connection from Server A: ``borgservera``
@ -19,11 +20,14 @@ You want to backup a critical "guest" Server A onto a remote "host" Server B, yo
* Regular time schedule for your backups, see below
* Install Borg Backup App (``borg``) on guest Server A
* Install Borg Server App (``borgserver``) on host Server B
* Save the passphrase in another place than your server. Without the passphrase, you won't be able to restore data.
## Set up Borg Backup App on guest Server A
You should received an email after the first backup succeeded.
### Set up Borg Backup App on guest Server A
Firstly, set up the Borg Backup App (``borg``) on the guest Server A you want to backup:
```
$ yunohost app install borg_ynh
$ yunohost app install borg
Indicate the domain name of server B where to upload backups: host.serverb
Indicate the ssh user to use to connect on this server: servera
Indicate a strong passphrase, that you will keep preciously if you want to be able to use your backups: N0tAW3akp4ssw0rdYoloMacN!guets
@ -33,7 +37,7 @@ Which apps would you backup (list separated by comma or 'all') ? (default: all):
Indicate the backup frequency (see systemd OnCalendar format) (default: Daily):
```
### Syntax to define a backup time schedule
#### Syntax to define a backup time schedule
You can schedule regular backups at specific time. Only one regular time schedule is possible for one ``borg`` instance, see below for workaround. Some examples:
* Monthly :
* Weekly :
@ -44,7 +48,7 @@ You can schedule regular backups at specific time. Only one regular time schedul
* 5,17:00 : Every day at 5 AM and at 5 PM
See here for more info : https://wiki.archlinux.org/index.php/Systemd/Timers#Realtime_timer
### Information generated by Borg Backup
#### Information generated by Borg Backup
At the end of the installation, the Borg Backup App (``borg``) displays the SSH public key and the SSH user to give to the person who has access to the host Server B and will set up Borg Server App.
```
You should now install the "Borg Server" app on host.serverb and fill questions like this:
@ -58,7 +62,7 @@ $ cat /root/.ssh/id_borg_ed25519.pub
ssh-ed25519 AAAA[...] root@guest.servera
```
## Set up Borg Server App on host Server B
### Set up Borg Server App on host Server B
Secondly, set up the Borg Server App (``borgserver``) on the host Server B that will store your backups:
```
$ yunohost app install borgserver
@ -67,32 +71,151 @@ Indicate the public key given by Borg Backup app (borg) setup: ssh-ed25519 AAAA[
Indicate the storage quota: 5G
```
# Test the Borg Apps setup
### Test the Borg Apps setup
At this step your backup should run at the scheduled time. Note that the first backup can take very long, as much data has to be copied through ssh. Following backups are incremental: only newly generated data since last backup will be copied.
If you want to test correct Borg Apps setup before scheduled time, you can start a backup manually on guest Server A:
```
$ service borg start
$ systemctl start borg
```
Next you can check presence of your backup repository on host Server B:
```
$ borg list /home/servera/backup
$ BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list servera@host.serverb:~/backup
```
You will need the passphrase to run ``borg`` commands on the backup repository created on the host Server B.
YOU SHOULD REGULARLY RESTORE YOUR BACKUPS TO TEST THEIR VALIDITY.
## Check regularly your backup
If you want to be sure to be able to restore your server, you should try to restore regularly the archives. But this process is quite time consumming.
# Usage and documentation
You should at least:
* Keep your apps up to date (if apps are too old, they could be difficult to restore on a more recent recent version)
* Check regularly the presence of info.json and db.sql or dump.sql in your apps archives
```
borg list ./::ARCHIVE_NAME | grep info.json
borg list ./::ARCHIVE_NAME | grep db.sql
borg list ./::ARCHIVE_NAME | grep dump.sql
```
* Be sure to have your passphrase available even if your server is completely broken
## How to restore a complete system
*For infos on restoring process, check [this yunohost forum thread](https://forum.yunohost.org/t/restoring-whole-yunohost-from-borg-backups/12705/3) and [that one](https://forum.yunohost.org/t/how-to-properly-backup-and-restore/12583/3), also [using borg with sshkeys](https://thisiscasperslife.wordpress.com/2017/11/28/using-borg-backup-across-ssh-with-sshkeys/), the [`borg extract` documentation](https://borgbackup.readthedocs.io/en/stable/usage/extract.html), and this [general tutorial on borg backup](https://practical-admin.com/blog/backups-using-borg/).*
## Edit the list of YunoHost apps to backup
In the following explanations:
- the server to backup/restore will be called: `yuno`
- the remote server that receives and store the back will be called: `rem`
- `rem` is accessible at the domain `rem.tld`
- the remote user on `rem` which owns the borg backups will be called `yurem`
- backup files will be stored in `rem` in the directory: `/home/yurem/backup`
### Overview
The idea here, if you need to restore a whole yunohost system is:
1. Install a new debian VM
2. Install yunohost in it the usual way
3. Go through yunohost postinstall (parameters you will supply are not crucial, as they will be replaced by the restore)
4. Install borg
5. Setup `rem` to accept ssh connections from `yuno`
6. Use borg to import backups from `rem` to `yuno`
7. Restore borg backups with the `yunohost backup restore` command, first config, then data, then each app one at a time
8. Remove the borg app and restore it
### Make it possible for `yuno` to connect to `rem` with borg
At this stage, we will assume that `yuno` is a freshly installed yunohost (based on buster in my case). You should also have performed the yunohost postinstall.
If you don't want to restore the whole system, just some apps, you can skip some of the steps below.
#### Install the borg yunohost app in `yuno`
The idea here is just to install borg, not in order to create backups, but only to use borg commands to import remote backups.
So for example, you can install it doing the following:
```bash
sudo yunohost app install borg -a "server=rem.tld&ssh_user=yurem&conf=0&data=0&apps=hextris&on_calendar=2:30"
```
#### Make sure that `rem` accepts ssh connections from `yuno`
In `yuno` you will need to get the ssh key that borg just created while installing: `sudo cat /root/.ssh/id_borg_ed25519.pub`, copy it to clipboard.
Connect via ssh to `rem`, go to `/home/yurem/.ssh/authorized_keys`, and past the borg public key you got at previous step.
Now to make sure this worked, you can try to ssh from `yuno` to `rem`.
In `yuno` : `ssh -i /root/.ssh/id_borg_ed25519 yurem@rem.tld` . If you can get into `rem` , without it prompting for a password, then you're good to continue :)
### Restore backups to `yuno`
⚠️ For the commands in the following section to work, you will need to be root in `yuno` (you can become root running `sudo su`).
⚠️ Restoration of backups can take quite a while, you'd better do them in a separate process, so that it doesn't stop if your terminal session gets closed. For this, you can for example use [tmux](https://www.howtogeek.com/671422/how-to-use-tmux-on-linux-and-why-its-better-than-screen/).
In `yuno` now, you should be able to list backups in `rem` with the following command:
```bash
SRV=yurem@rem.tld:/home/yurem/backup
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list $SRV
```
You can then reimport one to `yuno` with:
```bash
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar $SRV::auto_BACKUP_NAME /home/yunohost.backup/archives/auto_BACKUP_NAME.tar.gz
```
And then restore the archive in `yuno` with:
```bash
yunohost backup restore auto_BACKUP_NAME --system # for config and data backups
yunohost backup restore auto_BACKUP_NAME --apps # for other backups (=apps)
```
### And nextcloud? It's super heavy!!
For nextcloud, the best is probably to reimport the backup without the data. And to import the data manually.
For that, you can do the following (as root):
```bash
SRV=yurem@rem.tld:/home/yurem/backup
# export the app without data
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar -e apps/nextcloud/backup/home/yunohost.app $SRV::auto_nextcloud_XX_XX_XX_XX:XX /home/yunohost.backup/archives/auto_nextcloud_XX_XX_XX_XX:XX.tar.gz
# extract the data from the backup to the nextcloud folder
cd /home/yunohost.app/nextcloud
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg extract $SRV::auto_nextcloud_XX_XX_XX_XX:XX apps/nextcloud/backup/home/yunohost.app/nextcloud/
mv apps/nextcloud/backup/home/yunohost.app/nextcloud/data data
rm -r apps
# now you can simply restore nextcloud app
yunohost backup restore auto_nextcloud_XX_XX_XX_XX:XX --apps
```
### Restore borg
Once you've restored the whole system, you will probably want to restore the borg app as well.
For that, remove the "dummy" borg you installed to do the restoration, and restore borg the same ways as for other apps:
```bash
sudo yunohost app remove borg
sudo yunohost backup restore auto_borg_XX_XX_XX_XX:XX --apps
```
## Tips
### Edit the list of YunoHost apps to backup
``yunohost app setting borg apps -v "nextcloud,wordpress"``
## Other usefull borg commands
### Other usefull borg commands
[Get the storage space used by the backup repository on the host server](https://borgbackup.readthedocs.io/en/stable/usage/info.html)
``borg info /home/servera/backup``
## Backup Yunohost apps with different criticallity levels
### Backup Yunohost apps with different criticallity levels
If you want to backup your guest server:
* with different YunoHost apps

17
conf/backup_method.j2
View file

@ -14,6 +14,11 @@ do_need_mount() {
true
}
LOGFILE=/var/log/backup_borg.err
log_with_timestamp() {
sed -e "s/^/[$(date +"%Y-%m-%d_%H:%M:%S")] /" >> $LOGFILE
}
do_backup() {
export BORG_PASSPHRASE
@ -23,12 +28,10 @@ do_backup() {
repo=$3
size=$4
description=$5
LOGFILE=/var/log/backup_borg.log
ERRFILE=/var/log/backup_borg.err
current_date=$(date +"%d_%m_%y_%H:%M")
pushd $work_dir
set +e
if borg init -e repokey $repo >> $LOGFILE 2>> $ERRFILE; then
if borg init -e repokey $repo 2>&1 >/dev/null | log_with_timestamp; then
#human_size=`echo $size | awk '{ suffix=" KMGT"; for(i=1; $1>1024 && i < length(suffix); i++) $1/=1024; print int($1) substr(suffix, i, 1), $3; }'`
# Speed in Kbps
#speed=1000
@ -41,10 +44,10 @@ This is an automated message from your beloved YunoHost server." | /usr/bin/mail
fi
set -e
borg create $repo::${name}_${current_date} ./ >> $LOGFILE 2>> $ERRFILE
borg create $repo::${name}_${current_date} ./ 2>&1 >/dev/null | log_with_timestamp
popd
borg prune $repo -P ${name} --keep-hourly 2 --keep-daily=7 --keep-weekly=8 --keep-monthly=12 >> $LOGFILE 2>> $ERRFILE
borg prune $repo -P ${name} --keep-hourly 2 --keep-daily=7 --keep-weekly=8 --keep-monthly=12 2>&1 >/dev/null | log_with_timestamp
}
do_mount() {
@ -55,9 +58,7 @@ do_mount() {
repo=$3
size=$4
description=$5
LOGFILE=/var/log/backup_borg.log
ERRFILE=/var/log/backup_borg.err
borg mount $repo::$name $work_dir >> $LOGFILE 2>> $ERRFILE
borg mount $repo::$name $work_dir 2>&1 >/dev/null | log_with_timestamp
}
work_dir=$2

4
manifest.json
View file

@ -6,7 +6,7 @@
"en": "Backup your server on a host server using Borg.",
"fr": "Sauvegardez votre serveur sur un serveur distant avec Borg."
},
"version": "1.1.10~ynh6",
"version": "1.1.13~ynh1",
"url": "https://borgbackup.readthedocs.io",
"license": "BSD-3-Clause",
"maintainer": {
@ -15,7 +15,7 @@
"url": "https://reflexlibre.net"
},
"requirements": {
"yunohost": ">= 2.7.14"
"yunohost": ">= 3.5.0"
},
"multi_instance": true,
"services": [],

90
scripts/_common.sh
View file

@ -12,14 +12,13 @@ pkg_dependencies="python3-pip python3-dev libacl1-dev libssl-dev liblz4-dev pyth
install_borg_with_pip () {
if [ ! -d /opt/borg-env ]; then
virtualenv --python=python3 /opt/borg-env
/opt/borg-env/bin/python /opt/borg-env/bin/pip install borgbackup==1.1.10
/opt/borg-env/bin/python /opt/borg-env/bin/pip install borgbackup==1.1.13
echo "#!/bin/bash
/opt/borg-env/bin/python /opt/borg-env/bin/borg \"\$@\"" > /usr/local/bin/borg
chmod u+x /usr/local/bin/borg
fi
}
#=================================================
# COMMON HELPERS
#=================================================
@ -48,22 +47,7 @@ ynh_save_args () {
done
}
# Render templates with Jinja2
#
# Attention : Variables should be exported before calling this helper to be
# accessible inside templates.
#
# usage: ynh_render_template some_template output_path
# | arg: some_template - Template file to be rendered
# | arg: output_path - The path where the output will be redirected to
ynh_render_template() {
local template_path=$1
local output_path=$2
# Taken from https://stackoverflow.com/a/35009576
python3 -c 'import os, sys, jinja2; sys.stdout.write(
jinja2.Template(sys.stdin.read()
).render(os.environ));' < $template_path > $output_path
}
ynh_configure () {
ynh_backup_if_checksum_is_different $2
@ -71,22 +55,7 @@ ynh_configure () {
ynh_store_file_checksum $2
}
# Remove any logs for all the following commands.
#
# usage: ynh_print_OFF
# WARNING: You should be careful with this helper, and never forgot to use ynh_print_ON as soon as possible to restore the logging.
ynh_print_OFF () {
set +x
}
# Restore the logging after ynh_print_OFF
#
# usage: ynh_print_ON
ynh_print_ON () {
set -x
# Print an echo only for the log, to be able to know that ynh_print_ON has been called.
echo ynh_print_ON > /dev/null
}
# Send an email to inform the administrator
#
@ -145,63 +114,8 @@ $(yunohost tools diagnosis | grep -B 100 "services:" | sed '/services:/d')"
echo "$mail_message" | $mail_bin -a "Content-Type: text/plain; charset=UTF-8" -s "$mail_subject" "$recipients"
}
# Read the value of a key in a ynh manifest file
#
# usage: ynh_read_manifest manifest key
# | arg: manifest - Path of the manifest to read
# | arg: key - Name of the key to find
ynh_read_manifest () {
manifest="$1"
key="$2"
python3 -c "import sys, json;print(json.load(open('$manifest', encoding='utf-8'))['$key'])"
}
# Checks the app version to upgrade with the existing app version and returns:
# - UPGRADE_APP if the upstream app version has changed
# - UPGRADE_PACKAGE if only the YunoHost package has changed
#
## It stops the current script without error if the package is up-to-date
#
# This helper should be used to avoid an upgrade of an app, or the upstream part
# of it, when it's not needed
#
# To force an upgrade, even if the package is up to date,
# you have to set the variable YNH_FORCE_UPGRADE before.
# example: sudo YNH_FORCE_UPGRADE=1 yunohost app upgrade MyApp
# usage: ynh_check_app_version_changed
ynh_check_app_version_changed () {
local force_upgrade=${YNH_FORCE_UPGRADE:-0}
local package_check=${PACKAGE_CHECK_EXEC:-0}
# By default, upstream app version has changed
local return_value="UPGRADE_APP"
local current_version=$(ynh_read_manifest "/etc/yunohost/apps/$YNH_APP_INSTANCE_NAME/manifest.json" "version" || echo 1.0)
local current_upstream_version="${current_version/~ynh*/}"
local update_version=$(ynh_read_manifest "../manifest.json" "version" || echo 1.0)
local update_upstream_version="${update_version/~ynh*/}"
if [ "$current_version" == "$update_version" ] ; then
# Complete versions are the same
if [ "$force_upgrade" != "0" ]
then
echo "Upgrade forced by YNH_FORCE_UPGRADE." >&2
unset YNH_FORCE_UPGRADE
elif [ "$package_check" != "0" ]
then
echo "Upgrade forced for package check." >&2
else
ynh_die "Up-to-date, nothing to do" 0
fi
elif [ "$current_upstream_version" == "$update_upstream_version" ] ; then
# Upstream versions are the same, only YunoHost package versions differ
return_value="UPGRADE_PACKAGE"
fi
echo $return_value
}
ynh_debian_release () {
lsb_release --codename --short
}