YunoHost/ynh-dev
1
0
Fork 0
mirror of https://github.com/YunoHost/ynh-dev.git synced 2024-09-03 20:05:59 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #54 from YunoHost/lxd

Browse source 
Vagrant is madness, LXD is much easy to setup and use
This commit is contained in:
Alexandre Aubin 2019-10-14 17:21:15 +02:00 • committed by GitHub
commit 4eecc1059b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
4 changed files with 124 additions and 228 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

125
README.md
View file

@ -53,21 +53,11 @@ your capacities and resources when aiming to setup a development environment.
Yunohost can be developed on using a combination of the following technologies: Yunohost can be developed on using a combination of the following technologies:
* Git (any version is sufficient) * Git (any version is sufficient)
* Vagrant (>= 2.x) * LXD (>= 2.x) (though only tested with 3.x for now)
* LXC (>= 3.x)
* Vagrant-LXC (>= 1.4.x)
LXC is typically lightweight but you may find the initial setup complex Because LXC are containers, they are typically lightweight and quick to start and stop.
(in particular network configuration). Alternatively, you may be able But you may find the initial setup complex (in particular network configuration).
to setup a local environnement using Virtualbox which is kinda more LXD makes managing an LXC ecosystem much simpler.
resource-hungry :
* Virtualbox (>= 6.x)
* Vagrant-virtualbox (>= ?.?.?)
Please keep in mind that these versions may not be available on your OS
distribution and you may be required to install them as binary or from source.
There are no guarantees of stability on newer major versions.
This local development path allows to work without an internet connection, This local development path allows to work without an internet connection,
but be aware that it will *not* allow you to easily test your email stack but be aware that it will *not* allow you to easily test your email stack
@ -75,14 +65,20 @@ or deal with deploying SSL certificates, for example, as your machine is
likely to not be exposed on the internet. A remote machine should be used likely to not be exposed on the internet. A remote machine should be used
for these cases. for these cases.
Depending on your needs, this setup can be very convenient.
If choosing this path, please keep reading at the [local development If choosing this path, please keep reading at the [local development
environment](#local-development-environment) section. environment](#local-development-environment) section.
Please note, there is also a setup guide for a local development path which Alternatively, you may be able to setup a local environnement using Vagrant and
does not require LXC. Please see the [Alternative: Only Virtualbox](#alternative-using-only-virtualbox) Virtualbox which is kinda more resource-hungry because it is fully virtualized,
section for more. but might be more familiar and user-friendly if you already know your way around
Virtualbox's UI.
* Virtualbox (>= 6.x)
* Vagrant (>= ?.?.?)
* Vagrant-virtualbox (>= ?.?.?)
See the [Alternative: Only Virtualbox](#alternative-using-only-virtualbox)
section for more info.
## Remote Development Path ## Remote Development Path
@ -92,8 +88,8 @@ Yunohost can be deployed as a typical install on a remote VPS. You can then use
This method can potentially be faster than the local development environment This method can potentially be faster than the local development environment
assuming you have familiarity with working on VPS machines, if you always have assuming you have familiarity with working on VPS machines, if you always have
internet connectivity when working, and if you're okay with paying a fee. It internet connectivity when working, and if you're okay with paying a fee. It
is also a good option if the required system dependencies (Vagrant, Virtualbox, is also a good option if the required system dependencies (LXD/LXC, Vagrant,
etc.) are not easily available to you on your distribution. Virtualbox, etc.) are not easily available to you on your distribution.
Please be aware that this method should **not** be used for a end-user facing Please be aware that this method should **not** be used for a end-user facing
production environment. production environment.
@ -113,58 +109,33 @@ Here is the development flow:
First you need to install the system dependencies. First you need to install the system dependencies.
`ynh-dev` essentially requires Git, Vagrant, and and the LXC ecosystem. Please `ynh-dev` essentially requires Git and the LXD/LXC ecosystem. Be careful that
see the [local development path](#local-development-path) section for some idea **LXD can conflict with other installed virtualization technologies such as
of the versions required. libvirt or vanilla LXCs**, especially because they all require a daemon based
on DNSmasq and therefore require to listen on port 53.
Please consider using the [latest Vagrant version from their website](https://www.vagrantup.com/downloads.html), distribution versions can include weird bugs that have been fixed upstream. If you still prefer to do that, here are the instructions: On a Debian-based system (regular Debian, Ubuntu, Mint ...), LXD can be
installed using `snapd`. On other systems like Archlinux, you will probably also
The following commands should work on **Linux Mint 19** (and possibly on any Debian Stretch?): be able to install `snapd` using the system package manager (or even
`lxd` directly).
```bash ```bash
$ sudo apt update apt install git snapd
$ sudo apt install git vagrant lxc-templates lxctl lxc cgroup-lite redir bridge-utils libc6 debootstrap libvirt-dev sudo snap install lxd
$ vagrant plugin install vagrant-lxc
$ echo "cgroup /sys/fs/cgroup cgroup defaults 0 0" | sudo tee -a /etc/fstab
$ sudo mount /sys/fs/cgroup
$ lxc-checkconfig
$ echo "veth" | sudo tee -a /etc/modules
```
If you have install libvirtd, you need to stop it and kill dnsmasq libvirtd process, to avoid conflict with dhcp. If you don't ynh-dev start will fail because the lxc container won't be able to get an ip.
On **Debian Buster**, I had to re-patch the driver.rb of vagrant-lxc plugin with [this version](https://raw.githubusercontent.com/fgrehm/vagrant-lxc/2a5510b34cc59cd3cb8f2dcedc3073852d841101/lib/vagrant-lxc/driver.rb) (especially the `roofs_path` function). I also had to install `apparmor` then `systemctl restart apparmor` for `lxc-start` to work. # Adding lxc/lxd to /usr/local/bin to make sure we can use them easily even
# with sudo for which the PATH is defined in /etc/sudoers and probably doesn't
Also check instruction on https://feeding.cloud.geek.nz/posts/lxc-setup-on-debian-stretch/. # include /snap/bin
sudo ln -s /snap/bin/lxc /usr/local/bin/lxc
If you run **Archlinux**, this page should be quite useful to setup LXC: https://github.com/fgrehm/vagrant-lxc/wiki/Usage-on-Arch-Linux-hosts sudo ln -s /snap/bin/lxd /usr/local/bin/lxd
On **both Debian and Archlinux**, typically `/etc/default/lxc-net` and `/etc/lxc/default.conf` should look like this :
```
$ cat /etc/default/lxc-net
USE_LXC_BRIDGE="true"
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
$ cat /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
``` ```
On **Debian Buster**, for backup stuff to work correctly with apparmor, I also had to add: Then you shall initialize LXD which will ask you a bunch of question. Usually
answering the default (just pressing enter) to all questions is fine.
```bash
sudo lxd init
``` ```
mount options=(ro, remount, bind, rbind)
mount options=(ro, remount, bind, relatime)
```
to `/etc/apparmor.d/lxc/lxc-default-cgns` and restart the apparmor service.
Then, go into your favorite development folder and deploy `ynh-dev` with: Then, go into your favorite development folder and deploy `ynh-dev` with:
@ -182,18 +153,26 @@ between the host and the LXC.
When ran on the host, the `./ynh-dev` command allows you to manage YunoHost's dev LXCs. When ran on the host, the `./ynh-dev` command allows you to manage YunoHost's dev LXCs.
First, you might want to start a new LXC with: First, you might want to build the base LXC with:
```bash ```bash
$ cd ynh-dev # if not already done $ cd ynh-dev # if not already done
$ ./ynh-dev rebuild
# ... This will take some time, grab your favorite beverage ...
```
This should create a fresh Debian Stretch LXC, install Yunohost inside and save
the result as `ynh-dev-base` which can then be used to create your actual dev
LXC. (This base can then be used to recreate a fresh Yunohost LXC if you need to
destroy your work LXC)
Then start your actual dev LXC using :
```bash
$ ./ynh-dev start $ ./ynh-dev start
``` ```
This should download an already built LXC from `build.yunohost.org`. If this does not work (or the LXC is outdated), you might want to (re)build a fresh LXC locally with `./ynh-dev rebuild`. After starting the LXC, your terminal will automatically be attached to it. If you later disconnect from the LXC, you can go back in with `./ynh-dev attach`. Later, you might want to destroy the LXC. You can do so with `./ynh-dev destroy`.
After starting the LXC, you should be automatically SSH'ed inside. If you later disconnect from the LXC, you can go back in with `./ynh-dev ssh`.
Later, you might want to destroy the LXC. You can do so with `./ynh-dev destroy`.
## 3. Development and container testing ## 3. Development and container testing
@ -227,9 +206,7 @@ Note that `./ynh-dev use-git yunohost-admin` has a particular behavior: it start
## Advanced: using snapshots ## Advanced: using snapshots
Vagrant is not well integrated with LXC snapshots. You can check `lxc snapshot --help` to learn how to manage lxc snapshots.
However, you may still use `lxc-snapshot` directly to manage snapshots.
## Alternative: Using Only Virtualbox ## Alternative: Using Only Virtualbox

35
Vagrantfile vendored
View file

@ -1,35 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
NETWORK = "10.0.3."
HOSTS = {
"ynh-dev" => ["83", "ynh-dev"],
"ynh-dev-buster" => ["84", "ynh-dev-buster"],
}
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Force guest type, because YunoHost /etc/issue can't be tuned
config.vm.guest = :debian
HOSTS.each do | (name, cfg) |
ipaddr, version = cfg
config.vm.define name do |machine|
machine.vm.box = "yunohost/" + version
# Force guest type, because YunoHost /etc/issue can't be tuned
machine.vm.guest = :debian
machine.vm.provider "lxc" do |lxc|
config.vm.box_url = "https://build.yunohost.org/" + version + "-lxc.box"
config.vm.synced_folder ".", "/ynh-dev", id: "vagrant-root"
config.vm.network :private_network, ip: NETWORK + ipaddr, lxc__bridge_name: 'lxcbr0'
end
end
end # HOSTS-each
end

58
prebuild/Vagrantfile vendored
View file

@ -1,58 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
$script = <<SCRIPT
export DEBIAN_FRONTEND=noninteractive
# Configure guest hostname
sudo bash -c 'echo 127.0.1.1 yunohost.yunohost.org yunohost >> /etc/hosts'
sudo hostname yunohost.yunohost.org
sudo bash -c 'echo yunohost.yunohost.org > /etc/hostname'
# Define root password
echo -e "yunohost\nyunohost" | sudo passwd root
# Allow sudo removal (YunoHost use sudo-ldap)
export SUDO_FORCE_REMOVE=yes
# Upgrade guest (done in install script)
sudo apt-get update
sudo apt-get -y --force-yes upgrade
sudo apt-get -y --force-yes dist-upgrade
# Install YunoHost
wget https://raw.githubusercontent.com/YunoHost/install_script/stretch/install_yunohost -q -O /tmp/install_yunohost
sudo bash /tmp/install_yunohost -a -d unstable
# Cleanup
sudo apt-get clean -y
SCRIPT
NETWORK = "10.0.3."
HOSTS = {
"ynh-dev" => ["83", "stretch64"],
"ynh-dev-buster" => ["84", "testing64"],
}
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
HOSTS.each do | (name, cfg) |
ipaddr, box = cfg
config.vm.define name do |machine|
machine.vm.box = "debian/" + box
machine.vm.provision "shell" do |s|
s.inline = $script
s.args = ""
end
machine.vm.provider "lxc" do |lxc|
machine.vm.network :private_network, ip: NETWORK + ipaddr, lxc__bridge_name: 'lxcbr0'
end
end
end
end

134
ynh-dev
View file

@ -7,7 +7,7 @@ function show_usage() {
${BLUE}==============================${NORMAL} ${BLUE}==============================${NORMAL}
start [NAME] (Create and) starts a LXC (ynh-dev by default) start [NAME] (Create and) starts a LXC (ynh-dev by default)
ssh [NAME] SSH into an already started LXC (ynh-dev by default) attach [NAME] Attach an already started LXC (ynh-dev by default)
destroy [NAME] Destroy the ynh-dev LXC (ynh-dev by default) destroy [NAME] Destroy the ynh-dev LXC (ynh-dev by default)
rebuild Rebuild a fresh, up-to-date box rebuild Rebuild a fresh, up-to-date box
@ -35,7 +35,7 @@ function main()
help|-h|--help) show_usage $ARGUMENTS ;; help|-h|--help) show_usage $ARGUMENTS ;;
start|--start) start_ynhdev $ARGUMENTS ;; start|--start) start_ynhdev $ARGUMENTS ;;
ssh|--ssh) ssh_ynhdev $ARGUMENTS ;; attach|--attach) attach_ynhdev $ARGUMENTS ;;
destroy|--destroy) destroy_ynhdev $ARGUMENTS ;; destroy|--destroy) destroy_ynhdev $ARGUMENTS ;;
rebuild|--rebuild) rebuild_ynhdev $ARGUMENTS ;; rebuild|--rebuild) rebuild_ynhdev $ARGUMENTS ;;
@ -111,80 +111,90 @@ function create_sym_link() {
# Actions # # Actions #
################################################################## ##################################################################
function check_lxc_setup() function check_lxd_setup()
{ {
local LXC_VERSION=$(lxc-info --version) # Check lxd is installed somehow
[[ -e /snap/bin/lxd ]] || which lxd 2>/dev/null \
|| critical "You need to have LXD installed for ynh-dev to be usable from the host machine. Refer to the README to know how to install it."
[[ "${LXC_VERSION:0:1}" == 3 ]] \ # Check that we'll be able to use lxc/lxd using sudo (for which the PATH is defined in /etc/sudoers and probably doesn't include /snap/bin)
|| critical "You need at least version 3 of LXC otherwise the box won't work" if [[ ! -e /usr/bin/lxc ]] && [[ ! -e /usr/bin/lxd ]]
then
[[ -e /usr/local/bin/lxc ]] && [[ -e /usr/local/bin/lxd ]] \
|| critical "You might want to add lxc and lxd inside /usr/local/bin so that there's no tricky PATH issue with sudo. If you installed lxd/lxc with snapd, this should do the trick: sudo ln -s /snap/bin/lxc /usr/local/bin/lxc && sudo ln -s /snap/bin/lxd /usr/local/bin/lxd"
fi
[[ $(systemctl is-active lxc-net) == "active" ]] \ ip a | grep -q lxdbr0 \
|| critical "Service lxc-net should be running ... You probably need to fix your lxc network conf before being able to use lxc's." || critical "There is no 'lxdbr0' interface... Did you ran 'lxd init' ?"
ip a | grep -q lxcbr0 \
|| critical "There is no 'lxcbr0' interface. You probably need to fix your lxc network conf before being able to use lxc's."
} }
function start_ynhdev() function start_ynhdev()
{ {
check_lxc_setup check_lxd_setup
local NAME=${1:-ynh-dev} local BOX=${1:-ynh-dev}
local BOX_NAME="yunohost/$NAME"
local BOX_URL="https://build.yunohost.org/yunohost-$BOX_NAME-lxc.box"
if ! sudo lxc info $BOX &>/dev/null
then
sudo lxc image info $BOX-base &>/dev/null || critical "You should first build the base YunoHost LXC using ./ynh-dev rebuild"
set -eu set -eu
# Download box if not available set -x
#if ! vagrant box list | grep -qc $BOX_NAME ; then sudo lxc launch $BOX-base $BOX
# info "Vagrant box '$BOX_NAME' is missing. Trying to download it" sudo lxc config set $BOX security.privileged true
# vagrant box add $BOX_NAME $BOX_URL sudo lxc config device add $BOX ynhdev-shared-folder disk path=/ynh-dev source="$PWD"
#fi set +x
info "Now attaching to the container"
else
info "Attaching to existing container"
fi
# Run VM attach_ynhdev $BOX
vagrant up $NAME
# Log into the VM
ssh_ynhdev $NAME
} }
function ssh_ynhdev() function attach_ynhdev()
{ {
check_lxc_setup # We need those link for the yunohost-admin build to work without r/w
local NAME=${1:-ynh-dev} # permission in the shared folder.
vagrant ssh $NAME -c "echo 'You are now inside the LXC !'; cd /ynh-dev; sudo su" #
# The target folders (/var/cache/ynh-dev/...) are then created during the
# './ynh-dev use-git yunohost-admin'
if [[ ! -L ./yunohost-admin/src/dist ]]
then
ln -s /var/cache/ynh-dev/yunohost-admin/bower_components ./yunohost-admin/src/
ln -s /var/cache/ynh-dev/yunohost-admin/node_modules ./yunohost-admin/src/
ln -s /var/cache/ynh-dev/yunohost-admin/dist ./yunohost-admin/src/
fi
check_lxd_setup
local BOX=${1:-ynh-dev}
sudo lxc start $BOX 2>/dev/null || true
sudo lxc exec $BOX --cwd /ynh-dev -- /bin/bash
} }
function destroy_ynhdev() function destroy_ynhdev()
{ {
check_lxc_setup check_lxd_setup
local NAME=${1:-ynh-dev} local BOX=${1:-ynh-dev}
vagrant destroy $NAME sudo lxc stop $BOX
sudo lxc delete $BOX
} }
function rebuild_ynhdev() function rebuild_ynhdev()
{ {
check_lxc_setup check_lxd_setup
local BOX="ynh-dev" local BOX=${1:-ynh-dev}
set -x set -x
cd prebuild sudo lxc info $BOX-rebuild >/dev/null && sudo lxc delete $BOX-rebuild --force
sudo lxc launch images:debian/stretch/amd64 $BOX-rebuild
# Create box sudo lxc config set $BOX-rebuild security.privileged true
vagrant up $BOX --provider=lxc --provision || critical "Could not create the box ?" sudo lxc restart $BOX-rebuild
sudo lxc exec $BOX-rebuild -- apt install curl -y
# Package box sudo lxc exec $BOX-rebuild -- /bin/bash -c "curl https://install.yunohost.org | bash -s -- -a -d unstable"
rm -f /tmp/ynh-dev.box sudo lxc stop $BOX-rebuild
vagrant package $BOX --output /tmp/$BOX.box || critical "Could not package package the box ?" sudo lxc publish $BOX-rebuild --alias $BOX-base
# Destroy current box
vagrant destroy $BOX
set +x set +x
# User message, and exit
info "The Vagrant box was packaged to /tmp/$BOX.box"
info "You might want to run : vagrant box add 'yunohost/ynh-dev' /tmp/$BOX.box --force"
} }
function show_vm_ip() function show_vm_ip()
@ -248,26 +258,28 @@ function use_git()
getent passwd ynhdev > /dev/null getent passwd ynhdev > /dev/null
if [ $? -eq 2 ]; then if [ $? -eq 2 ]; then
useradd ynhdev useradd ynhdev
chown -R ynhdev: /ynh-dev/yunohost-admin
fi fi
mkdir -p /var/cache/ynh-dev/yunohost-admin/{bower_components,node_modules,dist}
chown -R ynhdev /var/cache/ynh-dev/yunohost-admin/{bower_components,node_modules,dist}
# Install npm dependencies if needed # Install npm dependencies if needed
which gulp > /dev/null which gulp > /dev/null
if [ $? -eq 1 ] if [ $? -eq 1 ]
then then
info "Installing dependencies to develop in yunohost-admin ..." info "Installing dependencies to develop in yunohost-admin ..."
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash - curl -sL https://deb.nodesource.com/setup_6.x | bash -
sudo apt install nodejs apt install nodejs -y
npm install -g bower
npm install -g gulp
fi
cd /ynh-dev/yunohost-admin/src cd /ynh-dev/yunohost-admin/src
sudo npm install --no-bin-links npm install --no-bin-links
sudo npm install -g bower su ynhdev -c "bower install"
sudo npm install -g gulp su ynhdev -c "gulp build --dev"
fi
cd /ynh-dev/yunohost-admin/src
sudo su -c "bower install" ynhdev
sudo su -c "gulp build --dev" ynhdev
create_sym_link "/ynh-dev/yunohost-admin/src" "/usr/share/yunohost/admin" create_sym_link "/ynh-dev/yunohost-admin/src" "/usr/share/yunohost/admin"
@ -280,7 +292,7 @@ function use_git()
warn "don't need to re-run npm yourself everytime you change " warn "don't need to re-run npm yourself everytime you change "
warn "something ! " warn "something ! "
warn "-------------------------------------------------------- " warn "-------------------------------------------------------- "
sudo su -c "gulp watch --dev" ynhdev su ynhdev -c "gulp watch --dev"
;; ;;
esac esac