YunoHost/ynh-dev
1
0
Fork 0
mirror of https://github.com/YunoHost/ynh-dev.git synced 2024-09-03 20:05:59 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #71 from Salamandar/incus

Browse source 
Allow the use of incus instead of LXD.
This commit is contained in:
Salamandar 2024-03-09 18:40:36 +01:00 • committed by GitHub
commit b8c47886a5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 133 additions and 127 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

181
README.md
View file

@ -1,34 +1,31 @@
# ynh-dev - Yunohost dev environment manager
Please report issues on the following repository:
Please report issues on the following repository: <https://github.com/yunohost/issues>
> https://github.com/yunohost/issues
## Table Of Contents
# Table Of Contents
- [Introduction](#introduction)
* [Local Development Path](#local-development-path)
* [Remote Development Path](#remote-development-path)
- [Local Development Environment](#local-development-environment)
* [1. Setup `ynh-dev` and the development environment](#1-setup-ynh-dev-and-the-development-environment)
* [2. Manage YunoHost's dev LXCs](#2-manage-yunohosts-dev-lxcs)
* [3. Development and container testing](#3-development-and-container-testing)
* [4. Testing the web interface](#4-testing-the-web-interface)
* [Advanced: using snapshots](#advanced-using-snapshots)
* [Alternative: Only Virtualbox](#alternative-using-only-virtualbox)
* [Troubleshooting](#troubleshooting)
- [Remote Development Environment](#remote-development-environment)
* [1. Setup your VPS and install YunoHost](#1-setup-your-vps-and-install-yunohost)
* [2. Setup `ynh-dev` and the development environment](#2-setup-ynh-dev-and-the-development-environment)
* [3. Develop and test](#3-develop-and-test)
- [Further Resources](#further-resources)
- [ynh-dev - Yunohost dev environment manager](#ynh-dev---yunohost-dev-environment-manager)
- [Table Of Contents](#table-of-contents)
- [Introduction](#introduction)
- [Local Development Path](#local-development-path)
- [Remote Development Path](#remote-development-path)
- [Local Development Environment](#local-development-environment)
- [1. Setup `ynh-dev` and the development environment](#1-setup-ynh-dev-and-the-development-environment)
- [2. Manage YunoHost's dev LXCs](#2-manage-yunohosts-dev-lxcs)
- [3. Development and container testing](#3-development-and-container-testing)
- [4. Testing the web interface](#4-testing-the-web-interface)
- [Advanced: using snapshots](#advanced-using-snapshots)
- [Alternative: Using Only Virtualbox](#alternative-using-only-virtualbox)
- [Troubleshooting](#troubleshooting)
- [Remote Development Environment](#remote-development-environment)
- [1. Setup your VPS and install YunoHost](#1-setup-your-vps-and-install-yunohost)
- [2. Setup `ynh-dev` and the development environment](#2-setup-ynh-dev-and-the-development-environment)
- [3. Develop and test](#3-develop-and-test)
- [Further Resources](#further-resources)
---
# Introduction
## Introduction
`ynh-dev` is a CLI tool to manage your local development environment for YunoHost.
@ -36,8 +33,8 @@ This allow you to develop on the various repositories of the YunoHost project.
In particular, it allows you to:
* Create a directory with a clone of each repository of the YunoHost project
* Replace Yunohost debian packages with symlinks to those git clones
- Create a directory with a clone of each repository of the YunoHost project
- Replace Yunohost debian packages with symlinks to those git clones
Because there are many diverse constraints on the development of the Yunohost
project, there is no "official" one-size-fits-all development environment.
@ -49,16 +46,16 @@ your capacities and resources when aiming to setup a development environment.
`yhn-dev` can be used for the following scenarios:
## Local Development Path
### Local Development Path
Yunohost can be developed on using a combination of the following technologies:
* Git (any version is sufficient)
* LXD (>= 2.x) (though only tested with 3.x for now)
- Git (any version is sufficient)
- Incus
Because LXC are containers, they are typically lightweight and quick to start and stop.
But you may find the initial setup complex (in particular network configuration).
LXD makes managing an LXC ecosystem much simpler.
Because LXC are containers, they are typically lightweight and quick to start and stop.
But you may find the initial setup complex (in particular network configuration).
Incus makes managing an LXC ecosystem much simpler.
This local development path allows to work without an internet connection,
but be aware that it will *not* allow you to easily test your email stack
@ -70,18 +67,18 @@ If choosing this path, please keep reading at the [local development
environment](#local-development-environment) section.
Alternatively, you may be able to setup a local environnement using Vagrant and
Virtualbox which is kinda more resource-hungry because it is fully virtualized,
but might be more familiar and user-friendly if you already know your way around
Virtualbox which is kinda more resource-hungry because it is fully virtualized,
but might be more familiar and user-friendly if you already know your way around
Virtualbox's UI.
* Virtualbox (>= 6.x)
* Vagrant (>= ?.?.?)
* Vagrant-virtualbox (>= ?.?.?)
- Virtualbox (>= 6.x)
- Vagrant (>= ?.?.?)
- Vagrant-virtualbox (>= ?.?.?)
See the [Alternative: Only Virtualbox](#alternative-using-only-virtualbox)
section for more info.
## Remote Development Path
### Remote Development Path
Yunohost can be deployed as a typical install on a remote VPS. You can then use
`ynh-dev` to configure a development environment on the server.
@ -89,7 +86,7 @@ Yunohost can be deployed as a typical install on a remote VPS. You can then use
This method can potentially be faster than the local development environment
assuming you have familiarity with working on VPS machines, if you always have
internet connectivity when working, and if you're okay with paying a fee. It
is also a good option if the required system dependencies (LXD/LXC, Vagrant,
is also a good option if the required system dependencies (Incus/LXC, Vagrant,
Virtualbox, etc.) are not easily available to you on your distribution.
Please be aware that this method should **not** be used for a end-user facing
@ -98,7 +95,7 @@ production environment.
If choosing this path, please keep reading at the [remote development
environment](#remote-development-environment) section.
# Local Development Environment
## Local Development Environment
Here is the development flow:
@ -106,53 +103,48 @@ Here is the development flow:
2. Manage YunoHost's development LXCs
3. Develop on your local host and testing in the container
## 1. Setup `ynh-dev` and the development environment
### 1. Setup `ynh-dev` and the development environment
First you need to install the system dependencies.
`ynh-dev` essentially requires Git and the LXD/LXC ecosystem. Be careful that
**LXD can conflict with other installed virtualization technologies such as
libvirt or vanilla LXCs**, especially because they all require a daemon based
`ynh-dev` essentially requires Git and the Incus/LXC ecosystem. Be careful that
**Incus/LXC can conflict with other installed virtualization technologies such as
libvirt or vanilla LXCs**, especially because they all require a daemon based
on DNSmasq and therefore require to listen on port 53.
On a Debian-based system (regular Debian, Ubuntu, Mint ...), LXD can be
installed using `snapd`. On other systems like Archlinux, you will probably also
be able to install `snapd` using the system package manager (or even
`lxd` directly).
Incus can be installed with your Linux distribution package manager, such as:
```bash
apt install git snapd
sudo snap install core
sudo snap install lxd
# Adding lxc/lxd to /usr/local/bin to make sure we can use them easily even
# with sudo for which the PATH is defined in /etc/sudoers and probably doesn't
# include /snap/bin
sudo ln -s /snap/bin/lxc /usr/local/bin/lxc
sudo ln -s /snap/bin/lxd /usr/local/bin/lxd
apt install incus
```
Then you shall initialize LXD which will ask you a bunch of question. Usually
You then need to add yourself in the incus-admin group, to run incus without sudo every time:
```bash
sudo usermod -a -G incus-admin $(whoami)
```
Then you shall initialize Incus which will ask you a bunch of question. Usually
answering the default (just pressing enter) to all questions is fine.
```bash
sudo lxd init
incus admin init
```
Pre-built images are centralized on `devbaseimgs.yunohost.org` and we'll download them from there to speed things up:
```bash
sudo lxc remote add yunohost https://devbaseimgs.yunohost.org --public
incus remote add yunohost https://devbaseimgs.yunohost.org --public
```
On Archlinux-based distributions (Arch, Manjaro, ...) it was found that it's needed
that LXC/LXD will throw some error about "newuidmap failed to write mapping / Failed
that Incus/LXC will throw some error about "newuidmap failed to write mapping / Failed
to set up id mapping" ... It can be [fixed with the following](https://discuss.linuxcontainers.org/t/solved-arch-linux-containers-only-run-when-security-privileged-true/4006/4) :
```
# N.B.: this is ONLY for Arch-based distros
$ echo "root:1000000:65536" > /etc/subuid
$ echo "root:1000000:65536" > /etc/subgid
```bash
## N.B.: this is ONLY for Arch-based distros
echo "root:1000000:65536" > /etc/subuid
echo "root:1000000:65536" > /etc/subgid
```
Then, go into your favorite development folder and deploy `ynh-dev` with:
@ -167,24 +159,30 @@ In particular, you shall notice that there are clones or the various git
repositories. In the next step, we shall start a LXC and 'link' those folders
between the host and the LXC.
## 2. Manage YunoHost's dev LXCs
### 2. Manage YunoHost's dev LXCs
When ran on the host, the `./ynh-dev` command allows you to manage YunoHost's dev LXCs.
Start your actual dev LXC using :
```bash
$ cd ynh-dev # if not already done
$ ./ynh-dev start
cd ynh-dev # if not already done
./ynh-dev start
```
This should automatically download from `devbaseimgs.yunohost.org` a pre-build ynh-dev LXC image running Yunohost unstable, and create a fresh container from it.
This should automatically download from `devbaseimgs.yunohost.org` a pre-build
ynh-dev LXC image running Yunohost unstable, and create a fresh container from it.
After starting the LXC, your terminal will automatically be attached to it. If you later disconnect from the LXC, you can go back in with `./ynh-dev attach`. Later, you might want to destroy the LXC. You can do so with `./ynh-dev destroy`.
After starting the LXC, your terminal will automatically be attached to it. If you
later disconnect from the LXC, you can go back in with `./ynh-dev attach`. Later,
you might want to destroy the LXC. You can do so with `./ynh-dev destroy`.
If you container **doesn't have an ip address nor access to internet**, this is likely because you either have a conflict with another virtualization system or that a program running on the host is using the port 53 and therefore prevent LXD's dnsmasq to run correctly (as stated before in the setup section.)
If you container **doesn't have an ip address nor access to internet**, this is
likely because you either have a conflict with another virtualization system or
that a program running on the host is using the port 53 and therefore prevent
Incus's dnsmasq to run correctly (as stated before in the setup section.)
## 3. Development and container testing
### 3. Development and container testing
After SSH-ing inside the container, you should notice that the *directory* `/ynh-dev` is a shared folder with your host. In particular, it contains the various git clones `yunohost`, `yunohost-admin` and so on - as well as the `./ynh-dev` script itself.
@ -195,7 +193,7 @@ Inside the container, `./ynh-dev` can be used to link the git clones living in t
For instance, after running:
```bash
$ ./ynh-dev use-git yunohost
./ynh-dev use-git yunohost
```
The code of the git clone `'yunohost'` will be directly available inside the container. Which mean that running any `yunohost` command inside the container will use the code from the host... This allows to develop with any tool you want on your host, then test the changes in the container.
@ -204,7 +202,7 @@ The `use-git` action can be used for any package among `yunohost`, `yunohost-adm
***Note***: The `use-git` operation can't be reverted now. Do **not** do this in production.
## 4. Testing the web interface
### 4. Testing the web interface
You should be able to access the web interface via the IP address of the container. The IP can be known from inside the container using either from `ip a` or with `./ynh-dev ip`.
@ -216,25 +214,26 @@ If you want to access to the interface using the domain name, you shall tweak yo
Note that `./ynh-dev use-git yunohost-admin` has a particular behavior: it starts a `gulp` watcher that shall re-compile automatically any changes in the javascript code. Hence this particular `use-git` will keep running until you kill it after your work is done.
## Advanced: using snapshots
### Advanced: using snapshots
You can check `lxc snapshot --help` to learn how to manage lxc snapshots.
You can check `incus snapshot --help` to learn how to manage incus snapshots.
## Alternative: Using Only Virtualbox
### Alternative: Using Only Virtualbox
A Vagrant and Virtualbox (without LXC) guide is provided on another branch of
this repository. This is a known working setup used by some developers. Please
see the ["virtualbox" branch](https://github.com/YunoHost/ynh-dev/tree/virtualbox#develop-on-your-local-machine)
for more.
## Troubleshooting
If you experiment network issues with your lxd during rebuild container steps. Probably your container are not able to get a local IP with DHCP.
### Troubleshooting
It could be due to bridge conflict (for example if you have lxc installed too) or dnsmasq port already used.
If you experiment network issues with your incus during rebuild container steps. Probably your container are not able to get a local IP with DHCP.
It could be due to bridge conflict (for example if you have incus installed too) or dnsmasq port already used.
This [ticket](https://github.com/YunoHost/issues/issues/1664) could help.
# Remote Development Environment
## Remote Development Environment
Here is the development flow:
@ -242,36 +241,36 @@ Here is the development flow:
2. Setup `ynh-dev` and the development environment
3. Develop and test
## 1. Setup your VPS and install YunoHost
### 1. Setup your VPS and install YunoHost
Setup a VPS somewhere (e.g. Scaleway, Digital Ocean, etc.) and install YunoHost following the [usual instructions](https://yunohost.org/#/install_manually).
Depending on what you want to achieve, you might want to run the postinstall right away - and/or setup a domain with an actually working DNS.
## 2. Setup `ynh-dev` and the development environment
### 2. Setup `ynh-dev` and the development environment
Deploy a `ynh-dev` folder at the root of the filesystem with:
```
$ cd /
$ curl https://raw.githubusercontent.com/yunohost/ynh-dev/master/deploy.sh | bash
$ cd /ynh-dev
```bash
cd /
curl https://raw.githubusercontent.com/yunohost/ynh-dev/master/deploy.sh | bash
cd /ynh-dev
```
## 3. Develop and test
### 3. Develop and test
Inside the VPS, `./ynh-dev` can be used to link the git clones to actual the code being ran.
For instance, after running:
```bash
$ ./ynh-dev use-git yunohost
./ynh-dev use-git yunohost
```
Any `yunohost` command will run from the code of the git clone.
The `use-git` action can be used for any package among `yunohost`, `yunohost-admin`, `moulinette` and `ssowat` with similar consequences.
# Further Resources
## Further Resources
* [yunohost.org/dev](https://yunohost.org/dev)
- [yunohost.org/dev](https://yunohost.org/dev)

79
ynh-dev
View file

@ -3,12 +3,12 @@
function show_usage() {
cat <<EOF
${BLUE}On the host, to manage the LXC${NORMAL}
${BLUE}==============================${NORMAL}
${BLUE}On the host, to manage boxes${NORMAL}
${BLUE}============================${NORMAL}
start [DIST] [NAME] [YNH_BRANCH] (Create and) starts a LXC (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
attach [DIST] [NAME] [YNH_BRANCH] Attach an already started LXC (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
destroy [DIST] [NAME] [YNH_BRANCH] Destroy the ynh-dev LXC (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
start [DIST] [NAME] [YNH_BRANCH] (Create and) starts a box (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
attach [DIST] [NAME] [YNH_BRANCH] Attach an already started box (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
destroy [DIST] [NAME] [YNH_BRANCH] Destroy the ynh-dev box (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
rebuild [DIST] [NAME] [YNH_BRANCH] Rebuild a fresh, up-to-date box (DIST=bookworm, NAME=ynh-dev and YNH_BRANCH=unstable by default)
${BLUE}Inside the dev instance${NORMAL}
@ -175,42 +175,49 @@ function prepare_cache_and_deps() {
# Actions #
##################################################################
function check_lxd_setup()
function check_incus_setup()
{
# Check lxd is installed somehow
[[ -e /snap/bin/lxd ]] || which lxd &>/dev/null \
|| critical "You need to have LXD installed for ynh-dev to be usable from the host machine. Refer to the README to know how to install it."
# Check that we'll be able to use lxc/lxd using sudo (for which the PATH is defined in /etc/sudoers and probably doesn't include /snap/bin)
if [[ ! -e /usr/bin/lxc ]] && [[ ! -e /usr/bin/lxd ]] && [[ -e /snap ]]
then
[[ -e /usr/local/bin/lxc ]] && [[ -e /usr/local/bin/lxd ]] \
|| critical "You might want to add lxc and lxd inside /usr/local/bin so that there's no tricky PATH issue with sudo. If you installed lxd/lxc with snapd, this should do the trick: sudo ln -s /snap/bin/lxc /usr/local/bin/lxc && sudo ln -s /snap/bin/lxd /usr/local/bin/lxd"
# Check incus is installed somehow
if ! which incus &>/dev/null; then
critical "You need to have Incus installed for ynh-dev to be usable from the host machine. Refer to the README to know how to install it."
fi
if ! id -nG "$(whoami)" | grep -qw "incus-admin"; then
critical "You need to be in the incus-admin group!"
fi
ip a | grep -q lx[cd]br0 \
|| critical "There is no 'lxcbr0' or 'lxdbr0' interface... Did you ran 'lxd init' ?"
ip a | grep -q incusbr0 \
|| warn "There is no 'incusbr0' interface... Did you ran 'incus admin init' ?"
set_incus_remote
}
function set_incus_remote()
{
configured=$(incus remote list -f json | jq 'has("yunohost")')
if [[ "$configured" != "true" ]]; then
incus remote add yunohost https://devbaseimgs.yunohost.org --public
fi
}
function start_ynhdev()
{
check_lxd_setup
check_incus_setup
local DIST=${1:-bookworm}
local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
if ! sudo lxc info $BOX &>/dev/null
if ! incus info $BOX &>/dev/null
then
if ! sudo lxc image info $BOX-base &>/dev/null
if ! incus image info $BOX-base &>/dev/null
then
LXC_BASE="ynh-dev-$DIST-amd64-$YNH_BRANCH-base"
sudo lxc launch yunohost:$LXC_BASE $BOX -c security.nesting=true -c security.privileged=true \
incus launch yunohost:$LXC_BASE $BOX -c security.nesting=true -c security.privileged=true \
|| critical "Failed to launch the container ?"
else
sudo lxc launch $BOX-base $BOX -c security.nesting=true -c security.privileged=true
incus launch $BOX-base $BOX -c security.nesting=true -c security.privileged=true
fi
sudo lxc config device add $BOX ynhdev-shared-folder disk path=/ynh-dev source="$(readlink -f $(pwd))"
incus config device add $BOX ynhdev-shared-folder disk path=/ynh-dev source="$(readlink -f $(pwd))"
info "Now attaching to the container"
else
info "Attaching to existing container"
@ -221,41 +228,41 @@ function start_ynhdev()
function attach_ynhdev()
{
check_lxd_setup
check_setup
local DIST=${1:-bookworm}
local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
sudo lxc start $BOX 2>/dev/null || true
sudo lxc exec $BOX --cwd /ynh-dev -- /bin/bash
incus start $BOX 2>/dev/null || true
incus exec $BOX --cwd /ynh-dev -- /bin/bash
}
function destroy_ynhdev()
{
check_lxd_setup
check_setup
local DIST=${1:-bookworm}
local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
sudo lxc stop $BOX
sudo lxc delete $BOX
incus stop $BOX
incus delete $BOX
}
function rebuild_ynhdev()
{
check_lxd_setup
check_setup
local DIST=${1:-bookworm}
local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
set -x
sudo lxc info $BOX-rebuild >/dev/null && sudo lxc delete $BOX-rebuild --force
sudo lxc launch images:debian/$DIST/amd64 $BOX-rebuild -c security.nesting=true -c security.privileged=true
incus info $BOX-rebuild >/dev/null && incus delete $BOX-rebuild --force
incus launch images:debian/$DIST/amd64 $BOX-rebuild -c security.nesting=true -c security.privileged=true
sleep 5
sudo lxc exec $BOX-rebuild -- apt install curl -y
incus exec $BOX-rebuild -- apt install curl -y
INSTALL_SCRIPT="https://install.yunohost.org/$DIST"
sudo lxc exec $BOX-rebuild -- /bin/bash -c "curl $INSTALL_SCRIPT | bash -s -- -a -d $YNH_BRANCH"
sudo lxc stop $BOX-rebuild
sudo lxc publish $BOX-rebuild --alias $BOX-base
incus exec $BOX-rebuild -- /bin/bash -c "curl $INSTALL_SCRIPT | bash -s -- -a -d $YNH_BRANCH"
incus stop $BOX-rebuild
incus publish $BOX-rebuild --alias $BOX-base
set +x
}