From 383aa5dcb50909dd06fb08e19a14727c85bf7af0 Mon Sep 17 00:00:00 2001 From: Luke Murphy Date: Tue, 21 May 2019 13:16:48 +0200 Subject: [PATCH 1/4] Further typo fixes Follows from afbb3356db643407105f3bf5949c6a68fda6cfd3. --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3925017..8bc0a21 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ In particular, it allows : * to replace already installed yunohost debian packages by symlinks to those git clones ; * to manage yunohost instances on your host machine with Vagrant by: * creating a Vagrant VM with a pre-installed Yunohost system ; - * sharing the dev environnement your host the VM (so you can develop directly on your host) + * sharing the dev environment your host the VM (so you can develop directly on your host) * finding the ip address of your yunohost vagrant vm yhn-dev can be used either : @@ -21,7 +21,7 @@ yhn-dev can be used either : Here is the development flow: -1. Setup ynh-dev and the development environnement +1. Setup ynh-dev and the development environment 2. Manage YunoHost's dev LXCs 3. Developping on your host, and testing in the container @@ -137,7 +137,7 @@ Instead of running a LXC locally, you may choose (or need) to develop on a dev V Since you do not need to manage LXC, the setup is somewhat "easier" : 1. Setup your VPS and install YunoHost -2. Setup ynh-dev and the development environnement +2. Setup ynh-dev and the development environment 3. Develop and test ### 1. Setup your VPS and install YunoHost @@ -146,7 +146,7 @@ Setup a VPS somewhere (e.g. Scaleway, Digital Ocean, ...) and install YunoHost f Depending on what you want to achieve, you might want to run the postinstall right away - and/or setup a domain with an actually working DNS. -### 2. Setup ynh-dev and the development environnement +### 2. Setup ynh-dev and the development environment Deploy a `ynh-dev` folder at the root of the filesystem with : From ab36afb1515e9b590145bbf0a19bb92c8b477003 Mon Sep 17 00:00:00 2001 From: Luke Murphy Date: Tue, 21 May 2019 13:57:13 +0200 Subject: [PATCH 2/4] Revise and clarify paths for development environment Closes https://github.com/YunoHost/issues/issues/1353. --- README.md | 221 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 153 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index 8bc0a21..61b95d7 100644 --- a/README.md +++ b/README.md @@ -1,56 +1,131 @@ # ynh-dev - Yunohost dev environment manager -Report issues here: https://github.com/yunohost/issues +Please report issues on the following repository: -ynh-dev is a CLI tool to manage your local development environment for YunoHost. This allow you to develop on the various repository of the YunoHost project. +> https://github.com/yunohost/issues -In particular, it allows : +# Table Of Contents - * to create a directory with a clone of each repository of the YunoHost project ; - * to replace already installed yunohost debian packages by symlinks to those git clones ; - * to manage yunohost instances on your host machine with Vagrant by: - * creating a Vagrant VM with a pre-installed Yunohost system ; - * sharing the dev environment your host the VM (so you can develop directly on your host) - * finding the ip address of your yunohost vagrant vm +- [Introduction](#introduction) + * [Local Development Path](#local-development-path) + * [Remote Development Path](#remote-development-path) -yhn-dev can be used either : -1. on your local machine with LXCs (you can peacefully develop independently of your internet connection) -2. on a remote machine dedicated to dev (e.g. if you need the VM to be exposed on internet : test let's encrypt, email stack ...) +- [Local Development Environment](#local-development-environment) + * [1. Setup `ynh-dev` and the development environment](#1-setup-ynh-dev-and-the-development-environment) + * [2. Manage YunoHost's dev LXCs](#2-manage-yunohosts-dev-lxcs) + * [3. Development and container testing](#3-development-and-container-testing) + * [4. Testing the web interface](#4-testing-the-web-interface) + * [Advanced: using snapshots](#advanced-using-snapshots) -## Develop on your local machine +- [Remote Development Environment](#remote-development-environment) + * [1. Setup your VPS and install YunoHost](#1-setup-your-vps-and-install-yunohost) + * [2. Setup `ynh-dev` and the development environment](#2-setup-ynh-dev-and-the-development-environment) + * [3. Develop and test](#3-develop-and-test) + +- [Further Resources](#further-resources) + +--- + +# Introduction + +`ynh-dev` is a CLI tool to manage your local development environment for YunoHost. + +This allow you to develop on the various repositories of the YunoHost project. + +In particular, it allows you to: + + * Create a directory with a clone of each repository of the YunoHost project + * Replace Yunohost debian packages with symlinks to those git clones + +Because there are many diverse constraints on the development of the Yunohost +project, there is no "official" one-size-fits-all development environment. +However, we do provide documentation for what developers are using now in +various circumstances. + +Please keep this in mind when reviewing the following options with regard to +your capacities and resources when aiming to setup a development environment. + +`yhn-dev` can be used for the following scenarios: + +## Local Development Path + +Yunohost can be developed on using a combination of the following technologies: + + * Git (any version is sufficient) + * Virtualbox (>= 6.x) + * Vagrant (>= 2.x) + * LXC (>= 3.x) + * Vagrant-LXC (>= 1.4.x) + +Please keep in mind that these versions may not be available on your OS +distribution and you may be required to install them as binary or from source. +There are no guarantees of stability on newer major versions. + +This local development path will not allow you to easily test your email stack +or deal with deploying SSL certificates, for example. A remote machine should +be used for these cases. + +Depending on your needs, this setup can be very convenient. + +If choosing this path, please keep reading at the [local development +environment](#local-development-environment) section. + +## Remote Development Path + +Yunohost can be deployed as a typical install on a remote VPS. You can then use +`ynh-dev` to configure a development environment on the server. + +This method can potentially be faster than the local development environment +assuming you have familiarity with working on VPS machines. It is also a good +option if the required system dependencies (Vagrant, Virtualbox, etc.) are not +easily available to you on your distribution. + +Please be aware that this method should **not** be used for a end-user facing +production environment. + +If choosing this path, please keep reading at the [remote development +environment](#remote-development-environment) section. + +# Local Development Environment Here is the development flow: -1. Setup ynh-dev and the development environment -2. Manage YunoHost's dev LXCs -3. Developping on your host, and testing in the container +1. Setup `ynh-dev` and the development environment +2. Manage YunoHost's development LXCs +3. Develop on your local host and testing in the container -### 1. Setup ynh-dev and the development environment +## 1. Setup `ynh-dev` and the development environment -First you need to install the dependencies. ynh-dev essentially requires git, vagrant, and an LXC ecosystem. +First you need to install the system dependencies. + +`ynh-dev` essentially requires Git, Vagrant, and and the LXC ecosystem. Please +see the [local development path](#local-development-path) section for some idea +of the versions required. Please consider using the [latest Vagrant version from their website](https://www.vagrantup.com/downloads.html), distribution versions can include weird bugs that have been fixed upstream. If you still prefer to do that, here are the instructions: -The following commands should work on **Linux Mint 19** (and possibly on any Debian Stretch?) : +The following commands should work on **Linux Mint 19** (and possibly on any Debian Stretch?): ```bash -sudo apt update -sudo apt install git vagrant lxc-templates lxctl lxc cgroup-lite redir bridge-utils libc6 debootstrap -vagrant plugin install vagrant-lxc -echo "cgroup /sys/fs/cgroup cgroup defaults 0 0" | sudo tee -a /etc/fstab -sudo mount /sys/fs/cgroup -lxc-checkconfig -echo "veth" | sudo tee -a /etc/modules +$ sudo apt update +$ sudo apt install git vagrant lxc-templates lxctl lxc cgroup-lite redir bridge-utils libc6 debootstrap +$ vagrant plugin install vagrant-lxc +$ echo "cgroup /sys/fs/cgroup cgroup defaults 0 0" | sudo tee -a /etc/fstab +$ sudo mount /sys/fs/cgroup +$ lxc-checkconfig +$ echo "veth" | sudo tee -a /etc/modules ``` -On **Debian Buster**, I had to re-patch the driver.rb of vagrant-lxc plugin with [this version](https://raw.githubusercontent.com/fgrehm/vagrant-lxc/2a5510b34cc59cd3cb8f2dcedc3073852d841101/lib/vagrant-lxc/driver.rb) (especially the `roofs_path` function). I also had to install `apparmor` then `systemctl restart apparmor` for `lxc-start` to work. Also check instruction on https://feeding.cloud.geek.nz/posts/lxc-setup-on-debian-stretch/ +On **Debian Buster**, I had to re-patch the driver.rb of vagrant-lxc plugin with [this version](https://raw.githubusercontent.com/fgrehm/vagrant-lxc/2a5510b34cc59cd3cb8f2dcedc3073852d841101/lib/vagrant-lxc/driver.rb) (especially the `roofs_path` function). I also had to install `apparmor` then `systemctl restart apparmor` for `lxc-start` to work. -If you run **Archlinux**, this page should be quite useful to setup LXC : https://github.com/fgrehm/vagrant-lxc/wiki/Usage-on-Arch-Linux-hosts +Also check instruction on https://feeding.cloud.geek.nz/posts/lxc-setup-on-debian-stretch/. + +If you run **Archlinux**, this page should be quite useful to setup LXC: https://github.com/fgrehm/vagrant-lxc/wiki/Usage-on-Arch-Linux-hosts On **both Debian and Archlinux**, typically `/etc/default/lxc-net` and `/etc/lxc/default.conf` should look like this : ``` - > cat /etc/default/lxc-net +$ cat /etc/default/lxc-net USE_LXC_BRIDGE="true" LXC_BRIDGE="lxcbr0" LXC_ADDR="10.0.3.1" @@ -59,115 +134,125 @@ LXC_NETWORK="10.0.3.0/24" LXC_DHCP_RANGE="10.0.3.2,10.0.3.254" LXC_DHCP_MAX="253" -> cat /etc/lxc/default.conf +$ cat /etc/lxc/default.conf lxc.net.0.type = veth lxc.net.0.link = lxcbr0 lxc.net.0.flags = up lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx ``` -On **Debian Buster**, for backup stuff to work correctly with apparmor, I also had to add `mount options=(ro, remount, bind, rbind),` and ` mount options=(ro, remount, bind, relatime),` to `/etc/apparmor.d/lxc/lxc-default-cgns` and restart the apparmor service. +On **Debian Buster**, for backup stuff to work correctly with apparmor, I also had to add: -Then, go into your favorite development folder and deploy ynh-dev with : +``` +mount options=(ro, remount, bind, rbind) +mount options=(ro, remount, bind, relatime) +``` + +to `/etc/apparmor.d/lxc/lxc-default-cgns` and restart the apparmor service. + +Then, go into your favorite development folder and deploy `ynh-dev` with: ```bash curl https://raw.githubusercontent.com/yunohost/ynh-dev/master/deploy.sh | bash ``` -This will create a new `ynh-dev` folder with everything you need inside. In particular, you shall notice that there are clones or the various git repositories. In the next step, we shall start a LXC and 'link' those folders between the host and the LXC. +This will create a new `ynh-dev` folder with everything you need inside. -### 2. Learn how to manage YunoHost's dev LXCs +In particular, you shall notice that there are clones or the various git +repositories. In the next step, we shall start a LXC and 'link' those folders +between the host and the LXC. + +## 2. Manage YunoHost's dev LXCs When ran on the host, the `./ynh-dev` command allows you to manage YunoHost's dev LXCs. -First, you might want to start a new LXC with : +First, you might want to start a new LXC with: ```bash -cd ynh-dev # if not already done -./ynh-dev start +$ cd ynh-dev # if not already done +$ ./ynh-dev start ``` This should download an already built LXC from `build.yunohost.org`. If this does not work (or the LXC is outdated), you might want to (re)build a fresh LXC locally with `./ynh-dev rebuild`. -After starting the LXC, you should be automatically SSH'ed inside. If you later disconnect from the LXC, you can go back in with `./ynh-dev ssh` +After starting the LXC, you should be automatically SSH'ed inside. If you later disconnect from the LXC, you can go back in with `./ynh-dev ssh`. Later, you might want to destroy the LXC. You can do so with `./ynh-dev destroy`. - -### 3. Developping on your host, and testing in the container +## 3. Development and container testing After SSH-ing inside the container, you should notice that the *directory* `/ynh-dev` is a shared folder with your host. In particular, it contains the various git clones `yunohost`, `yunohost-admin` and so on - as well as the `./ynh-dev` script itself. Inside the container, `./ynh-dev` can be used to link the git clones living in the host to the code being ran inside the container. -For instance, after running +For instance, after running: ```bash -./ynh-dev use-git yunohost +$ ./ynh-dev use-git yunohost ``` -the code of the git clone `'yunohost'` will be directly available inside the container. Which mean that running any `yunohost` command inside the container will use the code from the host... This allows to develop with any tool you want on your host, then test the changes in the container. +The code of the git clone `'yunohost'` will be directly available inside the container. Which mean that running any `yunohost` command inside the container will use the code from the host... This allows to develop with any tool you want on your host, then test the changes in the container. The `use-git` action can be used for any package among `yunohost`, `yunohost-admin`, `moulinette` and `ssowat` with similar consequences. You might want to run use-git several times depending on what you want to develop precisely. -***Note***: The `use-git` operation can't be reverted now. DON'T DO THIS IN PRODUCTION ! +***Note***: The `use-git` operation can't be reverted now. Do **not** do this in production. - -#### Testing the web interface +## 4. Testing the web interface You should be able to access the web interface via the IP address of the container. The IP can be known from inside the container using either from `ip a` or with `./ynh-dev ip`. -If you want to access to the interface using the domain name, you shall tweak your /etc/hosts and add a line such as: +If you want to access to the interface using the domain name, you shall tweak your `/etc/hosts` and add a line such as: ```bash 111.222.333.444 yolo.test ``` -Note that `./ynh-dev use-git yunohost-admin` has a particular behavior : it starts a `gulp` watcher that shall re-compile automatically any changes in the javascript code. Hence this particular `use-git` will keep running until you kill it after your work is done. +Note that `./ynh-dev use-git yunohost-admin` has a particular behavior: it starts a `gulp` watcher that shall re-compile automatically any changes in the javascript code. Hence this particular `use-git` will keep running until you kill it after your work is done. +## Advanced: using snapshots -#### Advanced : using snapshots +Vagrant is not well integrated with LXC snapshots. -Vagrant is not well integrated with LXC snapshots. However, you may still use `lxc-snapshot` directly to manage snapshots. +However, you may still use `lxc-snapshot` directly to manage snapshots. -## Develop on a remote server +# Remote Development Environment -Instead of running a LXC locally, you may choose (or need) to develop on a dev VPS. *Be aware that this is a dev tool : do NOT run this procedure on a production environment !*. - -Since you do not need to manage LXC, the setup is somewhat "easier" : +Here is the development flow: 1. Setup your VPS and install YunoHost -2. Setup ynh-dev and the development environment +2. Setup `ynh-dev` and the development environment 3. Develop and test -### 1. Setup your VPS and install YunoHost +## 1. Setup your VPS and install YunoHost -Setup a VPS somewhere (e.g. Scaleway, Digital Ocean, ...) and install YunoHost following https://yunohost.org/#/install_manually +Setup a VPS somewhere (e.g. Scaleway, Digital Ocean, etc.) and install YunoHost following the [usual instructions](https://yunohost.org/#/install_manually). Depending on what you want to achieve, you might want to run the postinstall right away - and/or setup a domain with an actually working DNS. -### 2. Setup ynh-dev and the development environment +## 2. Setup `ynh-dev` and the development environment -Deploy a `ynh-dev` folder at the root of the filesystem with : +Deploy a `ynh-dev` folder at the root of the filesystem with: ``` -cd / -curl https://raw.githubusercontent.com/yunohost/ynh-dev/master/deploy.sh | bash -cd /ynh-dev +$ cd / +$ curl https://raw.githubusercontent.com/yunohost/ynh-dev/master/deploy.sh | bash +$ cd /ynh-dev ``` -### 3. Develop and test +## 3. Develop and test Inside the VPS, `./ynh-dev` can be used to link the git clones to actual the code being ran. -For instance, after running +For instance, after running: ```bash -./ynh-dev use-git yunohost +$ ./ynh-dev use-git yunohost ``` -any `yunohost` command will run from the code of the git clone. The `use-git` action can be used for any package among `yunohost`, `yunohost-admin`, `moulinette` and `ssowat` with similar consequences. +Any `yunohost` command will run from the code of the git clone. -## More info +The `use-git` action can be used for any package among `yunohost`, `yunohost-admin`, `moulinette` and `ssowat` with similar consequences. -* [yunohost.org/dev](https://yunohost.org/dev) not up-to-date. +# Further Resources + + * [yunohost.org/dev](https://yunohost.org/dev) From a3e372d5a7dccbb82d8a911344e398e519763c9e Mon Sep 17 00:00:00 2001 From: Alexandre Aubin Date: Tue, 21 May 2019 16:45:04 +0200 Subject: [PATCH 3/4] More comments on the tradeoff LXC vs Virtualbox, and local vs remote --- README.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 61b95d7..e6a7511 100644 --- a/README.md +++ b/README.md @@ -52,18 +52,27 @@ your capacities and resources when aiming to setup a development environment. Yunohost can be developed on using a combination of the following technologies: * Git (any version is sufficient) - * Virtualbox (>= 6.x) * Vagrant (>= 2.x) * LXC (>= 3.x) * Vagrant-LXC (>= 1.4.x) +LXC is typically lightweight but you may find the initial setup complex +(in particular network configuration). Alternatively, you may be able +to setup a local environnement using Virtualbox which is kinda more +resource-hungry : + + * Virtualbox (>= 6.x) + * Vagrant-virtualbox (>= ?.?.?) + Please keep in mind that these versions may not be available on your OS distribution and you may be required to install them as binary or from source. There are no guarantees of stability on newer major versions. -This local development path will not allow you to easily test your email stack -or deal with deploying SSL certificates, for example. A remote machine should -be used for these cases. +This local development path allows to work without an internet connection, +but be aware that it will *not* allow you to easily test your email stack +or deal with deploying SSL certificates, for example, as your machine is +likely to not be exposed on the internet. A remote machine should be used +for these cases. Depending on your needs, this setup can be very convenient. @@ -76,9 +85,10 @@ Yunohost can be deployed as a typical install on a remote VPS. You can then use `ynh-dev` to configure a development environment on the server. This method can potentially be faster than the local development environment -assuming you have familiarity with working on VPS machines. It is also a good -option if the required system dependencies (Vagrant, Virtualbox, etc.) are not -easily available to you on your distribution. +assuming you have familiarity with working on VPS machines, if you always have +internet connectivity when working, and if you're okay with paying a fee. It +is also a good option if the required system dependencies (Vagrant, Virtualbox, +etc.) are not easily available to you on your distribution. Please be aware that this method should **not** be used for a end-user facing production environment. From 3f80c88316b40e2f04c8957ba4b125ed5e8677a5 Mon Sep 17 00:00:00 2001 From: Luke Murphy Date: Wed, 22 May 2019 22:06:47 +0200 Subject: [PATCH 4/4] Mention the Virtualbox branch --- README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/README.md b/README.md index e6a7511..657eb42 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ Please report issues on the following repository: * [3. Development and container testing](#3-development-and-container-testing) * [4. Testing the web interface](#4-testing-the-web-interface) * [Advanced: using snapshots](#advanced-using-snapshots) + * [Alternative: Only Virtualbox](#alternative-using-only-virtualbox) - [Remote Development Environment](#remote-development-environment) * [1. Setup your VPS and install YunoHost](#1-setup-your-vps-and-install-yunohost) @@ -79,6 +80,10 @@ Depending on your needs, this setup can be very convenient. If choosing this path, please keep reading at the [local development environment](#local-development-environment) section. +Please note, there is also a setup guide for a local development path which +does not require LXC. Please see the [Alternative: Only Virtualbox](#alternative-using-only-virtualbox) +section for more. + ## Remote Development Path Yunohost can be deployed as a typical install on a remote VPS. You can then use @@ -225,6 +230,13 @@ Vagrant is not well integrated with LXC snapshots. However, you may still use `lxc-snapshot` directly to manage snapshots. +## Alternative: Using Only Virtualbox + +A Vagrant and Virtualbox (without LXC) guide is provided on another branch of +this repository. This is a known working setup used by some developers. Please +see the ["virtualbox" branch](https://github.com/YunoHost/ynh-dev/tree/virtualbox#develop-on-your-local-machine) +for more. + # Remote Development Environment Here is the development flow: