YunoHost/ynh-dev
1
0
Fork 0
mirror of https://github.com/YunoHost/ynh-dev.git synced 2024-09-03 20:05:59 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'master' of https://github.com/YunoHost/ynh-dev

Browse source
This commit is contained in:
ljf 2021-05-08 15:10:08 +02:00
commit 32fec007e7
2 changed files with 122 additions and 99 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

49
README.md
View file

@ -17,6 +17,7 @@ Please report issues on the following repository:
* [4. Testing the web interface](#4-testing-the-web-interface) * [4. Testing the web interface](#4-testing-the-web-interface)
* [Advanced: using snapshots](#advanced-using-snapshots) * [Advanced: using snapshots](#advanced-using-snapshots)
* [Alternative: Only Virtualbox](#alternative-using-only-virtualbox) * [Alternative: Only Virtualbox](#alternative-using-only-virtualbox)
* [Troubleshooting](#troubleshooting)
- [Remote Development Environment](#remote-development-environment) - [Remote Development Environment](#remote-development-environment)
* [1. Setup your VPS and install YunoHost](#1-setup-your-vps-and-install-yunohost) * [1. Setup your VPS and install YunoHost](#1-setup-your-vps-and-install-yunohost)
@ -138,6 +139,22 @@ answering the default (just pressing enter) to all questions is fine.
sudo lxd init sudo lxd init
``` ```
Pre-built images are centralized on `devbaseimgs.yunohost.org` and we'll download them from there to speed things up:
```bash
sudo lxc remote add yunohost https://devbaseimgs.yunohost.org --public
```
On Archlinux-based distributions (Arch, Manjaro, ...) it was found that it's needed
that LXC/LXD will throw some error about "newuidmap failed to write mapping / Failed
to set up id mapping" ... It can be [fixed with the following](https://discuss.linuxcontainers.org/t/solved-arch-linux-containers-only-run-when-security-privileged-true/4006/4) :
```
# N.B.: this is ONLY for Arch-based distros
$ echo "root:1000000:65536" > /etc/subuid
$ echo "root:1000000:65536" > /etc/subgid
```
Then, go into your favorite development folder and deploy `ynh-dev` with: Then, go into your favorite development folder and deploy `ynh-dev` with:
```bash ```bash
@ -154,34 +171,15 @@ between the host and the LXC.
When ran on the host, the `./ynh-dev` command allows you to manage YunoHost's dev LXCs. When ran on the host, the `./ynh-dev` command allows you to manage YunoHost's dev LXCs.
First, you might want to build the base LXC with: Start your actual dev LXC using :
```bash ```bash
$ cd ynh-dev # if not already done $ cd ynh-dev # if not already done
$ ./ynh-dev rebuild
# ... This will take some time, grab your favorite beverage ...
```
This should create a fresh Debian Stretch LXC, install Yunohost inside and save
the result as `ynh-dev-base` which can then be used to create your actual dev
LXC. (This base can then be used to recreate a fresh Yunohost LXC if you need to
destroy your work LXC)
On Archlinux-based distributions (Arch, Manjaro, ...) it was found that it's needed
that LXC/LXD will throw some error about "newuidmap failed to write mapping / Failed
to set up id mapping" ... It can be [fixed with the following](https://discuss.linuxcontainers.org/t/solved-arch-linux-containers-only-run-when-security-privileged-true/4006/4) :
```
$ echo "root:1000000:65536" > /etc/subuid
$ echo "root:1000000:65536" > /etc/subgid
```
Then start your actual dev LXC using :
```bash
$ ./ynh-dev start $ ./ynh-dev start
``` ```
This should automatically download from `devbaseimgs.yunohost.org` a pre-build ynh-dev LXC image running Yunohost unstable, and create a fresh container from it.
After starting the LXC, your terminal will automatically be attached to it. If you later disconnect from the LXC, you can go back in with `./ynh-dev attach`. Later, you might want to destroy the LXC. You can do so with `./ynh-dev destroy`. After starting the LXC, your terminal will automatically be attached to it. If you later disconnect from the LXC, you can go back in with `./ynh-dev attach`. Later, you might want to destroy the LXC. You can do so with `./ynh-dev destroy`.
## 3. Development and container testing ## 3. Development and container testing
@ -225,6 +223,13 @@ this repository. This is a known working setup used by some developers. Please
see the ["virtualbox" branch](https://github.com/YunoHost/ynh-dev/tree/virtualbox#develop-on-your-local-machine) see the ["virtualbox" branch](https://github.com/YunoHost/ynh-dev/tree/virtualbox#develop-on-your-local-machine)
for more. for more.
## Troubleshooting
If you experiment network issues with your lxd during rebuild container steps. Probably your container are not able to get a local IP with DHCP.
It could be due to bridge conflict (for example if you have lxc installed too) or dnsmasq port already used.
This [ticket](https://github.com/YunoHost/issues/issues/1664) could help.
# Remote Development Environment # Remote Development Environment
Here is the development flow: Here is the development flow:

172
ynh-dev
View file

@ -1,4 +1,4 @@
#!/bin/bash #!/usr/bin/env bash
function show_usage() { function show_usage() {
cat <<EOF cat <<EOF
@ -6,10 +6,10 @@ function show_usage() {
${BLUE}On the host, to manage the LXC${NORMAL} ${BLUE}On the host, to manage the LXC${NORMAL}
${BLUE}==============================${NORMAL} ${BLUE}==============================${NORMAL}
start [DIST] [NAME] (Create and) starts a LXC (DIST=buster and NAME=ynh-dev by default) start [DIST] [NAME] [YNH_BRANCH] (Create and) starts a LXC (DIST=buster, NAME=ynh-dev and YNH_BRANCH=unstable by default)
attach [DIST] [NAME] Attach an already started LXC (DIST=buster and NAME=ynh-dev by default) attach [DIST] [NAME] [YNH_BRANCH] Attach an already started LXC (DIST=buster, NAME=ynh-dev and YNH_BRANCH=unstable by default)
destroy [DIST] [NAME] Destroy the ynh-dev LXC (DIST=buster and NAME=ynh-dev by default) destroy [DIST] [NAME] [YNH_BRANCH] Destroy the ynh-dev LXC (DIST=buster, NAME=ynh-dev and YNH_BRANCH=unstable by default)
rebuild [DIST] [NAME] Rebuild a fresh, up-to-date box (DIST=buster and NAME=ynh-dev by default) rebuild [DIST] [NAME] [YNH_BRANCH] Rebuild a fresh, up-to-date box (DIST=buster, NAME=ynh-dev and YNH_BRANCH=unstable by default)
${BLUE}Inside the dev instance${NORMAL} ${BLUE}Inside the dev instance${NORMAL}
${BLUE}=======================${NORMAL} ${BLUE}=======================${NORMAL}
@ -118,7 +118,7 @@ function check_lxd_setup()
|| critical "You need to have LXD installed for ynh-dev to be usable from the host machine. Refer to the README to know how to install it." || critical "You need to have LXD installed for ynh-dev to be usable from the host machine. Refer to the README to know how to install it."
# Check that we'll be able to use lxc/lxd using sudo (for which the PATH is defined in /etc/sudoers and probably doesn't include /snap/bin) # Check that we'll be able to use lxc/lxd using sudo (for which the PATH is defined in /etc/sudoers and probably doesn't include /snap/bin)
if [[ ! -e /usr/bin/lxc ]] && [[ ! -e /usr/bin/lxd ]] if [[ ! -e /usr/bin/lxc ]] && [[ ! -e /usr/bin/lxd ]] && [[ -e /snap ]]
then then
[[ -e /usr/local/bin/lxc ]] && [[ -e /usr/local/bin/lxd ]] \ [[ -e /usr/local/bin/lxc ]] && [[ -e /usr/local/bin/lxd ]] \
|| critical "You might want to add lxc and lxd inside /usr/local/bin so that there's no tricky PATH issue with sudo. If you installed lxd/lxc with snapd, this should do the trick: sudo ln -s /snap/bin/lxc /usr/local/bin/lxc && sudo ln -s /snap/bin/lxd /usr/local/bin/lxd" || critical "You might want to add lxc and lxd inside /usr/local/bin so that there's no tricky PATH issue with sudo. If you installed lxd/lxc with snapd, this should do the trick: sudo ln -s /snap/bin/lxc /usr/local/bin/lxc && sudo ln -s /snap/bin/lxd /usr/local/bin/lxd"
@ -133,41 +133,34 @@ function start_ynhdev()
check_lxd_setup check_lxd_setup
local DIST=${1:-buster} local DIST=${1:-buster}
local BOX=${2:-ynh-dev}-${DIST} local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
if ! sudo lxc info $BOX &>/dev/null if ! sudo lxc info $BOX &>/dev/null
then then
sudo lxc image info $BOX-base &>/dev/null || critical "You should first build the base YunoHost LXC using ./ynh-dev rebuild" if ! sudo lxc image info $BOX-base &>/dev/null
set -eu then
set -x LXC_BASE="ynh-dev-$DIST-amd64-$YNH_BRANCH-base"
sudo lxc launch $BOX-base $BOX sudo lxc launch yunohost:$LXC_BASE $BOX -c security.nesting=true -c security.privileged=true \
sudo lxc config set $BOX security.privileged true || critical "Failed to launch the container ?"
sudo lxc config set $BOX security.nesting true # Need this for apparmor for some reason else
sudo lxc launch $BOX-base $BOX -c security.nesting=true -c security.privileged=true
fi
sudo lxc config device add $BOX ynhdev-shared-folder disk path=/ynh-dev source="$(readlink -f $(pwd))" sudo lxc config device add $BOX ynhdev-shared-folder disk path=/ynh-dev source="$(readlink -f $(pwd))"
set +x
info "Now attaching to the container" info "Now attaching to the container"
else else
info "Attaching to existing container" info "Attaching to existing container"
fi fi
attach_ynhdev $BOX attach_ynhdev "$@"
} }
function attach_ynhdev() function attach_ynhdev()
{ {
# We need those link for the yunohost-admin build to work without r/w
# permission in the shared folder.
#
# The target folders (/var/cache/ynh-dev/...) are then created during the
# './ynh-dev use-git yunohost-admin'
if [[ ! -L ./yunohost-admin/src/dist ]]
then
ln -s /var/cache/ynh-dev/yunohost-admin/node_modules ./yunohost-admin/src/
ln -s /var/cache/ynh-dev/yunohost-admin/dist ./yunohost-admin/src/
fi
check_lxd_setup check_lxd_setup
local BOX=${1:-ynh-dev-buster} local DIST=${1:-buster}
local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
sudo lxc start $BOX 2>/dev/null || true sudo lxc start $BOX 2>/dev/null || true
sudo lxc exec $BOX --cwd /ynh-dev -- /bin/bash sudo lxc exec $BOX --cwd /ynh-dev -- /bin/bash
} }
@ -176,7 +169,8 @@ function destroy_ynhdev()
{ {
check_lxd_setup check_lxd_setup
local DIST=${1:-buster} local DIST=${1:-buster}
local BOX=${2:-ynh-dev}-${DIST} local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
sudo lxc stop $BOX sudo lxc stop $BOX
sudo lxc delete $BOX sudo lxc delete $BOX
} }
@ -186,18 +180,16 @@ function rebuild_ynhdev()
check_lxd_setup check_lxd_setup
local DIST=${1:-buster} local DIST=${1:-buster}
local BOX=${2:-ynh-dev}-${DIST} local YNH_BRANCH=${3:-unstable}
local BOX=${2:-ynh-dev}-${DIST}-${YNH_BRANCH}
set -x set -x
sudo lxc info $BOX-rebuild >/dev/null && sudo lxc delete $BOX-rebuild --force sudo lxc info $BOX-rebuild >/dev/null && sudo lxc delete $BOX-rebuild --force
sudo lxc launch images:debian/$DIST/amd64 $BOX-rebuild sudo lxc launch images:debian/$DIST/amd64 $BOX-rebuild -c security.nesting=true -c security.privileged=true
sudo lxc config set $BOX-rebuild security.privileged true
sudo lxc config set $BOX-rebuild security.nesting true # Need this for apparmor for some reason
sudo lxc restart $BOX-rebuild
sleep 5 sleep 5
sudo lxc exec $BOX-rebuild -- apt install curl -y sudo lxc exec $BOX-rebuild -- apt install curl -y
INSTALL_SCRIPT="https://install.yunohost.org" INSTALL_SCRIPT="https://install.yunohost.org/$DIST"
sudo lxc exec $BOX-rebuild -- /bin/bash -c "curl $INSTALL_SCRIPT | bash -s -- -a -d unstable" sudo lxc exec $BOX-rebuild -- /bin/bash -c "curl $INSTALL_SCRIPT | bash -s -- -a -d $YNH_BRANCH"
sudo lxc stop $BOX-rebuild sudo lxc stop $BOX-rebuild
sudo lxc publish $BOX-rebuild --alias $BOX-base sudo lxc publish $BOX-rebuild --alias $BOX-base
set +x set +x
@ -219,7 +211,7 @@ function use_git()
PACKAGES=('moulinette' 'ssowat' 'yunohost' 'yunohost-admin') PACKAGES=('moulinette' 'ssowat' 'yunohost' 'yunohost-admin')
fi fi
for i in ${!PACKAGES[@]}; for i in "${!PACKAGES[@]}";
do do
case ${PACKAGES[i]} in case ${PACKAGES[i]} in
ssowat) ssowat)
@ -228,7 +220,7 @@ function use_git()
;; ;;
moulinette) moulinette)
create_sym_link "/ynh-dev/moulinette/locales" "/usr/share/moulinette/locale" create_sym_link "/ynh-dev/moulinette/locales" "/usr/share/moulinette/locale"
create_sym_link "/ynh-dev/moulinette/moulinette" "/usr/lib/python2.7/dist-packages/moulinette" create_sym_link "/ynh-dev/moulinette/moulinette" "/usr/lib/python3/dist-packages/moulinette"
success "Now using Git repository for Moulinette" success "Now using Git repository for Moulinette"
;; ;;
yunohost) yunohost)
@ -241,6 +233,7 @@ function use_git()
create_sym_link "/ynh-dev/yunohost/sbin/yunohost-reset-ldap-password" "/usr/sbin/yunohost-reset-ldap-password" create_sym_link "/ynh-dev/yunohost/sbin/yunohost-reset-ldap-password" "/usr/sbin/yunohost-reset-ldap-password"
# data # data
python3 "/ynh-dev/yunohost/data/actionsmap/yunohost_completion.py"
create_sym_link "/ynh-dev/yunohost/data/bash-completion.d/yunohost" "/etc/bash_completion.d/yunohost" create_sym_link "/ynh-dev/yunohost/data/bash-completion.d/yunohost" "/etc/bash_completion.d/yunohost"
create_sym_link "/ynh-dev/yunohost/data/actionsmap/yunohost.yml" "/usr/share/moulinette/actionsmap/yunohost.yml" create_sym_link "/ynh-dev/yunohost/data/actionsmap/yunohost.yml" "/usr/share/moulinette/actionsmap/yunohost.yml"
create_sym_link "/ynh-dev/yunohost/data/hooks" "/usr/share/yunohost/hooks" create_sym_link "/ynh-dev/yunohost/data/hooks" "/usr/share/yunohost/hooks"
@ -249,9 +242,6 @@ function use_git()
create_sym_link "/ynh-dev/yunohost/data/helpers.d" "/usr/share/yunohost/helpers.d" create_sym_link "/ynh-dev/yunohost/data/helpers.d" "/usr/share/yunohost/helpers.d"
create_sym_link "/ynh-dev/yunohost/data/other" "/usr/share/yunohost/yunohost-config/moulinette" create_sym_link "/ynh-dev/yunohost/data/other" "/usr/share/yunohost/yunohost-config/moulinette"
# debian
create_sym_link "/ynh-dev/yunohost/debian/conf/pam/mkhomedir" "/usr/share/pam-configs/mkhomedir"
# lib # lib
create_sym_link "/ynh-dev/yunohost/lib/metronome/modules/ldap.lib.lua" "/usr/lib/metronome/modules/ldap.lib.lua" create_sym_link "/ynh-dev/yunohost/lib/metronome/modules/ldap.lib.lua" "/usr/lib/metronome/modules/ldap.lib.lua"
create_sym_link "/ynh-dev/yunohost/lib/metronome/modules/mod_auth_ldap2.lua" "/usr/lib/metronome/modules/mod_auth_ldap2.lua" create_sym_link "/ynh-dev/yunohost/lib/metronome/modules/mod_auth_ldap2.lua" "/usr/lib/metronome/modules/mod_auth_ldap2.lua"
@ -270,52 +260,80 @@ function use_git()
;; ;;
yunohost-admin) yunohost-admin)
getent passwd ynhdev > /dev/null rm -r /var/cache/ynh-dev/yunohost-admin
if [ $? -eq 2 ]; then
useradd ynhdev
fi
mkdir -p /var/cache/ynh-dev/yunohost-admin/ mkdir -p /var/cache/ynh-dev/yunohost-admin/
mkdir -p /var/cache/ynh-dev/yunohost-admin/dist create_sym_link "/ynh-dev/yunohost-admin/app/.env" "/var/cache/ynh-dev/yunohost-admin/.env"
chown -R ynhdev /var/cache/ynh-dev/yunohost-admin/ create_sym_link "/var/cache/ynh-dev/yunohost-admin/node_modules" "/ynh-dev/yunohost-admin/app/node_modules"
create_sym_link "/ynh-dev/yunohost-admin/src/package.json" "/var/cache/ynh-dev/yunohost-admin/package.json" create_sym_link "/ynh-dev/yunohost-admin/app/package.json" "/var/cache/ynh-dev/yunohost-admin/package.json"
create_sym_link "/ynh-dev/yunohost-admin/src/package-lock.json" "/var/cache/ynh-dev/yunohost-admin/package-lock.json" create_sym_link "/ynh-dev/yunohost-admin/app/package-lock.json" "/var/cache/ynh-dev/yunohost-admin/package-lock.json"
create_sym_link "/ynh-dev/yunohost-admin/src" "/usr/share/yunohost/admin" mv "/etc/nginx/conf.d/yunohost_admin.conf.inc" "/etc/nginx/conf.d/yunohost_admin.conf.inc.bkp"
trap 'on_exit' exit
on_exit()
{
mv "/etc/nginx/conf.d/yunohost_admin.conf.inc.bkp" "/etc/nginx/conf.d/yunohost_admin.conf.inc"
systemctl reload nginx
}
if [ ! -e /ynh-dev/yunohost-admin/src/dist ] cd /var/cache/ynh-dev/yunohost-admin/
# Create .env file with the vm ip
# Will be used by webpack-dev-server to proxy api requests.
if [[ ! -e .env ]]
then then
echo "If npm install fails to create the dist and node_modules folder, maybe you need to run the following *in the host!* : " info "Creating .env file"
echo " " IP=$(hostname -I | tr ' ' '\n' | grep "\.")
echo "cd /your/dev/env/yunohost-admin/src/" echo "VUE_APP_IP=$IP" > .env
echo "ln -s /var/cache/ynh-dev/yunohost-admin/dist ./dist"
echo "ln -s /var/cache/ynh-dev/yunohost-admin/node_modules ./node_modules"
echo " "
fi fi
# Install npm dependencies if needed # Allow port 8080 in config file or else the dev server will stop working after postinstall
cd /var/cache/ynh-dev/yunohost-admin/ if [[ ! -e /etc/yunohost/installed ]]
if [ ! -e node_modules/gulp/bin/gulp.js ] then
python3 - <<EOF
import os, yaml
setting_file = "/etc/yunohost/firewall.yml"
assert os.path.exists(setting_file), "Firewall yaml file %s does not exists ?" % setting_file
with open(setting_file) as f:
settings = yaml.load(f)
if 8080 not in settings["ipv4"]["TCP"]:
settings["ipv4"]["TCP"].append(8080)
with open(setting_file, "w") as f:
yaml.safe_dump(settings, f, default_flow_style=False)
EOF
fi
# Install npm if needed
if [[ ! -e node_modules/@vue/cli-service/bin/vue-cli-service.js ]]
then then
info "Installing dependencies to develop in yunohost-admin ..." info "Installing dependencies to develop in yunohost-admin ..."
apt install nodejs npm -y apt install nodejs npm -y
npm install -g npm@latest npm install -g npm@latest
fi fi
# Install dependencies
npm ci --no-bin-links npm ci --no-bin-links
success "Now using Git repository for yunohost-admin" success "Now using Git repository for yunohost-admin"
cat <<EOF > /etc/nginx/conf.d/yunohost_admin.conf.inc
location /yunohost/admin {
proxy_pass https://localhost:8080/yunohost/admin;
proxy_redirect off;
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-Proto \$scheme;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host \$server_name;
proxy_set_header X-Forwarded-Port \$server_port;
cd /ynh-dev/yunohost-admin/src proxy_http_version 1.1;
su ynhdev -c "./node_modules/gulp/bin/gulp.js build --dev" proxy_set_header Upgrade \$http_upgrade;
warn "-------------------------------------------------------- " proxy_set_header Connection "upgrade";
warn "Launching gulp ... " }
warn "NB : This command will keep running and watch for changes" EOF
warn " in the folder /ynh-dev/yunohost-admin/src, such that you"
warn "don't need to re-run npm yourself everytime you change "
warn "something ! "
warn "-------------------------------------------------------- "
su ynhdev -c "./node_modules/gulp/bin/gulp.js watch --dev"
systemctl reload nginx
cd /ynh-dev/yunohost-admin/app/
npm run serve
;;
*)
error "Invalid package '${PACKAGES[i]}': correct arguments are 'yunohost', 'ssowat', 'moulinette', 'yunohost-admin' or nothing for all"
;; ;;
*) *)
error "Invalid package '${PACKAGES[i]}': correct arguments are 'yunohost', 'ssowat', 'moulinette', 'yunohost-admin' or nothing for all" error "Invalid package '${PACKAGES[i]}': correct arguments are 'yunohost', 'ssowat', 'moulinette', 'yunohost-admin' or nothing for all"
@ -328,7 +346,7 @@ function run_tests()
{ {
assert_inside_vm assert_inside_vm
local PACKAGES="$@" local PACKAGES="$@"
for PACKAGE in "$PACKAGES"; for PACKAGE in $PACKAGES;
do do
TEST_FUNCTION=$(echo "$PACKAGE" | tr '/:' ' ' | awk '{print $3}') TEST_FUNCTION=$(echo "$PACKAGE" | tr '/:' ' ' | awk '{print $3}')
TEST_MODULE=$(echo "$PACKAGE" | tr '/:' ' ' | awk '{print $2}') TEST_MODULE=$(echo "$PACKAGE" | tr '/:' ' ' | awk '{print $2}')
@ -339,14 +357,14 @@ function run_tests()
# Pytest and tests dependencies # Pytest and tests dependencies
if ! type "pytest" > /dev/null 2>&1; then if ! type "pytest" > /dev/null 2>&1; then
info "> Installing pytest ..." info "> Installing pytest ..."
apt-get install python-pip -y apt-get install python3-pip -y
pip2 install pytest pytest-sugar pip3 install pytest pytest-sugar
fi fi
for DEP in "pytest-mock requests-mock mock" for DEP in "pytest-mock requests-mock mock"
do do
if [ -z "$(pip show $DEP)" ]; then if [ -z "$(pip3 show $DEP)" ]; then
info "Installing $DEP with pip" info "Installing $DEP with pip3"
pip2 install $DEP pip3 install $DEP
fi fi
done done