1
0
Fork 0
mirror of https://github.com/YunoHost-Apps/flohmarkt_ynh.git synced 2024-09-03 18:36:30 +02:00

Merge pull request #13 from YunoHost-Apps/testing

…finally
This commit is contained in:
Chris Vogel 2024-05-16 12:55:11 +02:00 committed by GitHub
commit d32575780a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 1364 additions and 415 deletions

1
.gitignore vendored Normal file
View file

@ -0,0 +1 @@
README*

View file

@ -1,6 +1,7 @@
# All available README files by language # All available README files by language
- [Read the README in English](README.md) - [Read the README in English](README.md)
- [Lee el README en español](README_es.md)
- [Irakurri README euskaraz](README_eu.md) - [Irakurri README euskaraz](README_eu.md)
- [Lire le README en français](README_fr.md) - [Lire le README en français](README_fr.md)
- [Le o README en galego](README_gl.md) - [Le o README en galego](README_gl.md)

View file

@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
* [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart) * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
**Shipped version:** 0.0~ynh4 **Shipped version:** 0.0~ynh5
**Demo:** <https://flohmarkt.ween.de/> **Demo:** <https://flohmarkt.ween.de/>
@ -61,26 +61,6 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
![Screenshot of flohmarkt](./doc/screenshots/screenshot.png) ![Screenshot of flohmarkt](./doc/screenshots/screenshot.png)
## Disclaimers / important information
## support
For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)
## :red_circle: Antifeatures ## :red_circle: Antifeatures
- **Alpha software**: Early development stage. May contain changing or unstable features, bugs, and security vulnerability. - **Alpha software**: Early development stage. May contain changing or unstable features, bugs, and security vulnerability.

View file

@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
* [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart) * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
**Paketatutako bertsioa:** 0.0~ynh4 **Paketatutako bertsioa:** 0.0~ynh5
**Demoa:** <https://flohmarkt.ween.de/> **Demoa:** <https://flohmarkt.ween.de/>
@ -61,26 +61,6 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
![flohmarkt(r)en pantaila-argazkia](./doc/screenshots/screenshot.png) ![flohmarkt(r)en pantaila-argazkia](./doc/screenshots/screenshot.png)
## Ezespena / informazio garrantzitsua
## support
For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)
## :red_circle: Ezaugarri zalantzagarriak ## :red_circle: Ezaugarri zalantzagarriak
- **Alfa softwarea**: Garapenaren hasierako fasean dago. Ezaugarri aldakor edo ezegonkorrak, erroreak eta segurtasuneko arazoak izan ditzazke. - **Alfa softwarea**: Garapenaren hasierako fasean dago. Ezaugarri aldakor edo ezegonkorrak, erroreak eta segurtasuneko arazoak izan ditzazke.

View file

@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
* [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart) * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
**Version incluse:** 0.0~ynh4 **Version incluse:** 0.0~ynh5
**Démo:** <https://flohmarkt.ween.de/> **Démo:** <https://flohmarkt.ween.de/>
@ -61,26 +61,6 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
![Capture décran de flohmarkt](./doc/screenshots/screenshot.png) ![Capture décran de flohmarkt](./doc/screenshots/screenshot.png)
## Avertissements / informations importantes
## support
For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)
## :red_circle: Anti-fonctionnalités ## :red_circle: Anti-fonctionnalités
- **Logiciel en version alpha **: Le logiciel est au tout début de son développement. Il pourrait contenir des fonctionnalités changeantes ou instables, des bugs, et des failles de sécurité. - **Logiciel en version alpha **: Le logiciel est au tout début de son développement. Il pourrait contenir des fonctionnalités changeantes ou instables, des bugs, et des failles de sécurité.

View file

@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
* [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart) * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
**Versión proporcionada:** 0.0~ynh4 **Versión proporcionada:** 0.0~ynh5
**Demo:** <https://flohmarkt.ween.de/> **Demo:** <https://flohmarkt.ween.de/>
@ -61,26 +61,6 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
![Captura de pantalla de flohmarkt](./doc/screenshots/screenshot.png) ![Captura de pantalla de flohmarkt](./doc/screenshots/screenshot.png)
## Avisos / información importante
## support
For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)
## :red_circle: Debes considerar ## :red_circle: Debes considerar
- **Alpha software**: Early development stage. May contain changing or unstable features, bugs, and security vulnerability. - **Alpha software**: Early development stage. May contain changing or unstable features, bugs, and security vulnerability.

View file

@ -53,7 +53,7 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
* [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart) * [Service compatibility chart](https://codeberg.org/flohmarkt/flohmarkt/wiki/Service-compatibility-chart)
**分发版本:** 0.0~ynh4 **分发版本:** 0.0~ynh5
**演示:** <https://flohmarkt.ween.de/> **演示:** <https://flohmarkt.ween.de/>
@ -61,26 +61,6 @@ Generally the [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki) is a good so
![flohmarkt 的截图](./doc/screenshots/screenshot.png) ![flohmarkt 的截图](./doc/screenshots/screenshot.png)
## 免责声明 / 重要信息
## support
For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)
## :red_circle: 负面特征 ## :red_circle: 负面特征
- **Alpha software**: Early development stage. May contain changing or unstable features, bugs, and security vulnerability. - **Alpha software**: Early development stage. May contain changing or unstable features, bugs, and security vulnerability.

View file

@ -1,6 +1,3 @@
[chttpd] [chttpd]
; prevent non admin users to be able to access the list of all databases ; prevent non admin users to be able to access the list of all databases
admin_only_all_dbs = true admin_only_all_dbs = true
; Defines the port number to listen:
port = __PORT_COUCHDB__

View file

@ -6,7 +6,12 @@ JwtSecret = __JWTSECRET__
DataPath = __DATA_DIR__ DataPath = __DATA_DIR__
[Database] [Database]
Server = http://__APP__:__PASSWORD_COUCHDB_FLOHMARKT__@127.0.0.1:__PORT_COUCHDB__/ UseHttps = 0
Host = 127.0.0.1
User = __APP__
Password = __PASSWORD_COUCHDB_FLOHMARKT__
Port = 5984
Database = __APP__
[SMTP] [SMTP]
Server = 127.0.0.1 Server = 127.0.0.1

View file

@ -6,10 +6,13 @@ After=network.target couchdb.service
Type=simple Type=simple
User=__APP__ User=__APP__
Group=__APP__ Group=__APP__
WorkingDirectory=__INSTALL_DIR__/__APP__ WorkingDirectory=__FLOHMARKT_APP_DIR__
Environment="VENV_DIR=__INSTALL_DIR__/venv/" Environment="VENV_DIR=__FLOHMARKT_VENV_DIR__"
ExecStart=/bin/bash -c "/opt/flohmarkt/venv/bin/uvicorn --host 127.0.0.1 --port 8000 --reload flohmarkt.web:start 2>&1 | /usr/bin/ts '%%Y-%%m-%%d %%H:%%M:%%S'" ExecStart=/bin/bash -c "__FLOHMARKT_VENV_DIR__/bin/uvicorn --factory --host 127.0.0.1 --port __PORT__ flohmarkt.web:start 2>&1 | /usr/bin/ts '%%Y-%%m-%%d %%H:%%M:%%S'"
StandardOutput=append:/var/log/__APP__/__APP__.log Restart=on-failure
RestartSec=5s
# nicer logfile naming https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/38
StandardOutput=append:__FLOHMARKT_LOGFILE__
StandardError=inherit StandardError=inherit
# Sandboxing options to harden security # Sandboxing options to harden security

View file

@ -1,7 +1,27 @@
After installation, look at /var/log/flohmarkt/flohmarkt.log to find the setup URL. The message should look like this: ## support
``` For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
Flohmarkt is not initialized yet. Please go to
https://flohmarkt.test/setup/<secret string> To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
in order to complete the setup process
``` ## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)
## removing this app
Removing a flohmarkt instance will **delete all of its data** for the time being. If you remove a flohmarkt **we strongly suggest to make a backup** beforehand. This [issue](https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/29) describes how we'll handle `remove` with and without `--purge` in the future.
**Warning:** This might break any existing installation of couchdb (there's an couchdb app to install just couchdb and expose its port via nginx reverse-proxy and possibly other software installing a couchdb). This could happen if you installed the couchdb app after you installed flohmarkt.
https://codeberg.org/flohmarkt/flohmarkt_ynh/src/commit/7721103bac61787f31a4b2f2ae695c65d4f26fc9/scripts/remove#L44
When installing flohmarkt on a a domain and letting it talk to other ActivityPub instances it will propagate a key associated to your domain. If you remove your flohmarkt from that domain and loose that key other instances might not want to talk to you anymore after you installed flohmarkt again on the same domain generating a new key.

View file

@ -23,30 +23,33 @@ Your opinion is appreciated on this topic.
The codeberg repository is meant to be bleeding edge and we'll try to follow the HEAD of [flohmarkts](https://codeberg.org/flohmarkt/flohmarkt) repository closely. The codeberg repository is meant to be bleeding edge and we'll try to follow the HEAD of [flohmarkts](https://codeberg.org/flohmarkt/flohmarkt) repository closely.
Once in a while we'll reach the point to tag a new `-ynhX` version for changes of the yunohost integration or we'll want to release a new version of flohmarkt or both. Once in a while we'll reach the point to tag a new `~ynhX` version for changes of the yunohost integration or we'll want to release a new version of flohmarkt or both.
To do so we'll push the according changes to the [flohmarkt repository at github](https://github.com/YunoHost-Apps/flohmarkt_ynh) to make upgrades available to the yunohost community. To do so we'll push the according changes to the [flohmarkt repository at github](https://github.com/YunoHost-Apps/flohmarkt_ynh) to make upgrades available to the yunohost community.
One **pitfall** doing so is that we can't rely on the yunohost CI for testing for our codeberg repository this way. If need'll be and developers would like to use yunohost as their base for active work on flohmarkt we might release another app **flohmarkt-devel_ynh** in future that closely follows the repository we use for development. One **pitfall** doing so is that we can't rely on the yunohost CI for testing for our codeberg repository this way. If need'll be and developers would like to use yunohost as their base for active work on flohmarkt we might release another app **flohmarkt-devel_ynh** in future that closely follows the repository we use for development.
#### master and main #### master, main and testing
The development startet on codeberg on a branch named **main**. That worked all the way until the repository got mirrored to github to be included into the yunohost app catalog. The last change that needed to be done for flohmarkt to appear on the catalog has been to change the name of the branch to **master** for the workflows on github to recognize it for catalog inclusion. The development startet on codeberg on a branch named **main**. That worked all the way until the repository got mirrored to github to be included into the yunohost app catalog. The last change that needed to be done for flohmarkt to appear on the catalog has been to change the name of the branch to **master** for the workflows on github to recognize it for catalog inclusion.
Short: for **historic** reason we use the branch **main** on codeberg and publish the versions for the catalog as **master** on github. Short: for **historic** reason we use the branch **main** on codeberg and publish the versions for the catalog as **master** on github.
Because of https://github.com/YunoHost-Apps/flohmarkt_ynh/pull/2#issuecomment-2081108150 we decided to name the working branch on codeberg **testing** which also signals that this is not the place to install from if testing and development is not wanted.
### pushing to github ### pushing to github
* make sure the local git and the codeberg git are in sync on their main branch * make sure the local git and the codeberg git are in sync on their testing branch
* tag a new version on codeberg `<major>.<minor>-ynh<X>`, e.g. `0.01-ynh5`. `<major>.<minor>` is the flohmarkt version. `ynhX` is the version of the integration into flohmarkt (this repo). * tag a new version on codeberg `<major>.<minor>~ynh<X>`, e.g. `0.01~ynh5`. `<major>.<minor>` is the flohmarkt version. `ynhX` is the version of the integration into flohmarkt (this repo).
* new flohmarkt version: only `manifest.toml` changed to point to the newer source archive * new flohmarkt version: `manifest.toml` changed to point to the newer source archive
→ change `<major>.<minor>-ynh<X>` according to the new flohmarkt version → change `<major>.<minor>~ynh<X>` according to the new flohmarkt version
* changes in integration: scripts, conf files or `doc/*` changed → reset `X` in `~ynh<X>` to `0`
* **only** changes in integration, **no new flohmarkt version in manifest.toml**
→ increment `<X>` to signal a new version of the yunohost integration → increment `<X>` to signal a new version of the yunohost integration
* try to push the local main branch to github which might fail * push the local testing branch to github, if this fails:
* there might for some reason exist an old main branch that had not been deleted after the PR to the github master branch - check carefully and delete the existing main branch * there might for some reason exist an old testing branch that had not been deleted after the PR to the github master branch - check carefully and delete the existing testing branch
* on github open an PR from the main branch into the master branch * on github open an PR from the testing branch into the master branch
* the PR can be tested on the CI workflow if a comment containing `!testme` is added to the PR * test the PR on the CI workflow by commenting it containing `!testme` on one line by itself to the PR
* the PR will be included after * the PR will be included after
* it successfully ran through the CI workflow (results will show up inside the PR) * it successfully ran through the CI workflow (results will show up inside the PR)
* it has been reviewed * it has been reviewed

View file

@ -1,17 +0,0 @@
## support
For questions about running flohmarkt on yunohost please use the [yunohost support matrix channel](https://yunohost.org/en/chat_rooms#help-and-support-chat-roo) or the [forum](https://forum.yunohost.org/t/ynh-flohmarkt-flohmarkt-as-an-app-for-yunohost/28455?u=chrichri).
To get help for **flohmarkt** itself please look at its [wiki](https://codeberg.org/flohmarkt/flohmarkt/wiki), [open an issue](https://codeberg.org/flohmarkt/flohmarkt/issues) with your request or join the [IRC channel flohmarkt](https://web.libera.chat/?nick=GithubGuest?#flohmarkt) on [libera.chat](https://libera.chat/).
## bugs, requesting features
Please use the issue tracker at https://codeberg.org/flohmarkt/flohmarkt_ynh/issues
## upstream repository
The yunohost integration of **flohmarkt** is developed by the flohmarkt organization on https://codeberg.org/ .
Please refer to https://codeberg.org/flohmarkt/flohmarkt_ynh/ to take part in development.
More about development of **flohmarkt**s yunohost integration can be found in [DEVELOPMENT.md](DEVELOPMENT.md)

13
doc/POST_INSTALL.md Normal file
View file

@ -0,0 +1,13 @@
## finishing setup
To finish your flohmarkt setup you'll need to open its setup URL.
You can find the setup URL in flohmarkts logfile in /var/log/__APP__ if you look for entries like this:
```
2024-05-06 16:30:24 Flohmarkt is not initialized yet. Please go to
2024-05-06 16:30:24 https://YOUR_URL/setup/SECRET
2024-05-06 16:30:24 in order to complete the setup process
```
(YOUR_URL will reflect the DOMAIN/PATH you chose for installation and SECRET is a random string)

View file

@ -1,25 +1,33 @@
## Warning: Exclusive use of couchdb ## use of couchdb
flohmarkt expects to install CouchDB from the Apache repository for its own, exclusive use. Installation might break already existing installs of CouchDB. flohmarkt expects to install CouchDB from the Apache repository for its use. Installation of flohmarkt might break already existing installs of CouchDB.
**flohmarkt will probably not install if you're already running an instance of a couchdb** **flohmarkt will probably not install if you're already running an instance of a couchdb**
https://codeberg.org/flohmarkt/flohmarkt_ynh/src/commit/7721103bac61787f31a4b2f2ae695c65d4f26fc9/scripts/install#L9 https://codeberg.org/flohmarkt/flohmarkt_ynh/src/commit/7721103bac61787f31a4b2f2ae695c65d4f26fc9/scripts/install#L9
https://codeberg.org/ChriChri/flohmarkt_ynh/issues/9 https://codeberg.org/ChriChri/flohmarkt_ynh/issues/9
## Exclusive use of (sub)domain ## choosing a sender email address
flohmarkt expects to bei installed on its own (sub)domain. flohmarkt will send email and you'll be asked for an address used as the sender in these emails. yunohost will try to configure your chosen address as an allowed sender, but for the time being **this will only work if the sender address is not in use otherwise** - e.g. as an alias.
https://codeberg.org/ChriChri/flohmarkt_ynh/issues/4 . This [issue](https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/50) contains information about the problem.
## No integration in yunohost user database" ## multiple flohmarkt on the same subdomain
The installation will allow you to **test** this. The feature is not well tested, yet, and installing multiple productive flohmarkts in the same domain might not work on the fediverse.
Feedback is more than welcome!
## No integration in yunohost user database
flohmarkt mainanins its own user database in CouchDB. Users have to register to flohmarkt to get an account. Registration cannot be restricted to yunohost users. flohmarkt mainanins its own user database in CouchDB. Users have to register to flohmarkt to get an account. Registration cannot be restricted to yunohost users.
https://codeberg.org/ChriChri/flohmarkt_ynh/issues/5 . https://codeberg.org/ChriChri/flohmarkt_ynh/issues/5 .
## removing after installation ## removing
Removing a flohmarkt instance will **delete all of its data** for the time being. If you remove a flohmarkt **we strongly suggest to make a backup** beforehand. This [issue](https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/29) describes how we'll handle `remove` with and without `--purge` in the future.
**Warning:** This might break any existing installation of couchdb (there's an couchdb app to install just couchdb and expose its port via nginx reverse-proxy and possibly other software installing a couchdb). This could happen if you installed the couchdb app after you installed flohmarkt. **Warning:** This might break any existing installation of couchdb (there's an couchdb app to install just couchdb and expose its port via nginx reverse-proxy and possibly other software installing a couchdb). This could happen if you installed the couchdb app after you installed flohmarkt.

View file

@ -1,27 +0,0 @@
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"json": {
"()": "pythonjsonlogger.jsonlogger.JsonFormatter",
"fmt": "%(asctime)s %(name)s %(levelname)s %(message)s"
}
},
"handlers": {
"stderr": {
"formatter": "json",
"class": "logging.StreamHandler",
"stream": "ext://sys.stderr"
},
"stdout": {
"formatter": "json",
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout"
}
},
"loggers": {
"uvicorn": {"handlers": ["stderr"], "level": "INFO", "propagate": false},
"uvicorn.error": {"level": "INFO"},
"uvicorn.access": {"handlers": ["stdout"], "level": "INFO", "propagate": false}
}
}

View file

@ -5,8 +5,9 @@ packaging_format = 2
id = "flohmarkt" id = "flohmarkt"
name = "flohmarkt" name = "flohmarkt"
description.en = "A decentral federated small ads platform" description.en = "A decentral federated small ads platform"
description.fr = "Plateforme de petites annonces fédérées décentralisées"
version = "0.0~ynh4" version = "0.0~ynh5"
maintainers = ["Chris Vogel"] maintainers = ["Chris Vogel"]
@ -28,8 +29,7 @@ code = "https://codeberg.org/flohmarkt/flohmarkt"
yunohost = ">= 11.2.11" yunohost = ">= 11.2.11"
architectures = "all" architectures = "all"
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/11 # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/11
# multi_instance = true multi_instance = true
multi_instance = false
# the "ldap" key corresponds to wether or not a user *can* login on the app using # the "ldap" key corresponds to wether or not a user *can* login on the app using
# its YunoHost credentials. # its YunoHost credentials.
@ -45,7 +45,7 @@ ldap = false
sso = false sso = false
# **estimate** minimum disk and RAM requirements # **estimate** minimum disk and RAM requirements
disk = "100M" disk = "200M"
ram.build = "200M" ram.build = "200M"
ram.runtime = "100M" ram.runtime = "100M"
@ -54,13 +54,12 @@ ram.runtime = "100M"
# ask admin on which domain to run flohmarkt # ask admin on which domain to run flohmarkt
type = "domain" type = "domain"
[install.path]
# ask admin under which path flohmarkt will be reachable
# e.g. 'https://doma.in/path' - might not work, needs to be tested:
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/4 # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/4
# [install.path] type = "path"
# # ask admin under which path flohmarkt will be reachable default = "/"
# # e.g. 'https://doma.in/path' - might not work, needs to be tested:
# # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/4
# type = "path"
# default = "/"
[install.init_main_permission] [install.init_main_permission]
# who will be able to access the apps URL after installation? # who will be able to access the apps URL after installation?
@ -109,8 +108,8 @@ ram.runtime = "100M"
# This will pre-fetch the asset which can then be deployed during the install/upgrade scripts with : # This will pre-fetch the asset which can then be deployed during the install/upgrade scripts with :
# ynh_setup_source --dest_dir="$install_dir" # ynh_setup_source --dest_dir="$install_dir"
# You can also define other assets than "main" and add --source_id="foobar" in the previous command # You can also define other assets than "main" and add --source_id="foobar" in the previous command
url = "https://codeberg.org/flohmarkt/flohmarkt/archive/4a3f243d090549c66ffece46222b0f1cc358e91e.tar.gz" url = "https://codeberg.org/flohmarkt/flohmarkt/archive/78b2cc137d7c88d961e12dcb5017b17257bc4a71.tar.gz"
sha256 = "d59bb3bbbaf7c98d9f8bae46d9a2b2ccc407d674f51140a3d868a6afdf862e73" sha256 = "4acbfe20372583fdf75f4eea7b9c29d9ae20d04283559578f5e33f8347570c33"
# These infos are used by https://github.com/YunoHost/apps/blob/master/tools/autoupdate_app_sources/autoupdate_app_sources.py # These infos are used by https://github.com/YunoHost/apps/blob/master/tools/autoupdate_app_sources/autoupdate_app_sources.py
# to auto-update the previous asset urls and sha256sum + manifest version # to auto-update the previous asset urls and sha256sum + manifest version
@ -133,11 +132,8 @@ ram.runtime = "100M"
# @@ how to create a couchdb user if needed? in install-script manually? # @@ how to create a couchdb user if needed? in install-script manually?
[resources.install_dir] [resources.install_dir]
dir = "/opt/flohmarkt"
[resources.data_dir] [resources.data_dir]
# This will create/remove the data dir as /home/yunohost.app/$app
# and store the corresponding setting $data_dir
[resources.permissions] [resources.permissions]
# This will configure SSOwat permission for $domain/$path/ # This will configure SSOwat permission for $domain/$path/
@ -151,13 +147,16 @@ ram.runtime = "100M"
# uvicorn running the flohmarkt app will be started listening to this port # uvicorn running the flohmarkt app will be started listening to this port
# if 'main.default' is already in use another random port will be used # if 'main.default' is already in use another random port will be used
main.default = 8000 main.default = 8000
couchdb.default = 5984 # we'll rely on couchdb choosing the default port to listen to
# otherwise I'll set a new port for each flohmarkt that will be installed until the
# installation of couchdb is reworked by making it possible to do as a pure dependency
# couchdb.default = 5984
[resources.apt] [resources.apt]
# python dependencies shall be installed in a venv using pip. # python dependencies shall be installed in a venv using pip.
# moreutils is needed for `ts` used in systemd.service # moreutils is needed for `ts` used in systemd.service
packages = "python3-pip python3-full curl apt-transport-https gnupg moreutils" packages = "python3-pip python3-full curl apt-transport-https gnupg moreutils curl"
# repo for couchdb - doesn't work, yet # repo for couchdb - doesn't work, yet
# extras.couchdb.repo = "deb https://apache.jfrog.io/artifactory/couchdb-deb/ __YNH_DEBIAN_VERSION__ main" # extras.couchdb.repo = "deb https://apache.jfrog.io/artifactory/couchdb-deb/ __YNH_DEBIAN_VERSION__ main"
# extras.couchdb.key = "https://couchdb.apache.org/repo/keys.asc" # extras.couchdb.key = "https://couchdb.apache.org/repo/keys.asc"

View file

@ -1,55 +0,0 @@
/opt/flohmarkt/venv/bin/uvicorn --host 127.0.0.1 --port 8000 --reload --log-config logger.json flohmarkt.web:start
Fehler:
Traceback (most recent call last):
File "/usr/lib/python3.9/logging/config.py", line 385, in resolve
found = self.importer(used)
ModuleNotFoundError: No module named 'pythonjsonlogger'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.9/logging/config.py", line 543, in configure
formatters[name] = self.configure_formatter(
File "/usr/lib/python3.9/logging/config.py", line 655, in configure_formatter
result = self.configure_custom(config)
File "/usr/lib/python3.9/logging/config.py", line 470, in configure_custom
c = self.resolve(c)
File "/usr/lib/python3.9/logging/config.py", line 398, in resolve
raise v
File "/usr/lib/python3.9/logging/config.py", line 385, in resolve
found = self.importer(used)
ValueError: Cannot resolve 'pythonjsonlogger.jsonlogger.JsonFormatter': No module named 'pythonjsonlogger'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/flohmarkt/venv/bin/uvicorn", line 8, in <module>
sys.exit(main())
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/main.py", line 404, in main
run(
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/main.py", line 506, in run
config = Config(
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/config.py", line 299, in __init__
self.configure_logging()
File "/opt/flohmarkt/venv/lib/python3.9/site-packages/uvicorn/config.py", line 411, in configure_logging
logging.config.dictConfig(loaded_config)
File "/usr/lib/python3.9/logging/config.py", line 809, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python3.9/logging/config.py", line 546, in configure
raise ValueError('Unable to configure '
ValueError: Unable to configure formatter 'json'
(venv) root@yt:/opt/flohmarkt/flohmarkt# pip install python-json-logger
Requirement already satisfied: python-json-logger in /usr/lib/python3.9/site-packages (2.0.7)
Ausgabe in json will ich eigentlich auch nicht falls das in der config steht. Der Kram ist aber so dokumentiert, dass ich ihn nicht verstehe...

View file

@ -4,10 +4,232 @@
# COMMON VARIABLES # COMMON VARIABLES
#================================================= #=================================================
## new filenames starting 0.00~ynh5
# make a filename/service name from domain/path
if [[ "$path" == /* ]]; then
url_path="${path:1}"
fi
if [[ "__${url_path}__" == '____' ]]; then
flohmarkt_filename="$domain"
else
flohmarkt_filename="$domain-${url_path}"
fi
# this filename is used for logfile name and systemd.service name
# and for symlinking install_dir and data_dir
flohmarkt_filename="${YNH_APP_ID}_${flohmarkt_filename//[^A-Za-z0-9._-]/_}"
# directory flohmarkts software is installed to
# contains ./venv and ./src as sub-directories
flohmarkt_install="$install_dir"
flohmarkt_sym_install="$( dirname $flohmarkt_install )/$flohmarkt_filename"
flohmarkt_venv_dir="${flohmarkt_install}/venv"
flohmarkt_app_dir="${flohmarkt_install}/app"
# directory containing logfiles
flohmarkt_log_dir="/var/log/${app}"
flohmarkt_sym_log_dir="/var/log/${flohmarkt_filename}"
# filename for logfiles - ¡ojo! if not ends with .log will be interpreted
# as a directory by ynh_use_logrotate
# https://github.com/YunoHost/issues/issues/2383
flohmarkt_logfile="${flohmarkt_log_dir}/app.log"
# flohmarkt data_dir
flohmarkt_data_dir="$data_dir"
flohmarkt_sym_data_dir="$( dirname $flohmarkt_data_dir )/$flohmarkt_filename"
## old filenames before 0.00~ynh5 - for reference and needed to
# migrate (see below)
flohmarkt_old_install="/opt/flohmarkt"
flohmarkt_old_venv_dir="${flohmarkt_old_install}/venv"
flohmarkt_old_app_dir="${flohmarkt_old_install}/flohmarkt"
flohmarkt_old_log_dir="/var/log/flohmarkt/"
flohmarkt_old_service="flohmarkt"
#================================================= #=================================================
# PERSONAL HELPERS # PERSONAL HELPERS
#================================================= #=================================================
# set file permissions and owner for installation
flohmarkt_ynh_set_permission() {
# install dir - only root needs to write and $app reads
chown root:$app -R "$flohmarkt_install"
chmod g-w,o-rwx -R "$flohmarkt_install"
}
# start flohmarkt service
flohmarkt_ynh_start_service() {
ynh_systemd_action --service_name=$flohmarkt_filename --action="start" \
--line_match='INFO: *Application startup complete.' --log_path="$flohmarkt_logfile" \
--timeout=30
}
# stop flohmarkt service
flohmarkt_ynh_stop_service() {
ynh_systemd_action --service_name=$flohmarkt_filename --action="stop"
}
# start couchdb and wait for success
flohmarkt_ynh_start_couchdb() {
ynh_systemd_action --service_name=couchdb --action="start" --timeout=30 \
--log_path="/var/log/couchdb/couchdb.log" \
--line_match='Apache CouchDB has started on http://127.0.0.1'
}
# stop couchdb
flohmarkt_ynh_stop_couchdb() {
ynh_systemd_action --service_name=couchdb --action="stop" --timeout=30 \
--log_path="/var/log/couchdb/couchdb.log" \
--line_match='SIGTERM received - shutting down'
}
# install or upgrade couchdb
flohmarkt_ynh_up_inst_couchdb() {
echo "\
couchdb couchdb/mode select standalone
couchdb couchdb/mode seen true
couchdb couchdb/bindaddress string 127.0.0.1
couchdb couchdb/bindaddress seen true
couchdb couchdb/cookie string $couchdb_magic_cookie
couchdb couchdb/adminpass password $password_couchdb_admin
couchdb couchdb/adminpass seen true
couchdb couchdb/adminpass_again password $password_couchdb_admin
couchdb couchdb/adminpass_again seen true" | debconf-set-selections
DEBIAN_FRONTEND=noninteractive # apt-get install -y --force-yes couchdb
ynh_install_extra_app_dependencies \
--repo="deb https://apache.jfrog.io/artifactory/couchdb-deb/ $(lsb_release -c -s) main" \
--key="https://couchdb.apache.org/repo/keys.asc" \
--package="couchdb"
}
flohmarkt_ynh_dump_couchdb() {
../settings/scripts/couchdb-dump/couchdb-dump.sh -b -H 127.0.0.1 -d "${app}" \
-q -u admin -p "${password_couchdb_admin}" -f "${YNH_CWD}/${app}.json"
}
flohmarkt_ynh_import_couchdb() {
ls -l ../settings/scripts/couchdb-dump/couchdb-dump.sh ${YNH_CWD}/${app}.json
../settings/scripts/couchdb-dump/couchdb-dump.sh -r -c -H 127.0.0.1 -d "${app}" \
-q -u admin -p "${password_couchdb_admin}" -f "${YNH_CWD}/${app}.json"
}
flohmarkt_ynh_delete_couchdb_user() {
# https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/46 - more than one revision?
local couchdb_user_revision=$( curl -sX GET "http://127.0.0.1:5984/_users/org.couchdb.user%3A${app}" \
--user "admin:${password_couchdb_admin}" | jq -r ._rev )
curl -s -X DELETE "http://127.0.0.1:5984/_users/org.couchdb.user%3A${app}?rev=${couchdb_user_revision}" \
--user "admin:${password_couchdb_admin}"
}
flohmarkt_ynh_delete_couchdb_db() {
curl -s -X DELETE "http://127.0.0.1:5984/${app}" --user "admin:${password_couchdb_admin}"
}
flohmarkt_ynh_create_couchdb_user() {
curl -s -X PUT "http://127.0.0.1:5984/_users/org.couchdb.user:${app}" --user "admin:${password_couchdb_admin}"\
-H "Accept: application/json" -H "Content-Type: application/json" \
-d "{\"name\": \"${app}\", \"password\": \"${password_couchdb_flohmarkt}\", \"roles\": [], \"type\": \"user\"}"
# @@ check answer something like
# {"ok":true,"id":"org.couchdb.user:flohmarkt","rev":"35-9865694604ab384388eea0f978a6e728"}
}
flohmarkt_ynh_couchdb_user_permissions() {
curl -s -X PUT "http://127.0.0.1:5984/${app}/_security" --user "admin:${password_couchdb_admin}"\
-H "Accept: application/json" -H "Content-Type: application/json" \
-d "{\"members\":{\"names\": [\"${app}\"],\"roles\": [\"editor\"]}}"
}
flohmarkt_ynh_exists_couchdb_user() {
if [[ $( curl -sX GET "http://127.0.0.1:5984/_users/org.couchdb.user%3A${app}" \
--user "admin:${password_couchdb_admin}" | jq .error ) == '"not_found"' ]]
then
false
else
true
fi
}
flohmarkt_ynh_exists_couchdb_db() {
if [[ $( curl -sX GET "http://127.0.0.1:5984/flohmarkt__22" --user "admin:${password_couchdb_admin}" \
| jq .error ) == '"not_found"' ]]
then
false
else
true
fi
}
# check whether old couchdb user or database exist before creating the new ones
flohmarkt_ynh_check_old_couchdb() {
if flohmarkt_ynh_exists_couchdb_user; then
ynh_die --ret_code=100 --message="CouchDB user '$app' exists already. Stopping install."
elif flohmarkt_ynh_exists_couchdb_db; then
ynh_die --ret_code=100 --message="CouchDB database '$app' exists already. Stopping install."
fi
}
flohmarkt_ynh_restore_couchdb() {
flohmarkt_ynh_check_old_couchdb
flohmarkt_ynh_import_couchdb
flohmarkt_ynh_create_couchdb_user
flohmarkt_ynh_couchdb_user_permissions
}
# create venv
flohmarkt_ynh_create_venv() {
python3 -m venv --without-pip "$flohmarkt_venv_dir"
}
# install requirements.txt in venv
flohmarkt_ynh_venv_requirements() {
(
set +o nounset
source "$flohmarkt_venv_dir/bin/activate"
set -o nounset
set -x
$flohmarkt_venv_dir/bin/python3 -m ensurepip
$flohmarkt_venv_dir/bin/pip3 install -r "$flohmarkt_app_dir/requirements.txt"
)
}
# move files and directories to their new places
flohmarkt_ynh_upgrade_path_ynh5() {
# flohmarkt and couchdb are already stopped in upgrade script
# move app_dir into new 'app' folder
mv "$flohmarkt_install/flohmarkt" "$flohmarkt_app_dir"
# yunohost seems to move the venv dir automatically, but this
# doesn't work, because the paths inside the venv are not adjusted
# delete the old, not working venv and create a new one:
ynh_secure_remove --file="$flohmarkt_venv_dir"
flohmarkt_ynh_create_venv
flohmarkt_ynh_venv_requirements
# remove old $install_dir
ynh_secure_remove --file="$flohmarkt_old_install"
# move logfile directory
mkdir -p "$flohmarkt_log_dir"
# remove systemd.service - will be generated newly by upgrade
# ynh_remove_systemd_config --service="$flohmarkt_old_service"
ynh_systemd_action --action=stop --service_name="$flohmarkt_old_service"
ynh_systemd_action --action=disable --service_name="$flohmarkt_old_service"
ynh_secure_remove --file="/etc/systemd/system/multi-user.target.wants/flohmarkt.service"
ynh_secure_remove --file="/etc/systemd/system/flohmarkt.service"
# funktioniert nicht? issue?
#ynh_systemd_action --action=daemon-reload
# DEBUG + systemctl daemon-reload flohmarkt
# WARNING Too many arguments.
systemctl daemon-reload
# unit flohmarkt is automatically appended and therefor this fails:
#ynh_systemd_action --action=reset-failed
systemctl reset-failed
# create symlinks
ln -s "$flohmarkt_install" "$flohmarkt_sym_install"
ln -s "$flohmarkt_data_dir" "$flohmarkt_sym_data_dir"
}
#================================================= #=================================================
# EXPERIMENTAL HELPERS # EXPERIMENTAL HELPERS
#================================================= #=================================================

View file

@ -4,20 +4,23 @@ source ../settings/scripts/_common.sh
source /usr/share/yunohost/helpers source /usr/share/yunohost/helpers
ynh_print_info --message="Declaring files to be backed up..." ynh_print_info --message="Declaring files to be backed up..."
ynh_backup --src_path="$install_dir" ynh_backup --src_path="$flohmarkt_install"
ynh_backup --src_path="/etc/nginx/conf.d/$domain.d/$app.conf" ynh_backup --src_path="/etc/nginx/conf.d/$domain.d/$app.conf"
ynh_backup --src_path="/etc/fail2ban/jail.d/$app.conf" ynh_backup --src_path="/etc/fail2ban/jail.d/$app.conf"
ynh_backup --src_path="/etc/fail2ban/filter.d/$app.conf" ynh_backup --src_path="/etc/fail2ban/filter.d/$app.conf"
ynh_backup --src_path="/etc/logrotate.d/$app" ynh_backup --src_path="/etc/logrotate.d/$app"
ynh_backup --src_path="/opt/couchdb/etc/local.d/05-flohmarkt.ini" ynh_backup --src_path="/opt/couchdb/etc/local.d/05-flohmarkt.ini"
ynh_backup --src_path="/etc/systemd/system/$app.service" ynh_backup --src_path="/etc/systemd/system/$flohmarkt_filename.service"
ynh_backup --src_path="/var/log/$app" ynh_backup --src_path="${flohmarkt_log_dir}"
ynh_backup --src_path="$flohmarkt_sym_install"
ynh_backup --src_path="$flohmarkt_sym_data_dir"
ynh_backup --src_path="$flohmarkt_sym_log_dir"
# for the following backups we'll want to stop flohmarkt and couchdb # for the following backups we'll want to stop flohmarkt and couchdb
# to guarentee a consistant state # to guarentee a consistant state
ynh_print_info --message="Stopping flohmarkt and couchdb to backup data..." ynh_print_info --message="Stopping flohmarkt to backup data..."
yunohost service stop flohmarkt flohmarkt_ynh_stop_service
systemctl stop couchdb
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/24 # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/24
# since this might be re-installed as a dependency during 'remove' and # since this might be re-installed as a dependency during 'remove' and
@ -28,12 +31,12 @@ systemctl stop couchdb
# #
# if this becomes a pain we'll need to stop deleting this directories on 'remove' # if this becomes a pain we'll need to stop deleting this directories on 'remove'
# ynh_backup --src_path="$data_dir" --is_big # ynh_backup --src_path="$data_dir" --is_big
# ynh_backup --src_path="/var/lib/couchdb" --is_big ynh_backup --src_path="$flohmarkt_data_dir"
ynh_backup --src_path="$data_dir"
ynh_backup --src_path="/var/lib/couchdb"
ynh_print_info --message="...done. Starting couchdb and flohmarkt." ynh_print_info --message="Dumping couchdb..."
systemctl start couchdb flohmarkt_ynh_dump_couchdb
yunohost service start flohmarkt
ynh_print_info --message="Backup script completed for $app. (YunoHost will then actually copy those files to the archive)." ynh_print_info --message="Starting flohmarkt..."
flohmarkt_ynh_start_service
ynh_print_info --message="Backup script completed for $app."

View file

@ -0,0 +1,23 @@
The MIT License (MIT)
Copyright (c) 2015 Daniele Bailo - daniele@danielebailo.it - www.danielebailo.it
Darren Gibbard - dalgibbard@gmail.com - dgunix.com
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View file

@ -0,0 +1,70 @@
Couchdb-dump (& restore)
============
It works on LINUX/UNIX, Bash based systems (MacOSx)
**Bash command line script to EASILY Backup & Restore a CouchDB database**
* Needs bash (plus curl, tr, file, split, awk, sed)
* Dumped database is output to a file (configurable).
## Quickstart (& quickend)
* Backup:
```bash couchdb-dump.sh -b -H 127.0.0.1 -d my-db -f dumpedDB.json -u admin -p password```
* Restore:
```bash couchdb-dump.sh -r -H 127.0.0.1 -d my-db -f dumpedDB.json -u admin -p password```
## Why do you need it?
Surprisingly, there is not a straightforward way to dump a CouchDB database. Often you are suggested to replicate it or to dump it with the couchdb `_all_docs` directive.
**But, using the `_all_docs` directive provides you with JSON which cannot be directly re-import back into CouchDB**.
Hence, the goal of this script(s) is to give you a simple way to Dump & Restore your CouchDB database.
## NOTE
Attachments in Database documents are only supported in CouchDB 1.6+
## Usage
```
Usage: ./couchdb-dump.sh [-b|-r] -H <COUCHDB_HOST> -d <DB_NAME> -f <BACKUP_FILE> [-u <username>] [-p <password>] [-P <port>] [-l <lines>] [-t <threads>] [-a <import_attempts>]
-b Run script in BACKUP mode.
-r Run script in RESTORE mode.
-H CouchDB Hostname or IP. Can be provided with or without 'http(s)://'
-d CouchDB Database name to backup/restore.
-f File to Backup-to/Restore-from.
-P Provide a port number for CouchDB [Default: 5984]
-u Provide a username for auth against CouchDB [Default: blank]
-- can also set with 'COUCHDB_USER' environment var
-p Provide a password for auth against CouchDB [Default: blank]
-- can also set with 'COUCHDB_PASS' environment var
-l Number of lines (documents) to Restore at a time. [Default: 5000] (Restore Only)
-t Number of CPU threads to use when parsing data [Default: nProcs-1] (Backup Only)
-a Number of times to Attempt import before failing [Default: 3] (Restore Only)
-c Create DB on demand, if they are not listed.
-q Run in quiet mode. Suppress output, except for errors and warnings.
-z Compress output file (Backup Only)
-T Add datetime stamp to output file name (Backup Only)
-V Display version information.
-h Display usage information.
Example: ./couchdb-dump.sh -b -H 127.0.0.1 -d mydb -f dumpedDB.json -u admin -p password
```
### Bonus 1! Full Database Compaction
In the past, we've used this script to greatly compress a bloated database.
In our use case, we had non-sequential IDs which cause CouchDB's B-Tree to balloon out of control, even with daily compactions.
**How does this fix work?**
When running the export, all of the documents are pulled out in "ID Order"- When re-importing these (now sorted) documents again, the B-Tree can be created in a much more efficient manner. We've seen 15GB database files, containing only 2.1GB of raw JSON, reduced to 2.5GB on disk after import!
### Bonus 2! Purge Historic and Deleted Data
CouchDB is an append-only database. When you delete records, the metadata is maintained for future reference, and is never fully deleted. All documents also retain a historic revision count.
With the above points in mind; the export and import does not include Deleted documents, or old revisions; therefore, using this script, you can export and re-import your data, cleansing it of any previously (logically) deleted data!
If you pair this with deletion and re-creation of replication rules (using the 'update_seq' parameter to avoid re-pulling the entire DB/deleted documents from a remote node) you can manually compress and clean out an entire cluster of waste, node-by-node.
Note though; after creating all the rules with a fixed update_seq, once completed to the entire cluster, you will need to destroy and recreate all replication rules without the fixed update_seq - else, when restarting a node etc, replication will restart from the old seq.

View file

@ -0,0 +1,780 @@
#!/bin/bash
##
# AUTHOR: DANIELE BAILO
# https://github.com/danielebailo
# www.danielebailo.it
#
# Contributors:
# * dalgibbard - http://github.com/dalgibbard
# * epos-eu - http://github.com/epos-eu
# * maximilianhuber - http://github.com/maximilianhuber
# * ahodgkinson - http://github.com/ahodgkinson (quiet-mode, timestamp, compress)
##
## This script allow for the Backup and Restore of a CouchDB Database.
## Backups are produced in a format that can be later uploaded with the bulk docs directive (as used by this script)
## USAGE
## * To Backup:
## ** example: ./couchdb-dump.sh -b -H 127.0.0.1 -d mydb -u admin -p password -f mydb.json
## * To Restore:
## ** example: ./couchdb-dump.sh -r -H 127.0.0.1 -d mydb -u admin -p password -f mydb.json
###################### CODE STARTS HERE ###################
scriptversionnumber="1.1.10"
##START: FUNCTIONS
usage(){
echo
echo "Usage: $0 [-b|-r] -H <COUCHDB_HOST> -d <DB_NAME> -f <BACKUP_FILE> [-u <username>] [-p <password>] [-P <port>] [-l <lines>] [-t <threads>] [-a <import_attempts>]"
echo -e "\t-b Run script in BACKUP mode."
echo -e "\t-r Run script in RESTORE mode."
echo -e "\t-H CouchDB Hostname or IP. Can be provided with or without 'http(s)://'"
echo -e "\t-d CouchDB Database name to backup/restore."
echo -e "\t-f File to Backup-to/Restore-from."
echo -e "\t-P Provide a port number for CouchDB [Default: 5984]"
echo -e "\t-u Provide a username for auth against CouchDB [Default: blank]"
echo -e "\t -- can also set with 'COUCHDB_USER' environment var"
echo -e "\t-p Provide a password for auth against CouchDB [Default: blank]"
echo -e "\t -- can also set with 'COUCHDB_PASS' environment var"
echo -e "\t-l Number of lines (documents) to Restore at a time. [Default: 5000] (Restore Only)"
echo -e "\t-t Number of CPU threads to use when parsing data [Default: nProcs-1] (Backup Only)"
echo -e "\t-a Number of times to Attempt import before failing [Default: 3] (Restore Only)"
echo -e "\t-c Create DB on demand, if they are not listed."
echo -e "\t-q Run in quiet mode. Suppress output, except for errors and warnings."
echo -e "\t-z Compress output file (Backup Only)"
echo -e "\t-T Add datetime stamp to output file name (Backup Only)"
echo -e "\t-V Display version information."
echo -e "\t-h Display usage information."
echo
echo "Example: $0 -b -H 127.0.0.1 -d mydb -f dumpedDB.json -u admin -p password"
echo
exit 1
}
scriptversion(){
echo
echo -e "\t** couchdb-dump version: $scriptversionnumber **"
echo
echo -e "\t URL:\thttps://github.com/danielebailo/couchdb-dump"
echo
echo -e "\t Authors:"
echo -e "\t Daniele Bailo (bailo.daniele@gmail.com)"
echo -e "\t Darren Gibbard (dalgibbard@gmail.com)"
echo -e "\t Maximilian Huber (maximilian.huber@tngtech.com)"
echo
exit 1
}
checkdiskspace(){
## This function checks available diskspace for a required path, vs space required
## Example call: checkdiskspace /path/to/file/to/create 1024
location=$1
KBrequired=$2
if [ "x$location" = "x" ]||[ "x$KBrequired" = "x" ]; then
echo "... ERROR: checkdiskspace() was not passed the correct arguments."
exit 1
fi
stripdir=${location%/*}
KBavail=$(df -P -k ${stripdir} | tail -n 1 | awk '{print$4}' | $sed_cmd -e 's/K$//')
if [ $KBavail -ge $KBrequired ]; then
return 0
else
echo
echo "... ERROR: Insufficient Disk Space Available:"
echo " * Full Path: ${location}"
echo " * Affected Directory: ${stripdir}"
echo " * Space Available: ${KBavail} KB"
echo " * Total Space Required: ${KBrequired} KB"
echo " * Additional Space Req: $(expr $KBrequired - $KBavail) KB"
echo
exit 1
fi
}
## END FUNCTIONS
# Catch no args:
if [ "x$1" = "x" ]; then
usage
fi
# Default Args
username=""
password=""
backup=false
restore=false
port=5984
OPTIND=1
lines=5000
attempts=3
createDBsOnDemand=false
verboseMode=true
compress=false
timestamp=false
while getopts ":h?H:d:f:u:p:P:l:t:a:c?q?z?T?V?b?B?r?R?" opt; do
case "$opt" in
h) usage;;
b|B) backup=true ;;
r|R) restore=true ;;
H) url="$OPTARG" ;;
d) db_name="$OPTARG" ;;
f) file_name="$OPTARG" ;;
u) username="${OPTARG}";;
p) password="${OPTARG}";;
P) port="${OPTARG}";;
l) lines="${OPTARG}" ;;
t) threads="${OPTARG}" ;;
a) attempts="${OPTARG}";;
c) createDBsOnDemand=true;;
q) verboseMode=false;;
z) compress=true;;
T) timestamp=true;;
V) scriptversion;;
:) echo "... ERROR: Option \"-${OPTARG}\" requires an argument"; usage ;;
*|\?) echo "... ERROR: Unknown Option \"-${OPTARG}\""; usage;;
esac
done
# If quiet option: Setup echo mode and curl '--silent' opt
if [ "$verboseMode" = true ]; then
curlSilentOpt=""
echoVerbose=true
else
curlSilentOpt="--silent"
echoVerbose=false
fi
# Trap unexpected extra args
shift $((OPTIND-1))
[ "$1" = "--" ] && shift
if [ ! "x$@" = "x" ]; then
echo "... ERROR: Unknown Option \"$@\""
usage
fi
# Handle invalid backup/restore states:
if [ $backup = true ]&&[ $restore = true ]; then
echo "... ERROR: Cannot pass both '-b' and '-r'"
usage
elif [ $backup = false ]&&[ $restore = false ]; then
echo "... ERROR: Missing argument '-b' (Backup), or '-r' (Restore)"
usage
fi
# Handle empty args
# url
if [ "x$url" = "x" ]; then
echo "... ERROR: Missing argument '-H <COUCHDB_HOST>'"
usage
fi
# db_name
if [ "x$db_name" = "x" ]; then
echo "... ERROR: Missing argument '-d <DB_NAME>'"
usage
fi
# file_name
if [ "x$file_name" = "x" ]; then
echo "... ERROR: Missing argument '-f <FILENAME>'"
usage
fi
file_name_orig=$file_name
# Get OS TYPE (Linux for Linux, Darwin for MacOSX)
os_type=`uname -s`
# Pick sed or gsed
if [ "$os_type" = "FreeBSD" ]||[ "$os_type" = "Darwin" ]; then
sed_cmd="gsed";
else
sed_cmd="sed";
fi
## Make sure it's installed
echo | $sed_cmd 's/a//' >/dev/null 2>&1
if [ ! $? = 0 ]; then
echo "... ERROR: please install $sed_cmd (gnu-sed) and ensure it is in your path"
exit 1
fi
# Validate thread count
## If we're on a Mac, use sysctl
if [ "$os_type" = "Darwin" ]; then
cores=`sysctl -n hw.ncpu`
## If we're on FreeBSD, use sysctl
elif [ "$os_type" = "FreeBSD" ]; then
cores=`sysctl kern.smp.cpus | awk -F ": " '{print $2}'`;
## Check if nproc available- set cores=1 if not
elif ! type nproc >/dev/null; then
cores=1
## Otherwise use nproc
else
cores=`nproc`
fi
if [ ! "x$threads" = "x" ]; then
if [ $threads -gt $cores ]; then
echo "... WARN: Thread setting of $threads is more than CPU count. Setting to $cores"
threads=$cores
else
$echoVerbose && echo "... INFO: Setting parser threads to $threads"
fi
else
threads=`expr $cores - 1`
fi
# Validate Attempts, set to no-retry if zero/invalid.
case $attempts in
''|0|*[!0-9]*) echo "... WARN: Retry Attempt value of \"$attempts\" is invalid. Disabling Retry-on-Error."; attempts=1 ;;
*) true ;;
esac
## Manage the passing of http/https for $url:
# Note; if the user wants to use 'https://' on a non-443 port they must specify it exclusively in the '-H <HOSTNAME>' arg.
if [ ! "`echo $url | grep -c http`" = 1 ]; then
if [ "$port" == "443" ]; then
url="https://$url";
else
url="http://$url";
fi
fi
# Manage the addition of port
# If a port isn't already on our URL...
if [ ! "`echo $url | egrep -c ":[0-9]*$"`" = "1" ]; then
# add it.
url="$url:$port"
fi
# Check for empty user/pass and try reading in from Envvars
if [ "x$username" = "x" ]; then
username="$COUCHDB_USER"
fi
if [ "x$password" = "x" ]; then
password="$COUCHDB_PASS"
fi
## Manage the addition of user+pass if needed:
# Ensure, if one is set, both are set.
if [ ! "x${username}" = "x" ]; then
if [ "x${password}" = "x" ]; then
echo "... ERROR: Password cannot be blank, if username is specified."
usage
fi
elif [ ! "x${password}" = "x" ]; then
if [ "x${username}" = "x" ]; then
echo "... ERROR: Username cannot be blank, if password is specified."
usage
fi
fi
# Check for sed option
sed_edit_in_place='-i.sedtmp'
if [ "$os_type" = "Darwin" ]; then
sed_regexp_option='E'
else
sed_regexp_option='r'
fi
# Allow for self-signed/invalid certs if method is HTTPS:
if [ "`echo $url | grep -ic "^https://"`" = "1" ]; then
curlopt="-k"
fi
if [ ! "x${username}" = "x" ]&&[ ! "x${password}" = "x" ]; then
curlopt="${curlopt} -u ${username}:${password}"
fi
## Check for curl
curl --version >/dev/null 2>&1 || ( echo "... ERROR: This script requires 'curl' to be present."; exit 1 )
# Check for tr
echo | tr -d "" >/dev/null 2>&1 || ( echo "... ERROR: This script requires 'tr' to be present."; exit 1 )
##### SETUP OUR LARGE VARS FOR SPLIT PROCESSING (due to limitations in split on Darwin/BSD)
AZ2="`echo {a..z}{a..z}`"
AZ3="`echo {a..z}{a..z}{a..z}`"
### If user selected BACKUP, run the following code:
if [ $backup = true ]&&[ $restore = false ]; then
#################################################################
##################### BACKUP START ##############################
#################################################################
# If -T (timestamp) option, append datetime stamp ("-YYYYMMDD-hhmmss") before file extension
if [ "$timestamp" = true ]; then
datetime=`date "+%Y%m%d-%H%M%S"` # Format: YYYYMMDD-hhmmss
# Check for file_name extension, if so add the timestamp before it
if [[ $file_name =~ \.[a-zA-Z0-9][a-zA-Z0-9_]* ]]; then
file_name_ext=` echo "$file_name" | $sed_cmd 's/.*\.//'` # Get text after last '.'
file_name_base=`echo "$file_name" | $sed_cmd "s/\.${file_name_ext}$//"` # file_name without '.' & extension
file_name="$file_name_base-$datetime.$file_name_ext"
else # Otherwise add timestamp to the end of file_name
file_name="$file_name-$datetime"
fi
fi
$echoVerbose && echo "... INFO: Output file ${file_name}"
# Check if output already exists:
if [ -f ${file_name} ]; then
echo "... ERROR: Output file ${file_name} already exists."
exit 1
fi
# Grab our data from couchdb
curl ${curlSilentOpt} ${curlopt} -X GET "$url/$db_name/_all_docs?include_docs=true&attachments=true" -o ${file_name}
# Check for curl errors
if [ ! $? = 0 ]; then
echo "... ERROR: Curl encountered an issue whilst dumping the database."
rm -f ${file_name} 2>/dev/null
exit 1
fi
# Check for export errors
ERR_CHECK="`head -n 1 ${file_name} | grep '^{"error'`"
if [ ! "x${ERR_CHECK}" = "x" ]; then
echo "... ERROR: CouchDB reported: $ERR_CHECK"
exit 1
fi
# CouchDB has a tendancy to output Windows carriage returns in it's output -
# This messes up us trying to sed things at the end of lines!
if grep -qU $'\x0d' $file_name; then
$echoVerbose && echo "... INFO: File may contain Windows carriage returns- converting..."
filesize=$(du -P -k ${file_name} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
tr -d '\r' < ${file_name} > ${file_name}.tmp
if [ $? = 0 ]; then
mv ${file_name}.tmp ${file_name}
if [ $? = 0 ]; then
$echoVerbose && echo "... INFO: Completed successfully."
else
echo "... ERROR: Failed to overwrite ${file_name} with ${file_name}.tmp"
exit 1
fi
else
echo ".. ERROR: Failed to convert file."
exit 1
fi
fi
## Now we parse the output file to make it suitable for re-import.
$echoVerbose && echo "... INFO: Amending file to make it suitable for Import."
$echoVerbose && echo "... INFO: Stage 1 - Document filtering"
# If the input file is larger than 250MB, multi-thread the parsing:
if [ $(du -P -k ${file_name} | awk '{print$1}') -ge 256000 ]&&[ ! $threads -le 1 ]; then
filesize=$(du -P -k ${file_name} | awk '{print$1}')
KBreduction=$(($((`wc -l ${file_name} | awk '{print$1}'` * 80)) / 1024))
filesize=`expr $filesize + $(expr $filesize - $KBreduction)`
checkdiskspace "${file_name}" $filesize
$echoVerbose && echo "... INFO: Multi-Threaded Parsing Enabled."
if [ -f ${file_name}.thread000000 ]; then
echo "... ERROR: Split files \"${file_name}.thread*\" already present. Please remove before continuing."
exit 1
elif [ -f ${file_name}.tmp ]; then
echo "... ERROR: Tempfile ${file_name}.tmp already present. Please remove before continuing."
exit 1
fi
### SPLIT INTO THREADS
split_cal=$(( $((`wc -l ${file_name} | awk '{print$1}'` / $threads)) + $threads ))
#split --numeric-suffixes --suffix-length=6 -l ${split_cal} ${file_name} ${file_name}.thread
split -a 2 -l ${split_cal} ${file_name} ${file_name}.thread
if [ ! "$?" = "0" ]; then
echo "... ERROR: Unable to create split files."
exit 1
fi
# Capture if someone happens to breach the defined limits of AZ2 var. If this happens, we'll need to switch it out for AZ3 ...
if [[ $threads -gt 650 ]]; then
echo "Whoops- we hit a maximum limit here... \$AZ2 only allows for a maximum of 650 cores..."
exit 1
fi
count=0
for suffix in ${AZ2}; do
(( count++ ))
if [[ $count -gt $threads ]]; then
break
fi
PADNAME="${file_name}.thread${suffix}"
$sed_cmd ${sed_edit_in_place} 's/{"id".*,"doc"://g' ${PADNAME} &
done
wait
count=0
for suffix in ${AZ2}; do
(( count++ ))
if [[ $count -gt $threads ]]; then
break
fi
PADNAME="${file_name}.thread${suffix}"
cat ${PADNAME} >> ${file_name}.tmp
rm -f ${PADNAME} ${PADNAME}.sedtmp
(( NUM++ ))
done
if [ `wc -l ${file_name} | awk '{print$1}'` = `wc -l ${file_name}.tmp | awk '{print$1}'` ]; then
mv ${file_name}{.tmp,}
if [ ! $? = 0 ]; then
echo "... ERROR: Failed to overwrite ${file_name}"
exit 1
fi
else
echo "... ERROR: Multi-threaded data parsing encountered an error."
exit 1
fi
else
# Estimating 80byte saving per line... probably a little conservative depending on keysize.
KBreduction=$(($((`wc -l ${file_name} | awk '{print$1}'` * 80)) / 1024))
filesize=$(du -P -k ${file_name} | awk '{print$1}')
filesize=`expr $filesize - $KBreduction`
checkdiskspace "${file_name}" $filesize
$sed_cmd ${sed_edit_in_place} 's/{"id".*,"doc"://g' $file_name && rm -f ${file_name}.sedtmp
if [ ! $? = 0 ];then
echo "Stage failed."
exit 1
fi
fi
$echoVerbose && echo "... INFO: Stage 2 - Duplicate curly brace removal"
# Approx 1Byte per line removed
KBreduction=$((`wc -l ${file_name} | awk '{print$1}'` / 1024))
filesize=$(du -P -k ${file_name} | awk '{print$1}')
filesize=`expr $filesize - $KBreduction`
checkdiskspace "${file_name}" $filesize
$sed_cmd ${sed_edit_in_place} 's/}},$/},/g' ${file_name} && rm -f ${file_name}.sedtmp
if [ ! $? = 0 ];then
echo "Stage failed."
exit 1
fi
$echoVerbose && echo "... INFO: Stage 3 - Header Correction"
filesize=$(du -P -k ${file_name} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
$sed_cmd ${sed_edit_in_place} '1s/^.*/{"new_edits":false,"docs":[/' ${file_name} && rm -f ${file_name}.sedtmp
if [ ! $? = 0 ];then
echo "Stage failed."
exit 1
fi
$echoVerbose && echo "... INFO: Stage 4 - Final document line correction"
filesize=$(du -P -k ${file_name} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
$sed_cmd ${sed_edit_in_place} 's/}}$/}/g' ${file_name} && rm -f ${file_name}.sedtmp
if [ ! $? = 0 ];then
echo "Stage failed."
exit 1
fi
# If -z (compress) option then compress output file
if [ "$compress" = true ]; then
$echoVerbose && echo "... INFO: Stage 5 - File compression"
gzip $file_name
file_name="$file_name.gz"
fi
$echoVerbose && echo "... INFO: Export completed successfully. File available at: ${file_name}"
exit 0
### Else if user selected Restore:
elif [ $restore = true ]&&[ $backup = false ]; then
#################################################################
##################### RESTORE START #############################
#################################################################
# Check if input exists:
if [ ! -f ${file_name} ]; then
echo "... ERROR: Input file ${file_name} not found."
exit 1
fi
#### VALIDATION END
$echoVerbose && echo "... INFO: Checking for database"
attemptcount=0
A=0
until [ $A = 1 ]; do
(( attemptcount++ ))
existing_dbs=$(curl $curlSilentOpt $curlopt -X GET "${url}/_all_dbs")
if [ ! $? = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: Curl failed to get the list of databases - Stopping"
exit 1
else
echo "... WARN: Curl failed to get the list of databases - Attempt ${attemptcount}/${attempts}. Retrying..."
sleep 1
fi
else
A=1
fi
done
if [[ ! "$existing_dbs" = "["*"]" ]]; then
echo "... WARN: Curl failed to get the list of databases - Continuing"
if [ "x$existing_dbs" = "x" ]; then
echo "... WARN: Curl just returned: $existing_dbs"
fi
elif [[ ! "$existing_dbs" = *"\"${db_name}\""* ]]; then
# database was not listed as existing databasa
if [ $createDBsOnDemand = true ]; then
attemptcount=0
A=0
until [ $A = 1 ]; do
(( attemptcount++ ))
curl $curlSilentOpt $curlopt -X PUT "${url}/${db_name}" -o tmp.out
# If curl threw an error:
if [ ! $? = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: Curl failed to create the database ${db_name} - Stopping"
if [ -f tmp.out ]; then
echo -n "... ERROR: Error message was: "
cat tmp.out
else
echo ".. ERROR: See above for any errors"
fi
exit 1
else
echo "... WARN: Curl failed to create the database ${db_name} - Attempt ${attemptcount}/${attempts}. Retrying..."
sleep 1
fi
# If curl was happy, but CouchDB returned an error in the return JSON:
elif [ ! "`head -n 1 tmp.out | grep -c '^{"error":'`" = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: CouchDB Reported: `head -n 1 tmp.out`"
exit 1
else
echo "... WARN: CouchDB Reported an error during db creation - Attempt ${attemptcount}/${attempts} - Retrying..."
sleep 1
fi
# Otherwise, if everything went well, delete our temp files.
else
rm tmp.out
A=1
fi
done
else
echo "... ERROR: corresponding database ${db_name} not yet created - Stopping"
$echoVerbose && echo "... HINT: you could add the -c flag to create the database automatically"
exit 1
fi
fi
## Stop bash mangling wildcard...
set -o noglob
# Manage Design Documents as a priority, and remove them from the main import job
$echoVerbose && echo "... INFO: Checking for Design documents"
# Find all _design docs, put them into another file
design_file_name=${file_name}-design
grep '^{"_id":"_design' ${file_name} > ${design_file_name}
# Count the design file (if it even exists)
DESIGNS="`wc -l ${design_file_name} 2>/dev/null | awk '{print$1}'`"
# If there's no design docs for import...
if [ "x$DESIGNS" = "x" ]||[ "$DESIGNS" = "0" ]; then
# Cleanup any null files
rm -f ${design_file_name} 2>/dev/null
$echoVerbose && echo "... INFO: No Design Documents found for import."
else
$echoVerbose && echo "... INFO: Duplicating original file for alteration"
# Duplicate the original DB file, so we don't mangle the user's input file:
filesize=$(du -P -k ${file_name} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
cp -f ${file_name}{,-nodesign}
# Re-set file_name to be our new file.
file_name=${file_name}-nodesign
# Remove these design docs from (our new) main file.
$echoVerbose && echo "... INFO: Stripping _design elements from regular documents"
checkdiskspace "${file_name}" $filesize
$sed_cmd ${sed_edit_in_place} '/^{"_id":"_design/d' ${file_name} && rm -f ${file_name}.sedtmp
# Remove the final document's trailing comma
$echoVerbose && echo "... INFO: Fixing end document"
line=$(expr `wc -l ${file_name} | awk '{print$1}'` - 1)
filesize=$(du -P -k ${file_name} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
$sed_cmd ${sed_edit_in_place} "${line}s/,$//" ${file_name} && rm -f ${file_name}.sedtmp
$echoVerbose && echo "... INFO: Inserting Design documents"
designcount=0
# For each design doc...
while IFS="" read -r; do
line="${REPLY}"
# Split the ID out for use as the import URL path
URLPATH=$(echo $line | awk -F'"' '{print$4}')
# Scrap the ID and Rev from the main data, as well as any trailing ','
echo "${line}" | $sed_cmd -${sed_regexp_option}e "s@^\{\"_id\":\"${URLPATH}\",\"_rev\":\"[0-9]*-[0-9a-zA-Z_\-]*\",@\{@" | $sed_cmd -e 's/,$//' > ${design_file_name}.${designcount}
# Fix Windows CRLF
if grep -qU $'\x0d' ${design_file_name}.${designcount}; then
$echoVerbose && echo "... INFO: File contains Windows carriage returns- converting..."
filesize=$(du -P -k ${design_file_name}.${designcount} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
tr -d '\r' < ${design_file_name}.${designcount} > ${design_file_name}.${designcount}.tmp
if [ $? = 0 ]; then
mv ${design_file_name}.${designcount}.tmp ${design_file_name}.${designcount}
if [ $? = 0 ]; then
$echoVerbose && echo "... INFO: Completed successfully."
else
echo "... ERROR: Failed to overwrite ${design_file_name}.${designcount} with ${design_file_name}.${designcount}.tmp"
exit 1
fi
else
echo ".. ERROR: Failed to convert file."
exit 1
fi
fi
# Insert this file into the DB
A=0
attemptcount=0
until [ $A = 1 ]; do
(( attemptcount++ ))
curl $curlSilentOpt ${curlopt} -T ${design_file_name}.${designcount} -X PUT "${url}/${db_name}/${URLPATH}" -H 'Content-Type: application/json' -o ${design_file_name}.out.${designcount}
# If curl threw an error:
if [ ! $? = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: Curl failed trying to restore ${design_file_name}.${designcount} - Stopping"
exit 1
else
echo "... WARN: Import of ${design_file_name}.${designcount} failed - Attempt ${attemptcount}/${attempts}. Retrying..."
sleep 1
fi
# If curl was happy, but CouchDB returned an error in the return JSON:
elif [ ! "`head -n 1 ${design_file_name}.out.${designcount} | grep -c '^{"error":'`" = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: CouchDB Reported: `head -n 1 ${design_file_name}.out.${designcount}`"
exit 1
else
echo "... WARN: CouchDB Reported an error during import - Attempt ${attemptcount}/${attempts} - Retrying..."
sleep 1
fi
# Otherwise, if everything went well, delete our temp files.
else
A=1
rm -f ${design_file_name}.out.${designcount}
rm -f ${design_file_name}.${designcount}
fi
done
# Increase design count - mainly used for the INFO at the end.
(( designcount++ ))
# NOTE: This is where we insert the design lines exported from the main block
done < <(cat ${design_file_name})
$echoVerbose && echo "... INFO: Successfully imported ${designcount} Design Documents"
fi
set +o noglob
# If the size of the file to import is less than our $lines size, don't worry about splitting
if [ `wc -l $file_name | awk '{print$1}'` -lt $lines ]; then
$echoVerbose && echo "... INFO: Small dataset. Importing as a single file."
A=0
attemptcount=0
until [ $A = 1 ]; do
(( attemptcount++ ))
curl $curlSilentOpt $curlopt -T $file_name -X POST "$url/$db_name/_bulk_docs" -H 'Content-Type: application/json' -o tmp.out
if [ "`head -n 1 tmp.out | grep -c '^{"error":'`" -eq 0 ]; then
$echoVerbose && echo "... INFO: Imported ${file_name_orig} Successfully."
rm -f tmp.out
rm -f ${file_name_orig}-design
rm -f ${file_name_orig}-nodesign
exit 0
else
if [ $attemptcount = $attempts ]; then
echo "... ERROR: Import of ${file_name_orig} failed."
if [ -f tmp.out ]; then
echo -n "... ERROR: Error message was: "
cat tmp.out
else
echo ".. ERROR: See above for any errors"
fi
rm -f tmp.out
exit 1
else
echo "... WARN: Import of ${file_name_orig} failed - Attempt ${attemptcount}/${attempts} - Retrying..."
sleep 1
fi
fi
done
# Otherwise, it's a large import that requires bulk insertion.
else
$echoVerbose && echo "... INFO: Block import set to ${lines} lines."
if [ -f ${file_name}.splitaaa ]; then
echo "... ERROR: Split files \"${file_name}.split*\" already present. Please remove before continuing."
exit 1
fi
importlines=`cat ${file_name} | grep -c .`
# Due to the file limit imposed by the pre-calculated AZ3 variable, max split files is 15600 (alpha x 3positions)
if [[ `expr ${importlines} / ${lines}` -gt 15600 ]]; then
echo "... ERROR: Pre-processed split variable limit of 15600 files reached."
echo " Please increase the '-l' parameter (Currently: $lines) and try again."
exit 1
fi
$echoVerbose && echo "... INFO: Generating files to import"
filesize=$(du -P -k ${file_name} | awk '{print$1}')
checkdiskspace "${file_name}" $filesize
### Split the file into many
split -a 3 -l ${lines} ${file_name} ${file_name}.split
if [ ! "$?" = "0" ]; then
echo "... ERROR: Unable to create split files."
exit 1
fi
HEADER="`head -n 1 $file_name`"
FOOTER="`tail -n 1 $file_name`"
count=0
for PADNUM in $AZ3; do
PADNAME="${file_name}.split${PADNUM}"
if [ ! -f ${PADNAME} ]; then
echo "... INFO: Import Cycle Completed."
break
fi
if [ ! "`head -n 1 ${PADNAME}`" = "${HEADER}" ]; then
$echoVerbose && echo "... INFO: Adding header to ${PADNAME}"
filesize=$(du -P -k ${PADNAME} | awk '{print$1}')
checkdiskspace "${PADNAME}" $filesize
$sed_cmd ${sed_edit_in_place} "1i${HEADER}" ${PADNAME} && rm -f ${PADNAME}.sedtmp
else
$echoVerbose && echo "... INFO: Header already applied to ${PADNAME}"
fi
if [ ! "`tail -n 1 ${PADNAME}`" = "${FOOTER}" ]; then
$echoVerbose && echo "... INFO: Adding footer to ${PADNAME}"
filesize=$(du -P -k ${PADNAME} | awk '{print$1}')
checkdiskspace "${PADNAME}" $filesize
$sed_cmd ${sed_edit_in_place} '$s/,$//g' ${PADNAME} && rm -f ${PADNAME}.sedtmp
echo "${FOOTER}" >> ${PADNAME}
else
$echoVerbose && echo "... INFO: Footer already applied to ${PADNAME}"
fi
$echoVerbose && echo "... INFO: Inserting ${PADNAME}"
A=0
attemptcount=0
until [ $A = 1 ]; do
(( attemptcount++ ))
curl $curlSilentOpt $curlopt -T ${PADNAME} -X POST "$url/$db_name/_bulk_docs" -H 'Content-Type: application/json' -o tmp.out
if [ ! $? = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: Curl failed trying to restore ${PADNAME} - Stopping"
exit 1
else
echo "... WARN: Failed to import ${PADNAME} - Attempt ${attemptcount}/${attempts} - Retrying..."
sleep 1
fi
elif [ ! "`head -n 1 tmp.out | grep -c '^{"error":'`" = 0 ]; then
if [ $attemptcount = $attempts ]; then
echo "... ERROR: CouchDB Reported: `head -n 1 tmp.out`"
exit 1
else
echo "... WARN: CouchDB Reported and error during import - Attempt ${attemptcount}/${attempts} - Retrying..."
sleep 1
fi
else
A=1
rm -f ${PADNAME}
rm -f tmp.out
(( count++ ))
fi
done
$echoVerbose && echo "... INFO: Successfully Imported `expr ${count}` Files"
A=1
rm -f ${file_name_orig}-design
rm -f ${file_name_orig}-nodesign
done
fi
fi

View file

@ -6,7 +6,11 @@ source /usr/share/yunohost/helpers
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/9 # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/9
# check if couchdb is already installed # check if couchdb is already installed
if [[ -e /opt/couchdb ]] || pgrep epmd > /dev/null || pgrep beam.smp || dpkg-query -W couchdb > /dev/null 2>&1; # if there's a couchdb configuration file for flohmarkt we'll assume that couchdb
# had been installed by flohmarkt and we thus can savely do whatever we want to it
# with the resulting damage hopefully only influencing flohmarkt instances
if [[ -e /opt/couchdb ]] || pgrep epmd > /dev/null || pgrep beam.smp || dpkg-query -W couchdb > /dev/null 2>&1 \
&& ! [[ -e /opt/couchdb/etc/local.d/05-flohmarkt.ini ]];
then then
ynh_die --message="CouchDB already installed on this host - will not proceed." ynh_die --message="CouchDB already installed on this host - will not proceed."
exit 1 exit 1
@ -29,22 +33,11 @@ ynh_script_progression --message="Installing CouchDB..." --weight=60
couchdb_magic_cookie=$(ynh_string_random --length=23 --filter='A-Za-z0-9_') couchdb_magic_cookie=$(ynh_string_random --length=23 --filter='A-Za-z0-9_')
ynh_app_setting_set --app=$app --key=couchdb_magic_cookie --value="$couchdb_magic_cookie" ynh_app_setting_set --app=$app --key=couchdb_magic_cookie --value="$couchdb_magic_cookie"
echo "\ # update couchdb dpkg
couchdb couchdb/mode select standalone flohmarkt_ynh_up_inst_couchdb
couchdb couchdb/mode seen true
couchdb couchdb/bindaddress string 127.0.0.1
couchdb couchdb/bindaddress seen true
couchdb couchdb/cookie string $couchdb_magic_cookie
couchdb couchdb/adminpass password $password_couchdb_admin
couchdb couchdb/adminpass seen true
couchdb couchdb/adminpass_again password $password_couchdb_admin
couchdb couchdb/adminpass_again seen true" | debconf-set-selections
DEBIAN_FRONTEND=noninteractive # apt-get install -y --force-yes couchdb
ynh_install_extra_app_dependencies \ # stop couchdb
--repo="deb https://apache.jfrog.io/artifactory/couchdb-deb/ $(lsb_release -c -s) main" \ flohmarkt_ynh_stop_couchdb
--key="https://couchdb.apache.org/repo/keys.asc" \
--package="couchdb"
# add couchdb configuration # add couchdb configuration
ynh_script_progression --message="Adding a configuration file..." --weight=2 ynh_script_progression --message="Adding a configuration file..." --weight=2
@ -54,25 +47,19 @@ chown root:couchdb /opt/couchdb/etc/local.d/05-flohmarkt.ini
chmod 640 /opt/couchdb/etc/local.d/05-flohmarkt.ini chmod 640 /opt/couchdb/etc/local.d/05-flohmarkt.ini
# restart couchdb to pick up changes # restart couchdb to pick up changes
systemctl restart couchdb ynh_script_progression --message="Starting CouchDB..." --weight=2
systemctl status couchdb flohmarkt_ynh_start_couchdb
# get flohmarkt # get flohmarkt
ynh_setup_source --dest_dir="$install_dir/$app/" # suspecting that this deletes all other sources for YNH_APP_ID
ynh_script_progression --message="Downloading flohmarkt..." --weight=2
ynh_setup_source --dest_dir="$flohmarkt_app_dir"
# setup python environment for flohmarkt # setup python environment for flohmarkt
ynh_secure_remove "$install_dir/venv" ynh_script_progression --message="Creating python venv..." --weight=2
python3 -m venv --without-pip "$install_dir/venv" ynh_secure_remove "$flohmarkt_venv_dir"
flohmarkt_ynh_create_venv
# install python dependencies flohmarkt_ynh_venv_requirements
(
set +o nounset
source "$install_dir/venv/bin/activate"
set -o nounset
set -x
$install_dir/venv/bin/python3 -m ensurepip
$install_dir/venv/bin/pip3 install -r "$install_dir/$app/requirements.txt"
)
# JwtSecret # JwtSecret
# workaround for https://github.com/YunoHost/issues/issues/2379 # workaround for https://github.com/YunoHost/issues/issues/2379
@ -83,24 +70,27 @@ password_couchdb_flohmarkt=$(ynh_string_random --length=31 --filter='A-Za-z0-9_.
ynh_app_setting_set --app=$app --key=password_couchdb_flohmarkt --value="$password_couchdb_flohmarkt" ynh_app_setting_set --app=$app --key=password_couchdb_flohmarkt --value="$password_couchdb_flohmarkt"
# generate flohmarkt.conf # generate flohmarkt.conf
ynh_add_config --template="../conf/flohmarkt.conf" --destination="$install_dir/$app/flohmarkt.conf" ynh_script_progression --message="Adding flohmarkt.conf configuration..." --weight=2
ynh_add_config --template="../conf/flohmarkt.conf" --destination="$flohmarkt_app_dir/flohmarkt.conf"
# setup couchdb # setup couchdb
ynh_script_progression --message="Setup couchdb database and user for flohmarkt..." --weight=2
flohmarkt_ynh_check_old_couchdb
( (
set +o nounset set +o nounset
source "$install_dir/venv/bin/activate" source "$flohmarkt_venv_dir/bin/activate"
set -o nounset set -o nounset
cd "$install_dir/$app" cd "$flohmarkt_app_dir"
# initialize_couchdb seems to re-try on connect problems endlessly blocking the yunohost api # initialize_couchdb seems to re-try on connect problems endlessly blocking the yunohost api
# give it 45 seconds to finish and then fail # give it 45 seconds to finish and then fail
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/13 # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/13
timeout 45 python3 initialize_couchdb.py $password_couchdb_admin $password_couchdb_flohmarkt timeout 45 python3 initialize_couchdb.py $password_couchdb_admin
) )
# SETUP LOGROTATE # SETUP LOGROTATE
ynh_script_progression --message="Configuring log rotation..." --weight=2 ynh_script_progression --message="Configuring log rotation..." --weight=2
# Use logrotate to manage application logfile(s) # Use logrotate to manage application logfile(s)
ynh_use_logrotate ynh_use_logrotate --logfile=$flohmarkt_logfile
# NGINX CONFIGURATION # NGINX CONFIGURATION
ynh_script_progression --message="Configuring NGINX web server..." --weight=3 ynh_script_progression --message="Configuring NGINX web server..." --weight=3
@ -110,19 +100,37 @@ ynh_add_nginx_config
# systemd.service # systemd.service
ynh_script_progression --message="Configuring a systemd service..." --weight=1 ynh_script_progression --message="Configuring a systemd service..." --weight=1
# Create a dedicated systemd config # Create a dedicated systemd config
ynh_add_systemd_config ynh_add_systemd_config --service=$flohmarkt_filename
# integrate into yunohost # integrate into yunohost
ynh_script_progression --message="Integrating service in YunoHost..." --weight=1 ynh_script_progression --message="Integrating service in YunoHost..." --weight=1
yunohost service add $app --description="A decentral federated small advertisement platform" --log="/var/log/$app/$app.log" yunohost service add $flohmarkt_filename --description="A decentral federated small advertisement platform" --log="$flohmarkt_logfile"
# logfile contains possibly the secret setup URL # logfile contains possibly the secret setup URL
mkdir -m755 -p "/var/log/$app" ynh_script_progression --message="Setting permissions on logfile..." --weight=2
touch "/var/log/$app/$app.log" mkdir -m755 -p "$flohmarkt_log_dir"
chmod 640 "/var/log/$app/$app.log" touch "$flohmarkt_logfile"
# start service chown root:$app "$flohmarkt_logfile"
ynh_systemd_action --service_name=$app --action="start" chmod 660 "$flohmarkt_logfile"
# set file permissions for install dir
ynh_script_progression --message="Setting permissions on install directory..." --weight=2
flohmarkt_ynh_set_permission
# SETUP FAIL2BAN # SETUP FAIL2BAN
ynh_script_progression --message="Configuring Fail2Ban..." --weight=3 ynh_script_progression --message="Configuring Fail2Ban..." --weight=3
ynh_add_fail2ban_config --logpath="/var/log/$app/$app.log" --failregex='INFO: +<HOST>:\d+ - "POST /token HTTP/\d+\.\d+" 403 Forbidden' --max_retry=5 ynh_add_fail2ban_config --logpath="$flohmarkt_logfile" --failregex='INFO: +<HOST>:\d+ - "POST /token HTTP/\d+\.\d+" 403 Forbidden' --max_retry=5
ynh_script_progression --message="Installation of $app completed" --last # symlink data_dir and install_dir to make it easier to find the
# files for a certain domain/path
ynh_script_progression --message="Creating symlinks..." --weight=1
ln -s "$flohmarkt_install" "$flohmarkt_sym_install"
ln -s "$flohmarkt_data_dir" "$flohmarkt_sym_data_dir"
ln -s "$flohmarkt_log_dir" "$flohmarkt_sym_log_dir"
# start service
ynh_script_progression --message="Debug before starting flohmarkt..." --weight=1
ls -l $flohmarkt_logfile /bin/bash /usr/bin/bash || true
ynh_script_progression --message="Starting flohmarkt..." --weight=10
flohmarkt_ynh_start_service
ynh_script_progression --message="Installation of $id completed" --last

View file

@ -18,44 +18,43 @@ ynh_script_progression --message="Removing system configurations related to $app
# Stop flohmarkt service # Stop flohmarkt service
ynh_script_progression --message="Stopping $app..." --weight=2 ynh_script_progression --message="Stopping $app..." --weight=2
yunohost service stop $app yunohost service stop $flohmarkt_filename
# Remove the service from the list of services known by YunoHost (added from `yunohost service add`) # Remove the service from the list of services known by YunoHost (added from `yunohost service add`)
if ynh_exec_warn_less yunohost service status $app >/dev/null if ynh_exec_warn_less yunohost service status $flohmarkt_filename >/dev/null
then then
ynh_script_progression --message="Removing $app service integration..." --weight=2 ynh_script_progression --message="Removing $app service integration..." --weight=2
yunohost service remove $app ynh_remove_systemd_config --service=$flohmarkt_filename
yunohost service remove $flohmarkt_filename
fi fi
ynh_script_progression --message="Stopping couchdb..." --weight=2 # https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/12
systemctl stop couchdb ynh_script_progression --message="Removing database and database user..." --weight=2
# remove DB
ynh_script_progression --message="Removing databases..." --weight=2 flohmarkt_ynh_delete_couchdb_db
# remove flohmarkt database and database user # remove DB user for this instance:
# ynh_script_progression --message="removing flohmarkt couchdb user and database" --weight=2 flohmarkt_ynh_delete_couchdb_user
# @@ wie komme ich hier an das admin-password, falls ich es brauche?
# * brauche ich es? → ja
# db:
# curl -X DELETE 'http://127.0.0.1:5984/flohmarkt' --user '<admin-user>:<admin-password>'
# user:
# curl -X DELETE "http://127.0.0.1:5984/_users/org.couchdb.user%3Aflohmarkt?rev=`curl -sX GET 'http://127.0.0.1:5984/_users/org.couchdb.user%3Aflohmarkt' --user 'admin:bla42fasel' | jq -r ._rev`" --user '<admin-user>:<admin-password>'
# @@ todo improve this like above
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/12
ynh_secure_remove --file=/var/lib/couchdb
# Remove the app-specific logrotate config # Remove the app-specific logrotate config
ynh_script_progression --message="Removing logrotate configuration..." --weight=2
ynh_remove_logrotate ynh_remove_logrotate
# remove log directory # remove logfiles directory
ynh_secure_remove --file="/var/log/$app/$app.log" # https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/29
# can only remove a subdir or one file
ynh_script_progression --message="Removing logfiles..." --weight=2
ynh_secure_remove --file="${flohmarkt_log_dir}"
# Remove the dedicated NGINX config # Remove the dedicated NGINX config
ynh_script_progression --message="Removing nginx configuration..." --weight=2
ynh_remove_nginx_config ynh_remove_nginx_config
# Remove the dedicated Fail2Ban config # Remove the dedicated Fail2Ban config
ynh_script_progression --message="Removing fail2ban configuration..." --weight=2
ynh_remove_fail2ban_config ynh_remove_fail2ban_config
# Remove home # Remove home
# https://codeberg.org/flohmarkt/flohmarkt_ynh/issues/29
ynh_script_progression --message="Removing data directory..." --weight=2 ynh_script_progression --message="Removing data directory..." --weight=2
ynh_secure_remove --file=$data_dir ynh_secure_remove --file=$data_dir
@ -63,6 +62,15 @@ ynh_secure_remove --file=$data_dir
ynh_script_progression --message="Removing data directory..." --weight=2 ynh_script_progression --message="Removing data directory..." --weight=2
ynh_remove_systemd_config ynh_remove_systemd_config
# remove symlinks
ynh_script_progression --message="Removing symlinks..." --weight=2
ynh_secure_remove --file="$flohmarkt_sym_install"
rm "$flohmarkt_sym_install" || true
ynh_secure_remove --file="$flohmarkt_sym_data_dir"
rm "$flohmarkt_sym_data_dir" || true
ynh_secure_remove --file="$flohmarkt_sym_log_dir"
rm "$flohmarkt_sym_log_dir" || true
#================================================= #=================================================
# END OF SCRIPT # END OF SCRIPT
#================================================= #=================================================

View file

@ -3,50 +3,37 @@
source ../settings/scripts/_common.sh source ../settings/scripts/_common.sh
source /usr/share/yunohost/helpers source /usr/share/yunohost/helpers
# restore couchdb directory
ynh_script_progression --message="Restoring couchdb directory..." --weight=1
ynh_restore_file --origin_path="/var/lib/couchdb"
# reinstall couchdb # reinstall couchdb
ynh_script_progression --message="Reinstalling couchdb..." --weight=40 ynh_script_progression --message="Reinstalling couchdb..." --weight=40
flohmarkt_ynh_up_inst_couchdb
echo "\
couchdb couchdb/mode select standalone
couchdb couchdb/mode seen true
couchdb couchdb/bindaddress string 127.0.0.1
couchdb couchdb/bindaddress seen true
couchdb couchdb/cookie string $couchdb_magic_cookie
couchdb couchdb/adminpass password $password_couchdb_admin
couchdb couchdb/adminpass seen true
couchdb couchdb/adminpass_again password $password_couchdb_admin
couchdb couchdb/adminpass_again seen true" | debconf-set-selections
DEBIAN_FRONTEND=noninteractive # apt-get install -y --force-yes couchdb
ynh_install_extra_app_dependencies \
--repo="deb https://apache.jfrog.io/artifactory/couchdb-deb/ $(lsb_release -c -s) main" \
--key="https://couchdb.apache.org/repo/keys.asc" \
--package="couchdb"
# stop couchdb
systemctl stop couchdb
# add couchdb configuration # add couchdb configuration
if ! [[ -e /opt/couchdb/etc/local.d/05-flohmarkt.ini ]]; then
ynh_script_progression --message="Adding a configuration file..." --weight=2 ynh_script_progression --message="Adding a configuration file..." --weight=2
flohmarkt_ynh_stop_couchdb
ynh_restore_file --origin_path="/opt/couchdb/etc/local.d/05-flohmarkt.ini" ynh_restore_file --origin_path="/opt/couchdb/etc/local.d/05-flohmarkt.ini"
chown root:couchdb /opt/couchdb/etc/local.d/05-flohmarkt.ini chown root:couchdb /opt/couchdb/etc/local.d/05-flohmarkt.ini
chmod 640 /opt/couchdb/etc/local.d/05-flohmarkt.ini chmod 640 /opt/couchdb/etc/local.d/05-flohmarkt.ini
# start couchdb ynh_script_progression --message="Starting couchdb..." --weight=4
systemctl start couchdb flohmarkt_ynh_start_couchdb
systemctl status couchdb else
ynh_script_progression --message="CouchDB configuration file already exists." --weight=1
fi
# restore couchdb from json
ynh_script_progression --message="Importing couchdb from json backup..." --weight=8
flohmarkt_ynh_restore_couchdb
# RESTORE THE APP MAIN DIR # RESTORE THE APP MAIN DIR
ynh_script_progression --message="Restoring the app main directory..." --weight=10 ynh_script_progression --message="Restoring the app main directory..." --weight=10
ynh_restore_file --origin_path="$install_dir" ynh_restore_file --origin_path="$flohmarkt_install"
# RESTORE THE DATA DIRECTORY # RESTORE THE DATA DIRECTORY
ynh_script_progression --message="Restoring the data directory..." --weight=1 ynh_script_progression --message="Restoring the data directory..." --weight=1
ynh_restore_file --origin_path="$data_dir" --not_mandatory ynh_restore_file --origin_path="$flohmarkt_data_dir" --not_mandatory
mkdir -p $data_dir mkdir -p $data_dir
# restore logfiles # restore logfiles
@ -56,11 +43,11 @@ mkdir -p $data_dir
# If the logfile(or maybe just -directory?) is missing the service will not start # If the logfile(or maybe just -directory?) is missing the service will not start
# on a fresh install, because directory and/or file do not exist. # on a fresh install, because directory and/or file do not exist.
ynh_script_progression --message="Restoring log directory and logfiles..." --weight=1 ynh_script_progression --message="Restoring log directory and logfiles..." --weight=1
ynh_restore_file --origin_path="/var/log/$app" --not_mandatory ynh_restore_file --origin_path="$flohmarkt_log_dir" --not_mandatory
mkdir -p -m755 "/var/log/$app" mkdir -p -m755 "$flohmarkt_log_dir"
touch "/var/log/$app/$app.log" touch "$flohmarkt_logfile"
# logfile contains possibly the secret setup URL # logfile contains possibly the secret setup URL
chmod 640 "/var/log/$app/$app.log" chmod 640 "$flohmarkt_logfile"
# RESTORE FAIL2BAN CONFIGURATION # RESTORE FAIL2BAN CONFIGURATION
ynh_script_progression --message="Restoring the Fail2Ban configuration..." --weight=2 ynh_script_progression --message="Restoring the Fail2Ban configuration..." --weight=2
@ -78,10 +65,10 @@ ynh_restore_file --origin_path="/etc/logrotate.d/$app"
# restore flohmarkt systemd service and integrate service into yunohost # restore flohmarkt systemd service and integrate service into yunohost
ynh_script_progression --message="Restoring flohmarkt.service and integrating service into YunoHost..." --weight=1 ynh_script_progression --message="Restoring flohmarkt.service and integrating service into YunoHost..." --weight=1
ynh_restore_file --origin_path="/etc/systemd/system/$app.service" ynh_restore_file --origin_path="/etc/systemd/system/$flohmarkt_filename.service"
ynh_systemd_action --action=enable ynh_systemd_action --action=enable --service_name=$flohmarkt_filename
yunohost service add $app --description="A decentral federated small advertisement platform" --log="/var/log/$app/$app.log" yunohost service add $flohmarkt_filename --description="A decentral federated small advertisement platform" --log="$flohmarkt_logfile"
ynh_systemd_action --action=start --line_match="INFO: Application startup complete." flohmarkt_ynh_start_service
# RELOAD NGINX # RELOAD NGINX
ynh_script_progression --message="Reloading NGINX web server..." --weight=1 ynh_script_progression --message="Reloading NGINX web server..." --weight=1

View file

@ -9,29 +9,32 @@ source /usr/share/yunohost/helpers
# UPGRADE_SAME # UPGRADE_SAME
upgrade_type=$(ynh_check_app_version_changed) upgrade_type=$(ynh_check_app_version_changed)
# show variables - used from installed or from to be upgraded version?
# variables of the new upgrade version are used
# env
ynh_script_progression --message="Stopping flohmarkt and CouchDB..." --weight=5 ynh_script_progression --message="Stopping flohmarkt and CouchDB..." --weight=5
# stop flohmarkt # stop flohmarkt
ynh_systemd_action --service_name=$app --action="stop" flohmarkt_ynh_stop_service
# stop couchdb # stop couchdb
ynh_systemd_action --service_name=couchdb --action="stop" flohmarkt_ynh_stop_couchdb
# Upgrade ynh4 → ynh5, change of paths and filenames
# https://matrix.to/#/!PauySEslPVuJCJCwlZ:matrix.org/$auU_rSL6MACt9wkyFGlCQbO8ivFlU0ztYW74TimZdcM?via=matrix.org&via=aria-net.org&via=sans-nuage.fr
# and follow-ups
if ynh_compare_current_package_version --comparison lt --version 0.0~ynh5 &&
( [[ "$upgrade_type" == UPGRADE_PACKAGE ]] || [[ "$upgrade_type" == UPGRADE_APP ]] )
then
flohmarkt_ynh_upgrade_path_ynh5
fi
# upgrade couchdb # upgrade couchdb
ynh_script_progression --message="Upgrading CouchDB..." --weight=50 ynh_script_progression --message="Upgrading CouchDB..." --weight=50
echo "\ flohmarkt_ynh_up_inst_couchdb
couchdb couchdb/mode select standalone
couchdb couchdb/mode seen true # stop couchdb
couchdb couchdb/bindaddress string 127.0.0.1 ynh_script_progression --message="Stop couchdb before updating configuration file..." --weight=1
couchdb couchdb/bindaddress seen true flohmarkt_ynh_stop_couchdb
couchdb couchdb/cookie string $couchdb_magic_cookie
couchdb couchdb/adminpass password $password_couchdb_admin
couchdb couchdb/adminpass seen true
couchdb couchdb/adminpass_again password $password_couchdb_admin
couchdb couchdb/adminpass_again seen true" | debconf-set-selections
DEBIAN_FRONTEND=noninteractive # apt-get install -y --force-yes couchdb
ynh_install_extra_app_dependencies \
--repo="deb https://apache.jfrog.io/artifactory/couchdb-deb/ $(lsb_release -c -s) main" \
--key="https://couchdb.apache.org/repo/keys.asc" \
--package="couchdb"
# upgrade couchdb config # upgrade couchdb config
ynh_script_progression --message="Updating couchdb configuration file..." --weight=1 ynh_script_progression --message="Updating couchdb configuration file..." --weight=1
@ -39,49 +42,44 @@ ynh_add_config --template="../conf/05-flohmarkt.ini" --destination="/opt/couchdb
chown root:couchdb /opt/couchdb/etc/local.d/05-flohmarkt.ini chown root:couchdb /opt/couchdb/etc/local.d/05-flohmarkt.ini
chmod 640 /opt/couchdb/etc/local.d/05-flohmarkt.ini chmod 640 /opt/couchdb/etc/local.d/05-flohmarkt.ini
# start couchdb to pick up changes # start couchdb
ynh_script_progression --message="Starting couchdb..." --weight=1 ynh_script_progression --message="Starting couchdb..." --weight=1
ynh_systemd_action --service_name=couchdb --action="start" flohmarkt_ynh_start_couchdb
systemctl status couchdb
# install upgrade for flohmarkt # install upgrade for flohmarkt
ynh_script_progression --message="Upgrading flohmarkt..." --weight=4 ynh_script_progression --message="Upgrading flohmarkt..." --weight=4
ynh_setup_source --dest_dir="$install_dir/$app/" ynh_setup_source --dest_dir="$flohmarkt_app_dir/"
ynh_script_progression --message="Upgrading flohmarkt python dependencies..." --weight=5 ynh_script_progression --message="Upgrading flohmarkt python dependencies..." --weight=6
# upgrade python environment / install new dependencies # upgrade python environment / install new dependencies
( flohmarkt_ynh_venv_requirements
set +o nounset
source "$install_dir/venv/bin/activate"
set -o nounset
set -x
$install_dir/venv/bin/python3 -m ensurepip
$install_dir/venv/bin/pip3 install -r "$install_dir/$app/requirements.txt"
)
# upgrade flohmarkt.conf # upgrade flohmarkt.conf
ynh_script_progression --message="Upgrading flohmarkt configuration..." --weight=1 ynh_script_progression --message="Upgrading flohmarkt configuration..." --weight=1
ynh_add_config --template="../conf/flohmarkt.conf" --destination="$install_dir/$app/flohmarkt.conf" ynh_add_config --template="../conf/flohmarkt.conf" --destination="$flohmarkt_app_dir/flohmarkt.conf"
ynh_script_progression --message="Upgrading flohmarkt couchdb..." --weight=10 ynh_script_progression --message="Upgrading flohmarkt couchdb..." --weight=10
# run initialize_couchdb.py # run initialize_couchdb.py
( (
set +o nounset set +o nounset
source "$install_dir/venv/bin/activate" source "$flohmarkt_venv_dir/bin/activate"
set -o nounset set -o nounset
cd "$install_dir/$app" cd "$flohmarkt_app_dir"
# initialize_couchdb seems to re-try on connect problems endlessly blocking the yunohost api # initialize_couchdb seems to re-try on connect problems endlessly blocking the yunohost api
# give it 45 seconds to finish and then fail # give it 45 seconds to finish and then fail
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/13 # https://codeberg.org/ChriChri/flohmarkt_ynh/issues/13
timeout 45 python3 initialize_couchdb.py $password_couchdb_admin $password_couchdb_flohmarkt timeout 45 python3 initialize_couchdb.py $password_couchdb_admin $password_couchdb_flohmarkt
) )
# set file permissions for install dir
flohmarkt_ynh_set_permission
# systemd service upgrade # systemd service upgrade
ynh_script_progression --message="Upgrading flohmarkt systemd service..." --weight=1 ynh_script_progression --message="Upgrading flohmarkt systemd service..." --weight=1
ynh_add_systemd_config ynh_add_systemd_config --service=$flohmarkt_filename
# INTEGRATE SERVICE IN YUNOHOST # INTEGRATE SERVICE IN YUNOHOST
ynh_script_progression --message="Integrating service in YunoHost..." --weight=2 ynh_script_progression --message="Integrating service in YunoHost..." --weight=2
yunohost service add $app --description="A decentral federated small advertisement platform" --log="/var/log/$app/$app.log" yunohost service add $flohmarkt_filename --description="A decentral federated small advertisement platform" --log="$flohmarkt_logfile"
# upgrade logrotate # upgrade logrotate
ynh_script_progression --message="Upgrading logrotate configuration..." --weight=1 ynh_script_progression --message="Upgrading logrotate configuration..." --weight=1
@ -94,10 +92,10 @@ ynh_add_nginx_config
# UPGRADE FAIL2BAN # UPGRADE FAIL2BAN
ynh_script_progression --message="Reconfiguring Fail2Ban..." --weight=3 ynh_script_progression --message="Reconfiguring Fail2Ban..." --weight=3
ynh_add_fail2ban_config --logpath="/var/log/$app/$app.log" --failregex='INFO: +<HOST>:\d+ - "POST /token HTTP/\d+\.\d+" 403 Forbidden' --max_retry=5 ynh_add_fail2ban_config --logpath="$flohmarkt_logfile" --failregex='INFO: +<HOST>:\d+ - "POST /token HTTP/\d+\.\d+" 403 Forbidden' --max_retry=5
# start flohmarkt # start flohmarkt
ynh_script_progression --message="Starting flohmarkt..." --weight=3 ynh_script_progression --message="Starting flohmarkt..." --weight=3
ynh_systemd_action --service_name=$app --action="start" --line_match="INFO: Application startup complete." flohmarkt_ynh_start_service
ynh_script_progression --message="Upgrade of $app completed" --last ynh_script_progression --message="Upgrade of $app completed" --last

View file

@ -1,13 +1,12 @@
test_format = 1.0 test_format = 1.0
[default] [default]
# still in development # changing the url of a fediverse instance doesn't make sense
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/4 # it'll break federation anyway so we do not offer it
# https://codeberg.org/ChriChri/flohmarkt_ynh/issues/11 exclude = ["change_url"]
exclude = ["install.multi", "change_url", "install_subdir"]
args.password_couchdb_admin = "TestPasswordCouchDBadmin" args.password_couchdb_admin = "TestPasswordCouchDBadmin"
args.password_couchdb_flohmarkt = "TestPasswordCouchDBflohmarkt" args.password_couchdb_flohmarkt = "TestPasswordCouchDBflohmarkt"
test_upgrade_from.9df93bc48e.name = "Upgrade from ynh2" test_upgrade_from.217fc0e.name = "Upgrade from ynh4"