YunoHost/moulinette
1
0
Fork 0
mirror of https://github.com/YunoHost/moulinette.git synced 2024-09-03 20:06:31 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #102 from jeromelebleu/test

Browse source 
Refactor the moulinette core
This commit is contained in:
Alexis Gavoty 2014-03-26 16:45:44 +01:00
commit 0e2281de43
45 changed files with 2767 additions and 1290 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.gitignore vendored
View file

@ -10,6 +10,7 @@ build
eggs
parts
bin
cache
var
sdist
develop-eggs
@ -22,10 +23,12 @@ pip-log.txt
.coverage
.tox
#Translations
# Translations
*.mo
#Mr Developer
# Mr Developer
.mr.developer.cfg
# Moulinette
doc/*.json
src/moulinette/package.py

204
README.md
View file

@ -1,172 +1,48 @@
YunoHost CLI
============
Moulinette
==========
The *moulinette* is a Python package that allows to quickly and easily
deploy an application for different interfaces.
Specifications
--------------
Overview
--------
Initially, the moulinette was an application made for the
[YunoHost](https://yunohost.org/) project in order to regroup all its
related operations into a single program called *moulinette*. Those
operations were available from a command-line interface and a Web server
providing an API. Moreover, the usage of these operations (e.g.
required/optional arguments) was defined into a simple yaml file -
called *actionsmap*. This file was parsed in order to construct an
*ArgumentParser* object and to parse the command arguments to process
the proper operation.
During a long refactoring with the goal of unify both interfaces, the
idea to separate the core of the YunoHost operations has emerged.
The core kept the same name *moulinette* and try to follow the same
initial principle. An [Actions Map](#actions-map) - which defines
available operations and their usage - is parsed and it's used to
process an operation from several unified [Interfaces](#interfaces). It
also supports a configuration mechanism - which allows to restrict an
operation on an interface for example (see
[Authenticators](#authenticators)).
### User
### Actions Map
...
yunohost user list [-h] [--fields FIELDS [FIELDS ...]] [-o OFFSET]
[-f FILTER] [-l LIMIT]
yunohost user create [-h] [-u USERNAME] [-l LASTNAME] [-f FIRSTNAME]
[-p PASSWORD] [-m MAIL]
yunohost user delete [-h] [--purge] users [users ...]
yunohost user update [-h] [-f FIRSTNAME]
[--add-mailalias MAIL [MAIL ...]] [-m MAIL]
[-l LASTNAME]
[--remove-mailforward MAIL [MAIL ...]]
[--remove-mailalias MAIL [MAIL ...]]
[--add-mailforward MAIL [MAIL ...]]
[-p PASSWORD]
user
yunohost user info [-h] [-u USER] [-m MAIL]
### Domain
yunohost domain list [-h] [-l LIMIT] [-o OFFSET] [-f FILTER]
yunohost domain add [-h] [domain [domain ...]]
yunohost domain remove [-h] [domain [domain ...]]
yunohost domain info [-h] domain
yunohost domain renewcert [-h] domain
### App
yunohost app updatelist [-h] [-u URL]
yunohost app list [-h] [--fields FIELDS [FIELDS ...]] [-o OFFSET]
[-f FILTER] [-l LIMIT]
yunohost app install [-h] [-d DOMAIN] [--public] [-l LABEL] [-p PATH]
[--protected]
app
yunohost app remove [-h] app [app ...]
yunohost app upgrade [-h] [app [app ...]]
yunohost app info [-h] app
yunohost app addaccess [-h] [-u USER [USER ...]] app [app ...]
yunohost app removeaccess [-h] [-u USER [USER ...]] app [app ...]
### Firewall
yunohost firewall list [-h]
yunohost firewall allow [-h] {UDP,TCP,Both} port name
yunohost firewall disallow [-h] name
### Monitoring
yunohost monitor disk [-h] [-m MOUNTPOINT] [-t] [-f] [-H]
yunohost monitor network [-h] [-u] [-i] [-H]
yunohost monitor system [-h] [-m] [-u] [-i] [-p] [-c] [-H]
### Services
yunohost service status [-h] [NAME [NAME ...]]
yunohost service start [-h] NAME [NAME ...]
yunohost service stop [-h] NAME [NAME ...]
yunohost service enable [-h] NAME [NAME ...]
yunohost service disable [-h] NAME [NAME ...]
### Tools
yunohost tools postinstall [-h] [-d DOMAIN] [-p PASSWORD]
yunohost tools maindomain [-h] [-o OLD_DOMAIN] [-n NEW_DOMAIN]
yunohost tools adminpw [-h] [-o OLD_PASSWORD] [-n NEW_PASSWORD]
yunohost tools ldapinit [-h] [-d DOMAIN]
How to use "as is" ?
--------------------
The executable file is yunohost, for example:
./yunohost user create
Contribute / FAQ
----------------
*What a lovely idea !* :)
### Dafuq is dat moulinette ?
We decided to regroup all YunoHost related operations into a single program called "moulinette". This will allow us to entirely manipulate our YunoHost instances through a wonderful CLI. Additionally the web interface will just have to call the same "moulinette" functions. Magic power inside :p
### Important files
* `` yunohost `` File executed on function calling - i.e `` ./yunohost user create ``.
* `` action_map.yml `` Defines all CLI actions and links arguments.
* `` yunohost.py `` Contains all YunoHost functions likely to be shared between moulinette files. Also contains service connections classes (erk).
* `` yunohost_*.py `` Files containing action functions. `` * `` is the category: user, domain, firewall, etc.
### How to add a function ?
1. Check if the action is already in the `` action_map.yml `` file. If not, follow the file documentation to add it.
2. Also check if the file `` yunohost_category.py `` is created in the working tree. If not, just create it (you may take example of `` yunohost_user.py `` file).
3. Add your function `` category_action() `` in this file - i.e `` user_create() ``
**Note:** `` category_action() `` takes one parameter,`` args `` which contains the arguments passed to the command. Refers to `` action_map.yml `` documentation for more informations.
### Error handling
Moulinette has a unified way to handle errors. First, you need to import the ``YunoHostError`` exception:
`` from yunohost import YunoHostError ``
Then you need to raise errors like this:
`` raise YunoHostError(<ERROR CODE>, <MESSAGE>) ``
For example:
`` raise YunoHostError(125, _("Interrupted, user not created")) ``
**Note:** Standard error codes can be found in the ``YunoHostError`` class in `` yunohost.py `` file.
### Print results
Moulinette also have a unified way to print results. In fact we don't only print result for the CLI, but we also have to export the result in a JSON way.
Results are automatically printed OR exported, you don't have to print it yourself in the action's functions. Your function just need is to return results as a dictionary, for example:
`` return { 'Fullname' : 'Homer Simpson', 'Mail' : 'homer@simpson.org', 'Username' : 'hsimpson' } ``
### i18n
We will have to translate YunoHost, and we have already initialized i18n module in the moulinette. As a result, do not forget to put potentially translated strings into `` _() `` function. For example:
`` raise YunoHostError(125, _("Interrupted, user not created")) ``
### Git is pissing me off !
OK, this is the workflow !
**For gitlab:**
Development is handle with git branches and you have your own (i.e dev_beudbeud).
```
git clone git@dev.yunohost.org:moulinette.git
git checkout -b dev_beudbeud ``
git rebase origin/dev
```
Do your modifications, then :
```
git commit -am 'My commit message'
git pull origin dev (merge manually if conflicts)
git push origin dev_beudbeud
```
Then you could ask for a 'merge request' in gitlab.
**For github (here):**
Development is handle with forked repos and you have your own (i.e beudbeud/moulinette).
```
git clone https://github.com/beudbeud/moulinette.git ``
git checkout -b dev
git rebase origin/dev
```
Do your modifications, then:
```
git commit -am 'My commit message'
git remote add vanilla https://github.com/YunoHost/moulinette.git
git pull vanilla dev (merge manually if conflicts)
git push origin dev
```
Then you could ask for a 'pull request' in github.
### Interfaces
...
### Authenticators
...
Requirements
------------
* Python 2.7
* python-bottle (>= 0.10)
* python-gnupg (>= 0.3)
* python-ldap (>= 2.4)

46
bin/yunohost Executable file
View file

@ -0,0 +1,46 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import os.path
from_source = False
# Run from source
basedir = os.path.abspath('%s/../' % os.path.dirname(__file__))
if os.path.isdir('%s/moulinette' % basedir):
sys.path.insert(0, basedir)
from_source = True
from moulinette import init, cli, MoulinetteError
from moulinette.helpers import YunoHostError, colorize
## Main action
if __name__ == '__main__':
# Run from source
init(_from_source=from_source)
# Additional arguments
use_cache = True
if '--no-cache' in sys.argv:
use_cache = False
sys.argv.remove('--no-cache')
try:
args = list(sys.argv)
args.pop(0)
# Check that YunoHost is installed
if not os.path.isfile('/etc/yunohost/installed') \
and (len(args) < 2 or args[1] != 'tools' or args[2] != 'postinstall'):
raise YunoHostError(17, _("YunoHost is not correctly installed, please execute 'yunohost tools postinstall'"))
# Execute the action
ret = cli(['yunohost'], args, use_cache)
except YunoHostError as e:
# TODO: Remove this and associated import when yunohost package has been revisited
print(colorize(_("Error: "), 'red') + e.message)
sys.exit(e.code)
sys.exit(ret)

52
bin/yunohost-api Executable file
View file

@ -0,0 +1,52 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import os.path
from_source = False
# Run from source
basedir = os.path.abspath('%s/../' % os.path.dirname(__file__))
if os.path.isdir('%s/moulinette' % basedir):
sys.path.insert(0, basedir)
from_source = True
from moulinette import init, api, MoulinetteError
## Callbacks for additional routes
def is_installed():
"""
Check whether YunoHost is installed or not
"""
installed = False
if os.path.isfile('/etc/yunohost/installed'):
installed = True
return { 'installed': installed }
## Main action
if __name__ == '__main__':
# Run from source
init(_from_source=from_source)
# Additional arguments
use_cache = True
if '--no-cache' in sys.argv:
use_cache = False
sys.argv.remove('--no-cache')
# TODO: Add log argument
try:
# Run the server
api(['yunohost'], 6787,
{('GET', '/installed'): is_installed}, use_cache)
except MoulinetteError as e:
from moulinette.interfaces.cli import colorize
print(_('%s: %s' % (colorize(_('Error'), 'red'), e.strerror)))
sys.exit(e.errno)
sys.exit(0)

59
data/actionsmap/test.yml Normal file
View file

@ -0,0 +1,59 @@
#############################
# Global parameters #
#############################
_global:
configuration:
authenticate:
- api
authenticator:
default:
vendor: ldap
help: Admin Password
parameters:
uri: ldap://localhost:389
base_dn: dc=yunohost,dc=org
user_rdn: cn=admin
ldap-anonymous:
vendor: ldap
parameters:
uri: ldap://localhost:389
base_dn: dc=yunohost,dc=org
test-profile:
vendor: ldap
help: Admin Password (profile)
parameters:
uri: ldap://localhost:389
base_dn: dc=yunohost,dc=org
user_rdn: cn=admin
argument_auth: true
#############################
# Test Actions #
#############################
test:
actions:
non-auth:
api: GET /test/non-auth
configuration:
authenticate: false
auth:
api: GET /test/auth
configuration:
authenticate: all
auth-profile:
api: GET /test/auth-profile
configuration:
authenticate: all
authenticator: test-profile
auth-cli:
api: GET /test/auth-cli
configuration:
authenticate:
- cli
anonymous:
api: GET /test/anon
configuration:
authenticate: all
authenticator: ldap-anonymous
argument_auth: false

142
action_map.yml → data/actionsmap/yunohost.yml
View file

@ -29,12 +29,27 @@
#
##########################################################################
# TODO: Add patern for all this
#############################
# General args #
# Global parameters #
#############################
general_arguments:
_global:
configuration:
authenticate:
- api
authenticator:
default:
vendor: ldap
help: Admin Password
parameters:
uri: ldap://localhost:389
base_dn: dc=yunohost,dc=org
user_rdn: cn=admin
ldap-anonymous:
vendor: ldap
parameters:
uri: ldap://localhost:389
base_dn: dc=yunohost,dc=org
arguments:
-v:
full: --version
help: Display moulinette version
@ -74,42 +89,53 @@ user:
-u:
full: --username
help: Must be unique
ask: "Username"
pattern: '^[a-z0-9_]+$'
extra:
ask: "Username"
pattern:
- '^[a-z0-9_]+$'
- "Must be alphanumeric and underscore characters only"
-f:
full: --firstname
ask: "Firstname"
extra:
ask: "Firstname"
-l:
full: --lastname
ask: "Lastname"
extra:
ask: "Lastname"
-m:
full: --mail
help: Main mail address must be unique
ask: "Mail address"
pattern: '^[\w.-]+@[\w.-]+\.[a-zA-Z]{2,6}$'
extra:
ask: "Mail address"
pattern:
- '^[\w.-]+@[\w.-]+\.[a-zA-Z]{2,6}$'
- "Must be a valid email address (e.g. someone@domain.org)"
-p:
full: --password
ask: "User password"
password: yes
extra:
password: "User password"
### user_delete()
delete:
action_help: Delete user
api: 'DELETE /users/{users}'
api: 'DELETE /users/<users>'
arguments:
-u:
full: --users
help: Username of users to delete
ask: "Users to delete"
pattern: '^[a-z0-9_]+$'
nargs: "*"
extra:
ask: "Users to delete"
pattern:
- '^[a-z0-9_]+$'
- "Must be alphanumeric and underscore characters only"
--purge:
action: store_true
### user_update()
update:
action_help: Update user informations
api: 'PUT /users/{username}'
api: 'PUT /users/<username>'
arguments:
username:
help: Username of user to update
@ -143,7 +169,7 @@ user:
### user_info()
info:
action_help: Get user informations
api: 'GET /users/{username}'
api: 'GET /users/<username>'
arguments:
username:
help: Username or mail to get informations
@ -179,7 +205,10 @@ domain:
domains:
help: Domain name to add
nargs: '+'
pattern: '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
extra:
pattern:
- '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
- "Must be a valid domain name (e.g. my-domain.org)"
-m:
full: --main
help: Is the main domain
@ -197,16 +226,22 @@ domain:
domains:
help: Domain(s) to delete
nargs: "+"
pattern: '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
extra:
pattern:
- '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
- "Must be a valid domain name (e.g. my-domain.org)"
### domain_info()
info:
action_help: Get domain informations
api: 'GET /domains/{domain}'
api: 'GET /domains/<domain>'
arguments:
domain:
help: ""
pattern: '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
extra:
pattern:
- '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
- "Must be a valid domain name (e.g. my-domain.org)"
#############################
@ -241,8 +276,11 @@ app:
-n:
full: --name
help: Name of the list to remove
ask: "List to remove"
pattern: '^[a-z0-9_]+$'
extra:
ask: "List to remove"
pattern:
- '^[a-z0-9_]+$'
- "Must be alphanumeric and underscore characters only"
### app_list()
list:
@ -266,7 +304,7 @@ app:
### app_info()
info:
action_help: Get app info
api: GET /app/{app}
api: GET /app/<app>
arguments:
app:
help: Specific app ID
@ -290,7 +328,10 @@ app:
-u:
full: --user
help: Allowed app map for a user
pattern: '^[a-z0-9_]+$'
extra:
pattern:
- '^[a-z0-9_]+$'
- "Must be alphanumeric and underscore characters only"
### app_install() TODO: Write help
@ -333,7 +374,7 @@ app:
### app_setting()
setting:
action_help: Set ou get an app setting value
api: GET /app/{app}/setting
api: GET /app/<app>/setting
arguments:
app:
help: App ID
@ -350,7 +391,7 @@ app:
### app_service()
service:
action_help: Add or remove a YunoHost monitored service
api: POST /app/service/{service}
api: POST /app/service/<service>
arguments:
service:
help: Service to add/remove
@ -375,7 +416,10 @@ app:
arguments:
port:
help: Port to check
pattern: '^([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$'
extra:
pattern:
- '^([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$'
- "Must be a valid port number (i.e. 0-65535)"
### app_checkurl()
checkurl:
@ -651,8 +695,11 @@ service:
-n:
full: --number
help: Number of lines to display
pattern: '^[0-9]+$'
default: "50"
extra:
pattern:
- '^[0-9]+$'
- "Must be a valid number"
#############################
@ -683,7 +730,10 @@ firewall:
arguments:
port:
help: Port to open
pattern: '^([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$'
extra:
pattern:
- '^([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$'
- "Must be a valid port number (i.e. 0-65535)"
protocol:
help: Protocol associated with port
choices:
@ -817,12 +867,12 @@ tools:
arguments:
-o:
full: --old-password
ask: "Current admin password"
password: yes
extra:
password: "Current admin password"
-n:
full: --new-password
ask: "New admin password"
password: yes
extra:
password: "New admin password"
### tools_maindomain()
maindomain:
@ -831,10 +881,17 @@ tools:
arguments:
-o:
full: --old-domain
pattern: '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
extra:
pattern:
- '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
- "Must be a valid domain name (e.g. my-domain.org)"
-n:
full: --new-domain
pattern: '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
extra:
ask: "New main domain"
pattern:
- '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
- "Must be a valid domain name (e.g. my-domain.org)"
### tools_postinstall()
postinstall:
@ -844,13 +901,16 @@ tools:
-d:
full: --domain
help: YunoHost main domain
ask: "Main domain"
pattern: '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
extra:
ask: "Main domain"
pattern:
- '^([a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)(\.[a-zA-Z0-9]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)*(\.[a-zA-Z]{1}([a-zA-Z0-9\-]*[a-zA-Z0-9])*)$'
- "Must be a valid domain name (e.g. my-domain.org)"
-p:
full: --password
help: YunoHost admin password
ask: "New admin password"
password: yes
extra:
password: "New admin password"
--dyndns:
help: Subscribe domain to a DynDNS service
action: store_true
@ -869,7 +929,7 @@ tools:
upgrade:
action_help: YunoHost upgrade
api: POST /upgrade
### tools_upgradelog()
upgradelog:
action_help: Show dpkg log

8
bash/yunohost_cli → data/moulinette_cli
View file

@ -7,14 +7,14 @@ COMPREPLY=()
argc=${COMP_CWORD}
cur="${COMP_WORDS[argc]}"
prev="${COMP_WORDS[argc-1]}"
prev="${COMP_WORDS[argc-1]}"
opts=$(yunohost -h | sed -n "/usage/,/}/p" | awk -F"{" '{print $2}' | awk -F"}" '{print $1}' | tr ',' ' ')
if [[ $argc = 1 ]];
then
COMPREPLY=( $(compgen -W "$opts --help" -- $cur ) )
fi
if [[ "$prev" != "--help" ]];
then
if [[ $argc = 2 ]];
@ -23,9 +23,9 @@ then
COMPREPLY=( $(compgen -W "$opts2 --help" -- $cur ) )
elif [[ $argc = 3 ]];
then
COMPREPLY=( $(compgen -W "--help" $cur ) )
COMPREPLY=( $(compgen -W "--help" $cur ) )
fi
else
else
COMPREPLY=()
fi

0
lib/test/__init__.py Executable file
View file

19
lib/test/test.py Normal file
View file

@ -0,0 +1,19 @@
def test_non_auth():
return {'action': 'non-auth'}
def test_auth(auth):
return {'action': 'auth',
'authenticator': 'default', 'authenticate': 'all'}
def test_auth_profile(auth):
return {'action': 'auth-profile',
'authenticator': 'test-profile', 'authenticate': 'all'}
def test_auth_cli():
return {'action': 'auth-cli',
'authenticator': 'default', 'authenticate': ['cli']}
def test_anonymous():
return {'action': 'anonymous',
'authenticator': 'ldap-anonymous', 'authenticate': 'all'}

0
lib/yunohost/__init__.py Executable file
View file

16
yunohost_app.py → lib/yunohost/app.py
View file

@ -23,6 +23,9 @@
Manage apps
"""
import logging
logging.warning('the module yunohost.app has not been revisited and updated yet')
import os
import sys
import json
@ -33,10 +36,11 @@ import time
import re
import socket
import urlparse
from yunohost import YunoHostError, YunoHostLDAP, win_msg, random_password, is_true, validate
from yunohost_domain import domain_list, domain_add
from yunohost_user import user_info, user_list
from yunohost_hook import hook_exec, hook_add, hook_remove
from domain import domain_list, domain_add
from user import user_info, user_list
from hook import hook_exec, hook_add, hook_remove
from moulinette.helpers import YunoHostError, YunoHostLDAP, win_msg, random_password, is_true, validate
repo_path = '/var/cache/yunohost/repo'
apps_path = '/usr/share/yunohost/apps'
@ -836,7 +840,7 @@ def app_ssowatconf():
unprotected_regex = []
protected_urls = []
protected_regex = []
apps = {}
for app in app_list()['Apps']:
if _is_installed(app['ID']):
@ -896,7 +900,7 @@ def app_ssowatconf():
conf_dict['unprotected_regex'] = unprotected_regex
conf_dict['protected_regex'] = protected_regex
conf_dict['users'] = users
with open('/etc/ssowat/conf.json', 'wb') as f:
json.dump(conf_dict, f)

6
yunohost_backup.py → lib/yunohost/backup.py
View file

@ -23,12 +23,16 @@
Manage backups
"""
import logging
logging.warning('the module yunohost.backup has not been revisited and updated yet')
import os
import sys
import json
import yaml
import glob
from yunohost import YunoHostError, YunoHostLDAP, validate, colorize, win_msg
from moulinette.helpers import YunoHostError, YunoHostLDAP, validate, colorize, win_msg
def backup_init(helper=False):
"""

2
bash/checkupdate → lib/yunohost/data/checkupdate
View file

@ -62,6 +62,6 @@ then
exit 3
else
if [ -x /usr/bin/apt-listchanges ] ; then
/usr/bin/apt-listchanges --which=both -f text $DEBS > /tmp/yunohost/changelog 2>/dev/null
/usr/bin/apt-listchanges --which=both -f text $DEBS > /tmp/yunohost/changelog 2>/dev/null
fi
fi

5
firewall.yml → lib/yunohost/data/firewall.yml
View file

@ -1,6 +1,6 @@
UPNP:
UPNP:
cron: false
ports:
ports:
TCP: [22, 25, 53, 80, 443, 465, 993, 5222, 5269, 5290]
UDP: [53, 137, 138]
ipv4:
@ -9,4 +9,3 @@ ipv4:
ipv6:
TCP: [22]
UDP: [53]

0
ldap_scheme.yml → lib/yunohost/data/ldap_scheme.yml
View file

0
services.yml → lib/yunohost/data/services.yml
View file

0
config/upgrade → lib/yunohost/data/upgrade
View file

8
yunohost_domain.py → lib/yunohost/domain.py
View file

@ -23,6 +23,9 @@
Manage domains
"""
import logging
logging.warning('the module yunohost.backup has not been revisited and updated yet')
import os
import sys
import datetime
@ -32,8 +35,9 @@ import json
import yaml
import requests
from urllib import urlopen
from yunohost import YunoHostError, YunoHostLDAP, win_msg, colorize, validate, get_required_args
from yunohost_dyndns import dyndns_subscribe
from dyndns import dyndns_subscribe
from moulinette.helpers import YunoHostError, YunoHostLDAP, win_msg, colorize, validate, get_required_args
def domain_list(filter=None, limit=None, offset=None):

6
yunohost_dyndns.py → lib/yunohost/dyndns.py
View file

@ -23,13 +23,17 @@
Subscribe and Update DynDNS Hosts
"""
import logging
logging.warning('the module yunohost.dyndns has not been revisited and updated yet')
import os
import sys
import requests
import json
import glob
import base64
from yunohost import YunoHostError, YunoHostLDAP, validate, colorize, win_msg
from moulinette.helpers import YunoHostError, YunoHostLDAP, validate, colorize, win_msg
def dyndns_subscribe(subscribe_host="dyndns.yunohost.org", domain=None, key=None):
"""

8
yunohost_firewall.py → lib/yunohost/firewall.py
View file

@ -23,6 +23,9 @@
Manage firewall rules
"""
import logging
logging.warning('the module yunohost.firewall has not been revisited and updated yet')
import os
import sys
try:
@ -36,8 +39,9 @@ except ImportError:
sys.stderr.write('Error: Yunohost CLI Require yaml lib\n')
sys.stderr.write('apt-get install python-yaml\n')
sys.exit(1)
from yunohost import YunoHostError, win_msg
from yunohost_hook import hook_callback
from hook import hook_callback
from moulinette.helpers import YunoHostError, win_msg
def firewall_allow(protocol=None, port=None, ipv6=None, upnp=False):

6
yunohost_hook.py → lib/yunohost/hook.py
View file

@ -23,11 +23,15 @@
Manage hooks
"""
import logging
logging.warning('the module yunohost.hook has not been revisited and updated yet')
import os
import sys
import re
import json
from yunohost import YunoHostError, YunoHostLDAP, win_msg, colorize
from moulinette.helpers import YunoHostError, YunoHostLDAP, win_msg, colorize
hook_folder = '/usr/share/yunohost/hooks/'

8
yunohost_monitor.py → lib/yunohost/monitor.py
View file

@ -23,6 +23,9 @@
Monitoring functions
"""
import logging
logging.warning('the module yunohost.monitor has not been revisited and updated yet')
import re
import json
import time
@ -34,10 +37,11 @@ import os.path
import cPickle as pickle
from urllib import urlopen
from datetime import datetime, timedelta
from yunohost import YunoHostError, win_msg
from yunohost_service import (service_enable, service_disable,
from service import (service_enable, service_disable,
service_start, service_stop, service_status)
from moulinette.helpers import YunoHostError, win_msg
glances_uri = 'http://127.0.0.1:61209'
stats_path = '/var/lib/yunohost/stats'
crontab_path = '/etc/cron.d/yunohost-monitor'

10
yunohost_service.py → lib/yunohost/service.py
View file

@ -23,11 +23,15 @@
Manage services
"""
import logging
logging.warning('the module yunohost.service has not been revisited and updated yet')
import yaml
import glob
import subprocess
import os.path
from yunohost import YunoHostError, win_msg
from moulinette.helpers import YunoHostError, win_msg
def service_start(names):
@ -169,7 +173,7 @@ def service_log(name, number=50):
if name not in services.keys():
raise YunoHostError(1, _("Unknown service '%s'") % service)
if 'log' in services[name]:
log_list = services[name]['log']
result = {}
@ -253,7 +257,7 @@ def _tail(file, n, offset=None):
pos = f.tell()
lines = f.read().splitlines()
if len(lines) >= to_read or pos == 0:
return lines[-to_read:offset and -offset or None]
return lines[-to_read:offset and -offset or None]
avg_line_length *= 1.3
except IOError: return []

20
yunohost_tools.py → lib/yunohost/tools.py
View file

@ -23,6 +23,9 @@
Specific tools
"""
import logging
logging.warning('the module yunohost.tools has not been revisited and updated yet')
import os
import sys
import yaml
@ -32,11 +35,12 @@ import subprocess
import requests
import json
from subprocess import Popen, PIPE
from yunohost import YunoHostError, YunoHostLDAP, validate, colorize, get_required_args, win_msg
from yunohost_domain import domain_add, domain_list
from yunohost_dyndns import dyndns_subscribe
from yunohost_backup import backup_init
from yunohost_app import app_ssowatconf
from domain import domain_add, domain_list
from dyndns import dyndns_subscribe
from backup import backup_init
from app import app_ssowatconf
from moulinette.helpers import YunoHostError, YunoHostLDAP, validate, colorize, get_required_args, win_msg
def tools_ldapinit(password=None):
@ -276,7 +280,7 @@ def tools_update():
raise YunoHostError(17, _("Error during update"))
else:
return { "Update" : stdout.splitlines() }
def tools_changelog():
"""
@ -289,7 +293,7 @@ def tools_changelog():
return { "Changelog" : read_data.splitlines() }
else:
raise YunoHostError(17, _("Launch update before upgrade"))
def tools_upgrade():
"""
@ -312,7 +316,7 @@ def tools_upgrade():
win_msg( _("Upgrade in progress"))
else:
raise YunoHostError(17, _("Launch update before upgrade"))
def tools_upgradelog():
"""

14
yunohost_user.py → lib/yunohost/user.py
View file

@ -23,6 +23,9 @@
Manage users
"""
import logging
logging.warning('the module yunohost.user has not been revisited and updated yet')
import os
import sys
import ldap
@ -30,9 +33,10 @@ import crypt
import random
import string
import getpass
from yunohost import YunoHostError, YunoHostLDAP, win_msg, colorize, validate, get_required_args
from yunohost_domain import domain_list
from yunohost_hook import hook_callback
from domain import domain_list
from hook import hook_callback
from moulinette.helpers import YunoHostError, YunoHostLDAP, win_msg, colorize, validate, get_required_args
def user_list(fields=None, filter=None, limit=None, offset=None):
"""
@ -117,7 +121,7 @@ def user_create(username, firstname, lastname, mail, password):
uid = str(random.randint(200, 99999))
uid_check = os.system("getent passwd " + uid)
gid_check = os.system("getent group " + uid)
# Adapt values for LDAP
fullname = firstname + ' ' + lastname
rdn = 'uid=' + username + ',ou=users'
@ -139,7 +143,7 @@ def user_create(username, firstname, lastname, mail, password):
'uidNumber' : uid,
'homeDirectory' : '/home/' + username,
'loginShell' : '/bin/false'
}
if yldap.add(rdn, attr_dict):

108
moulinette/__init__.py Executable file
View file

@ -0,0 +1,108 @@
# -*- coding: utf-8 -*-
__title__ = 'moulinette'
__version__ = '0.1'
__author__ = ['Kload',
'jlebleu',
'titoko',
'beudbeud',
'npze']
__license__ = 'AGPL 3.0'
__credits__ = """
Copyright (C) 2014 YUNOHOST.ORG
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program; if not, see http://www.gnu.org/licenses
"""
__all__ = [
'init', 'api', 'cli',
'init_interface', 'MoulinetteError',
]
from moulinette.core import init_interface, MoulinetteError
## Package functions
def init(**kwargs):
"""Package initialization
Initialize directories and global variables. It must be called
before any of package method is used - even the easy access
functions.
Keyword arguments:
- **kwargs -- See core.Package
At the end, the global variable 'pkg' will contain a Package
instance. See core.Package for available methods and variables.
"""
import sys
import __builtin__
from moulinette.core import Package, install_i18n
__builtin__.__dict__['pkg'] = Package(**kwargs)
# Initialize internationalization
install_i18n()
# Add library directory to python path
sys.path.insert(0, pkg.libdir)
## Easy access to interfaces
def api(namespaces, port, routes={}, use_cache=True):
"""Web server (API) interface
Run a HTTP server with the moulinette for an API usage.
Keyword arguments:
- namespaces -- The list of namespaces to use
- port -- Port number to run on
- routes -- A dict of additional routes to add in the form of
{(method, uri): callback}
- use_cache -- False if it should parse the actions map file
instead of using the cached one
"""
moulinette = init_interface('api',
kwargs={'routes': routes},
actionsmap={'namespaces': namespaces,
'use_cache': use_cache})
moulinette.run(port)
def cli(namespaces, args, use_cache=True):
"""Command line interface
Execute an action with the moulinette from the CLI and print its
result in a readable format.
Keyword arguments:
- namespaces -- The list of namespaces to use
- args -- A list of argument strings
- use_cache -- False if it should parse the actions map file
instead of using the cached one
"""
from moulinette.interfaces.cli import colorize
try:
moulinette = init_interface('cli',
actionsmap={'namespaces': namespaces,
'use_cache': use_cache})
moulinette.run(args)
except MoulinetteError as e:
print(_('%s: %s' % (colorize(_('Error'), 'red'), e.strerror)))
return e.errno
return 0

595
moulinette/actionsmap.py Normal file
View file

@ -0,0 +1,595 @@
# -*- coding: utf-8 -*-
import os
import re
import errno
import logging
import yaml
import cPickle as pickle
from collections import OrderedDict
from moulinette.core import (MoulinetteError, MoulinetteLock)
from moulinette.interfaces import BaseActionsMapParser
## Actions map Signals -------------------------------------------------
class ActionsMapSignals(object):
"""Actions map's Signals interface
Allow to easily connect signals of the actions map to handlers. They
can be given as arguments in the form of { signal: handler }.
"""
def __init__(self, **kwargs):
# Initialize handlers
for s in self.signals:
self.clear_handler(s)
# Iterate over signals to connect
for s, h in kwargs.items():
self.set_handler(s, h)
def set_handler(self, signal, handler):
"""Set the handler for a signal"""
if signal not in self.signals:
raise ValueError("unknown signal '%s'" % signal)
setattr(self, '_%s' % signal, handler)
def clear_handler(self, signal):
"""Clear the handler of a signal"""
if signal not in self.signals:
raise ValueError("unknown signal '%s'" % signal)
setattr(self, '_%s' % signal, self._notimplemented)
## Signals definitions
"""The list of available signals"""
signals = { 'authenticate', 'prompt' }
def authenticate(self, authenticator, help):
"""Process the authentication
Attempt to authenticate to the given authenticator and return
it.
It is called when authentication is needed (e.g. to process an
action).
Keyword arguments:
- authenticator -- The authenticator object to use
- help -- A help message for the authenticator
Returns:
The authenticator object
"""
if authenticator.is_authenticated:
return authenticator
return self._authenticate(authenticator, help)
def prompt(self, message, is_password=False, confirm=False):
"""Prompt for a value
Prompt the interface for a parameter value which is a password
if 'is_password' and must be confirmed if 'confirm'.
Is is called when a parameter value is needed and when the
current interface should allow user interaction (e.g. to parse
extra parameter 'ask' in the cli).
Keyword arguments:
- message -- The message to display
- is_password -- True if the parameter is a password
- confirm -- True if the value must be confirmed
Returns:
The collected value
"""
return self._prompt(message, is_password, confirm)
@staticmethod
def _notimplemented(**kwargs):
raise NotImplementedError("this signal is not handled")
shandler = ActionsMapSignals()
## Extra parameters ----------------------------------------------------
# Extra parameters definition
class _ExtraParameter(object):
"""
Argument parser for an extra parameter.
It is a pure virtual class that each extra parameter classes must
implement.
"""
def __init__(self, iface):
# TODO: Add conn argument which contains authentification object
self.iface = iface
## Required variables
# Each extra parameters classes must overwrite these variables.
"""The extra parameter name"""
name = None
## Optional variables
# Each extra parameters classes can overwrite these variables.
"""A list of interface for which the parameter doesn't apply"""
skipped_iface = {}
## Virtual methods
# Each extra parameters classes can implement these methods.
def __call__(self, parameter, arg_name, arg_value):
"""
Parse the argument
Keyword arguments:
- parameter -- The value of this parameter for the action
- arg_name -- The argument name
- arg_value -- The argument value
Returns:
The new argument value
"""
return arg_value
@staticmethod
def validate(value, arg_name):
"""
Validate the parameter value for an argument
Keyword arguments:
- value -- The parameter value
- arg_name -- The argument name
Returns:
The validated parameter value
"""
return value
class AskParameter(_ExtraParameter):
"""
Ask for the argument value if possible and needed.
The value of this parameter corresponds to the message to display
when asking the argument value.
"""
name = 'ask'
skipped_iface = { 'api' }
def __call__(self, message, arg_name, arg_value):
# TODO: Fix asked arguments ordering
if arg_value:
return arg_value
try:
# Ask for the argument value
return shandler.prompt(message)
except NotImplementedError:
return arg_value
@classmethod
def validate(klass, value, arg_name):
# Allow boolean or empty string
if isinstance(value, bool) or (isinstance(value, str) and not value):
logging.debug("value of '%s' extra parameter for '%s' argument should be a string" \
% (klass.name, arg_name))
value = arg_name
elif not isinstance(value, str):
raise TypeError("Invalid type of '%s' extra parameter for '%s' argument" \
% (klass.name, arg_name))
return value
class PasswordParameter(AskParameter):
"""
Ask for the password argument value if possible and needed.
The value of this parameter corresponds to the message to display
when asking the password.
"""
name = 'password'
def __call__(self, message, arg_name, arg_value):
if arg_value:
return arg_value
try:
# Ask for the password
return shandler.prompt(message, True, True)
except NotImplementedError:
return arg_value
class PatternParameter(_ExtraParameter):
"""
Check if the argument value match a pattern.
The value of this parameter corresponds to a list of the pattern and
the message to display if it doesn't match.
"""
name = 'pattern'
def __call__(self, arguments, arg_name, arg_value):
pattern, message = (arguments[0], arguments[1])
if not re.match(pattern, arg_value or ''):
raise MoulinetteError(errno.EINVAL, message)
return arg_value
@staticmethod
def validate(value, arg_name):
# Tolerate string type
if isinstance(value, str):
logging.warning("value of 'pattern' extra parameter for '%s' argument should be a list" % arg_name)
value = [value, _("'%s' argument is not matching the pattern") % arg_name]
elif not isinstance(value, list) or len(value) != 2:
raise TypeError("Invalid type of 'pattern' extra parameter for '%s' argument" % arg_name)
return value
"""
The list of available extra parameters classes. It will keep to this list
order on argument parsing.
"""
extraparameters_list = [AskParameter, PasswordParameter, PatternParameter]
# Extra parameters argument Parser
class ExtraArgumentParser(object):
"""
Argument validator and parser for the extra parameters.
Keyword arguments:
- iface -- The running interface
"""
def __init__(self, iface):
self.iface = iface
self.extra = OrderedDict()
# Append available extra parameters for the current interface
for klass in extraparameters_list:
if iface in klass.skipped_iface:
continue
self.extra[klass.name] = klass
def validate(self, arg_name, parameters):
"""
Validate values of extra parameters for an argument
Keyword arguments:
- arg_name -- The argument name
- parameters -- A dict of extra parameters with their values
"""
# Iterate over parameters to validate
for p, v in parameters.items():
klass = self.extra.get(p, None)
if not klass:
# Remove unknown parameters
del parameters[p]
else:
# Validate parameter value
parameters[p] = klass.validate(v, arg_name)
return parameters
def parse(self, arg_name, arg_value, parameters):
"""
Parse argument with extra parameters
Keyword arguments:
- arg_name -- The argument name
- arg_value -- The argument value
- parameters -- A dict of extra parameters with their values
"""
# Iterate over available parameters
for p, klass in self.extra.items():
if p not in parameters.keys():
continue
# Initialize the extra parser
parser = klass(self.iface)
# Parse the argument
if isinstance(arg_value, list):
for v in arg_value:
r = parser(parameters[p], arg_name, v)
if r not in arg_value:
arg_value.append(r)
else:
arg_value = parser(parameters[p], arg_name, arg_value)
return arg_value
## Main class ----------------------------------------------------------
class ActionsMap(object):
"""Validate and process actions defined into an actions map
The actions map defines the features - and their usage - of an
application which will be available through the moulinette.
It is composed by categories which contain one or more action(s).
Moreover, the action can have specific argument(s).
This class allows to manipulate one or several actions maps
associated to a namespace. If no namespace is given, it will load
all available namespaces.
Keyword arguments:
- parser -- The BaseActionsMapParser derived class to use for
parsing the actions map
- namespaces -- The list of namespaces to use
- use_cache -- False if it should parse the actions map file
instead of using the cached one.
"""
def __init__(self, parser, namespaces=[], use_cache=True):
self.use_cache = use_cache
if not issubclass(parser, BaseActionsMapParser):
raise MoulinetteError(errno.EINVAL, _("Invalid parser class '%s'" % parser.__name__))
self._parser_class = parser
logging.debug("initializing ActionsMap for the interface '%s'" % parser.interface)
if len(namespaces) == 0:
namespaces = self.get_namespaces()
actionsmaps = OrderedDict()
# Iterate over actions map namespaces
for n in namespaces:
logging.debug("loading '%s' actions map namespace" % n)
if use_cache:
try:
# Attempt to load cache
with open('%s/actionsmap/%s.pkl' % (pkg.cachedir, n)) as f:
actionsmaps[n] = pickle.load(f)
# TODO: Switch to python3 and catch proper exception
except IOError:
self.use_cache = False
actionsmaps = self.generate_cache(namespaces)
break
else:
with open('%s/actionsmap/%s.yml' % (pkg.datadir, n)) as f:
actionsmaps[n] = yaml.load(f)
# Generate parsers
self.extraparser = ExtraArgumentParser(parser.interface)
self._parser = self._construct_parser(actionsmaps)
@property
def parser(self):
"""Return the instance of the interface's actions map parser"""
return self._parser
def get_authenticator(self, profile='default'):
"""Get an authenticator instance
Retrieve the authenticator for the given profile and return a
new instance.
Keyword arguments:
- profile -- An authenticator profile name
Returns:
A new _BaseAuthenticator derived instance
"""
try:
auth = self.parser.get_global_conf('authenticator', profile)[1]
except KeyError:
raise MoulinetteError(errno.EINVAL, _("Unknown authenticator profile '%s'") % profile)
else:
return auth()
def connect(self, signal, handler):
"""Connect a signal to a handler
Connect a signal emitted by actions map while processing to a
handler. Note that some signals need a return value.
Keyword arguments:
- signal -- The name of the signal
- handler -- The method to handle the signal
"""
global shandler
shandler.set_handler(signal, handler)
def process(self, args, timeout=0, **kwargs):
"""
Parse arguments and process the proper action
Keyword arguments:
- args -- The arguments to parse
- timeout -- The time period before failing if the lock
cannot be acquired for the action
- **kwargs -- Additional interface arguments
"""
# Parse arguments
arguments = vars(self.parser.parse_args(args, **kwargs))
for an, parameters in (arguments.pop('_extra', {})).items():
arguments[an] = self.extraparser.parse(an, arguments[an], parameters)
# Retrieve action information
namespace, category, action = arguments.pop('_tid')
func_name = '%s_%s' % (category, action.replace('-', '_'))
# Lock the moulinette for the namespace
with MoulinetteLock(namespace, timeout):
try:
mod = __import__('%s.%s' % (namespace, category),
globals=globals(), level=0,
fromlist=[func_name])
func = getattr(mod, func_name)
except (AttributeError, ImportError):
raise MoulinetteError(errno.ENOSYS, _('Function is not defined'))
else:
# Process the action
return func(**arguments)
@staticmethod
def get_namespaces():
"""
Retrieve available actions map namespaces
Returns:
A list of available namespaces
"""
namespaces = []
for f in os.listdir('%s/actionsmap' % pkg.datadir):
if f.endswith('.yml'):
namespaces.append(f[:-4])
return namespaces
@classmethod
def generate_cache(klass, namespaces=None):
"""
Generate cache for the actions map's file(s)
Keyword arguments:
- namespaces -- A list of namespaces to generate cache for
Returns:
A dict of actions map for each namespaces
"""
actionsmaps = {}
if not namespaces:
namespaces = klass.get_namespaces()
# Iterate over actions map namespaces
for n in namespaces:
logging.debug("generating cache for '%s' actions map namespace" % n)
# Read actions map from yaml file
am_file = '%s/actionsmap/%s.yml' % (pkg.datadir, n)
with open(am_file, 'r') as f:
actionsmaps[n] = yaml.load(f)
# Cache actions map into pickle file
with pkg.open_cachefile('%s.pkl' % n, 'w', subdir='actionsmap') as f:
pickle.dump(actionsmaps[n], f)
return actionsmaps
## Private methods
def _construct_parser(self, actionsmaps):
"""
Construct the parser with the actions map
Keyword arguments:
- actionsmaps -- A dict of multi-level dictionnary of
categories/actions/arguments list for each namespaces
Returns:
An interface relevant's parser object
"""
## Get extra parameters
if not self.use_cache:
_get_extra = lambda an, e: self.extraparser.validate(an, e)
else:
_get_extra = lambda an, e: e
## Add arguments to the parser
def _add_arguments(parser, arguments):
extras = {}
for argn, argp in arguments.items():
names = top_parser.format_arg_names(argn,
argp.pop('full', None))
try:
extra = argp.pop('extra')
arg_dest = (parser.add_argument(*names, **argp)).dest
extras[arg_dest] = _get_extra(arg_dest, extra)
except KeyError:
# No extra parameters
parser.add_argument(*names, **argp)
if extras:
parser.set_defaults(_extra=extras)
# Instantiate parser
top_parser = self._parser_class(shandler)
# Iterate over actions map namespaces
for n, actionsmap in actionsmaps.items():
# Retrieve global parameters
_global = actionsmap.pop('_global', {})
# -- Parse global configuration
if 'configuration' in _global:
# Set global configuration
top_parser.set_global_conf(_global['configuration'])
# -- Parse global arguments
if 'arguments' in _global:
try:
# Get global arguments parser
parser = top_parser.add_global_parser()
except AttributeError:
# No parser for global arguments
pass
else:
# Add arguments
_add_arguments(parser, _global['arguments'])
# -- Parse categories
for cn, cp in actionsmap.items():
try:
actions = cp.pop('actions')
except KeyError:
# Invalid category without actions
logging.warning("no actions found in category '%s'" % cn)
continue
# Get category parser
cat_parser = top_parser.add_category_parser(cn, **cp)
# -- Parse actions
for an, ap in actions.items():
args = ap.pop('arguments', {})
tid = (n, cn, an)
try:
conf = ap.pop('configuration')
_set_conf = lambda p: p.set_conf(tid, conf)
except KeyError:
# No action configuration
_set_conf = lambda p: False
try:
# Get action parser
parser = cat_parser.add_action_parser(an, tid, **ap)
except AttributeError:
# No parser for the action
continue
except ValueError as e:
logging.warning("cannot add action (%s, %s, %s): %s" % (n, cn, an, e))
continue
else:
# Store action identifier and add arguments
parser.set_defaults(_tid=tid)
_add_arguments(parser, args)
_set_conf(cat_parser)
return top_parser

148
moulinette/authenticators/__init__.py Normal file
View file

@ -0,0 +1,148 @@
# -*- coding: utf-8 -*-
import errno
import gnupg
import logging
from moulinette.core import MoulinetteError
# Base Class -----------------------------------------------------------
class BaseAuthenticator(object):
"""Authenticator base representation
Each authenticators must implement an Authenticator class derived
from this class which must overrides virtual properties and methods.
It is used to authenticate and manage session. It implements base
methods to authenticate with a password or a session token.
Authenticators configurations are identified by a profile name which
must be given on instantiation - with the corresponding vendor
configuration of the authenticator.
Keyword arguments:
- name -- The authenticator profile name
"""
def __init__(self, name):
self._name = name
@property
def name(self):
"""Return the name of the authenticator instance"""
return self._name
## Virtual properties
# Each authenticator classes must implement these properties.
"""The vendor name of the authenticator"""
vendor = None
@property
def is_authenticated(self):
"""Either the instance is authenticated or not"""
raise NotImplementedError("derived class '%s' must override this property" % \
self.__class__.__name__)
## Virtual methods
# Each authenticator classes must implement these methods.
def authenticate(password=None):
"""Attempt to authenticate
Attempt to authenticate with given password. It should raise an
AuthenticationError exception if authentication fails.
Keyword arguments:
- password -- A clear text password
"""
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)
## Authentication methods
def __call__(self, password=None, token=None):
"""Attempt to authenticate
Attempt to authenticate either with password or with session
token if 'password' is None. If the authentication succeed, the
instance is returned and the session is registered for the token
if 'token' and 'password' are given.
The token is composed by the session identifier and a session
hash - to use for encryption - as a 2-tuple.
Keyword arguments:
- password -- A clear text password
- token -- The session token in the form of (id, hash)
Returns:
The authenticated instance
"""
if self.is_authenticated:
return self
store_session = True if password and token else False
if token:
try:
# Extract id and hash from token
s_id, s_hash = token
except TypeError:
if not password:
raise MoulinetteError(errno.EINVAL, _("Invalid format for token"))
else:
# TODO: Log error
store_session = False
else:
if password is None:
# Retrieve session
password = self._retrieve_session(s_id, s_hash)
try:
# Attempt to authenticate
self.authenticate(password)
except MoulinetteError:
raise
except Exception as e:
logging.error("authentication (name: '%s', type: '%s') fails: %s" % \
(self.name, self.vendor, e))
raise MoulinetteError(errno.EACCES, _("Unable to authenticate"))
# Store session
if store_session:
self._store_session(s_id, s_hash, password)
return self
## Private methods
def _open_sessionfile(self, session_id, mode='r'):
"""Open a session file for this instance in given mode"""
return pkg.open_cachefile('%s.asc' % session_id, mode,
subdir='session/%s' % self.name)
def _store_session(self, session_id, session_hash, password):
"""Store a session and its associated password"""
gpg = gnupg.GPG()
gpg.encoding = 'utf-8'
with self._open_sessionfile(session_id, 'w') as f:
f.write(str(gpg.encrypt(password, None, symmetric=True,
passphrase=session_hash)))
def _retrieve_session(self, session_id, session_hash):
"""Retrieve a session and return its associated password"""
try:
with self._open_sessionfile(session_id, 'r') as f:
enc_pwd = f.read()
except IOError:
# TODO: Set proper error code
raise MoulinetteError(167, _("Unable to retrieve session"))
else:
gpg = gnupg.GPG()
gpg.encoding = 'utf-8'
return str(gpg.decrypt(enc_pwd, passphrase=session_hash))

192
moulinette/authenticators/ldap.py Normal file
View file

@ -0,0 +1,192 @@
# -*- coding: utf-8 -*-
# TODO: Use Python3 to remove this fix!
from __future__ import absolute_import
import ldap
import ldap.modlist as modlist
from moulinette.core import MoulinetteError
from moulinette.authenticators import BaseAuthenticator
# LDAP Class Implementation --------------------------------------------
class Authenticator(BaseAuthenticator):
"""LDAP Authenticator
Initialize a LDAP connexion for the given arguments. It attempts to
authenticate a user if 'user_rdn' is given - by associating user_rdn
and base_dn - and provides extra methods to manage opened connexion.
Keyword arguments:
- uri -- The LDAP server URI
- base_dn -- The base dn
- user_rdn -- The user rdn to authenticate
"""
def __init__(self, name, uri, base_dn, user_rdn=None):
super(Authenticator, self).__init__(name)
self.uri = uri
self.basedn = base_dn
if user_rdn:
self.userdn = '%s,%s' % (user_rdn, base_dn)
self.con = None
else:
# Initialize anonymous usage
self.userdn = ''
self.authenticate(None)
## Implement virtual properties
vendor = 'ldap'
@property
def is_authenticated(self):
try:
# Retrieve identity
who = self.con.whoami_s()
except:
return False
else:
if who[3:] == self.userdn:
return True
return False
## Implement virtual methods
def authenticate(self, password):
try:
con = ldap.initialize(self.uri)
if self.userdn:
con.simple_bind_s(self.userdn, password)
else:
con.simple_bind_s()
except ldap.INVALID_CREDENTIALS:
raise MoulinetteError(errno.EACCES, _("Invalid password"))
else:
self.con = con
## Additional LDAP methods
# TODO: Review these methods
def search(self, base=None, filter='(objectClass=*)', attrs=['dn']):
"""
Search in LDAP base
Keyword arguments:
base -- Base to search into
filter -- LDAP filter
attrs -- Array of attributes to fetch
Returns:
Boolean | Dict
"""
if not base:
base = self.basedn
try:
result = self.con.search_s(base, ldap.SCOPE_SUBTREE, filter, attrs)
except:
raise MoulinetteError(169, _('An error occured during LDAP search'))
if result:
result_list = []
for dn, entry in result:
if attrs != None:
if 'dn' in attrs:
entry['dn'] = [dn]
result_list.append(entry)
return result_list
else:
return False
def add(self, rdn, attr_dict):
"""
Add LDAP entry
Keyword arguments:
rdn -- DN without domain
attr_dict -- Dictionnary of attributes/values to add
Returns:
Boolean | MoulinetteError
"""
dn = rdn + ',' + self.basedn
ldif = modlist.addModlist(attr_dict)
try:
self.con.add_s(dn, ldif)
except:
raise MoulinetteError(169, _('An error occured during LDAP entry creation'))
else:
return True
def remove(self, rdn):
"""
Remove LDAP entry
Keyword arguments:
rdn -- DN without domain
Returns:
Boolean | MoulinetteError
"""
dn = rdn + ',' + self.basedn
try:
self.con.delete_s(dn)
except:
raise MoulinetteError(169, _('An error occured during LDAP entry deletion'))
else:
return True
def update(self, rdn, attr_dict, new_rdn=False):
"""
Modify LDAP entry
Keyword arguments:
rdn -- DN without domain
attr_dict -- Dictionnary of attributes/values to add
new_rdn -- New RDN for modification
Returns:
Boolean | MoulinetteError
"""
dn = rdn + ',' + self.basedn
actual_entry = self.search(base=dn, attrs=None)
ldif = modlist.modifyModlist(actual_entry[0], attr_dict, ignore_oldexistent=1)
try:
if new_rdn:
self.con.rename_s(dn, new_rdn)
dn = new_rdn + ',' + self.basedn
self.con.modify_ext_s(dn, ldif)
except:
raise MoulinetteError(169, _('An error occured during LDAP entry update'))
else:
return True
def validate_uniqueness(self, value_dict):
"""
Check uniqueness of values
Keyword arguments:
value_dict -- Dictionnary of attributes/values to check
Returns:
Boolean | MoulinetteError
"""
for attr, value in value_dict.items():
if not self.search(filter=attr + '=' + value):
continue
else:
raise MoulinetteError(17, _('Attribute already exists') + ' "' + attr + '=' + value + '"')
return True

290
moulinette/core.py Normal file
View file

@ -0,0 +1,290 @@
# -*- coding: utf-8 -*-
import os
import sys
import time
import errno
import gettext
import logging
from importlib import import_module
# Package manipulation -------------------------------------------------
def install_i18n(namespace=None):
"""Install internationalization
Install translation based on the package's default gettext domain or
on 'namespace' if provided.
Keyword arguments:
- namespace -- The namespace to initialize i18n for
"""
if namespace:
try:
t = gettext.translation(namespace, pkg.localedir)
except IOError:
# TODO: Log error
return
else:
t.install()
else:
gettext.install('moulinette', pkg.localedir)
class Package(object):
"""Package representation and easy access methods
Initialize directories and variables for the package and give them
easy access.
Keyword arguments:
- _from_source -- Either the package is running from source or
not (only for debugging)
"""
def __init__(self, _from_source=False):
if _from_source:
import sys
basedir = os.path.abspath(os.path.dirname(sys.argv[0]) +'/../')
# Set local directories
self._datadir = '%s/data' % basedir
self._libdir = '%s/lib' % basedir
self._localedir = '%s/po' % basedir
self._cachedir = '%s/cache' % basedir
else:
import package
# Set system directories
self._datadir = package.datadir
self._libdir = package.libdir
self._localedir = package.localedir
self._cachedir = package.cachedir
def __setattr__(self, name, value):
if name[0] == '_' and self.__dict__.has_key(name):
# Deny reassignation of package directories
raise TypeError("cannot reassign constant '%s'")
self.__dict__[name] = value
## Easy access to package directories
@property
def datadir(self):
"""Return the data directory of the package"""
return self._datadir
@property
def libdir(self):
"""Return the lib directory of the package"""
return self._libdir
@property
def localedir(self):
"""Return the locale directory of the package"""
return self._localedir
@property
def cachedir(self):
"""Return the cache directory of the package"""
return self._cachedir
## Additional methods
def get_cachedir(self, subdir='', make_dir=True):
"""Get the path to a cache directory
Return the path to the cache directory from an optional
subdirectory and create it if needed.
Keyword arguments:
- subdir -- A cache subdirectory
- make_dir -- False to not make directory if it not exists
"""
path = os.path.join(self.cachedir, subdir)
if make_dir and not os.path.isdir(path):
os.makedirs(path)
return path
def open_cachefile(self, filename, mode='r', **kwargs):
"""Open a cache file and return a stream
Attempt to open in 'mode' the cache file 'filename' from the
default cache directory and in the subdirectory 'subdir' if
given. Directories are created if needed and a stream is
returned if the file can be written.
Keyword arguments:
- filename -- The cache filename
- mode -- The mode in which the file is opened
- **kwargs -- Optional arguments for get_cachedir
"""
# Set make_dir if not given
kwargs['make_dir'] = kwargs.get('make_dir',
True if mode[0] == 'w' else False)
return open('%s/%s' % (self.get_cachedir(**kwargs), filename), mode)
# Interfaces & Authenticators management -------------------------------
def init_interface(name, kwargs={}, actionsmap={}):
"""Return a new interface instance
Retrieve the given interface module and return a new instance of its
Interface class. It is initialized with arguments 'kwargs' and
connected to 'actionsmap' if it's an ActionsMap object, otherwise
a new ActionsMap instance will be initialized with arguments
'actionsmap'.
Keyword arguments:
- name -- The interface name
- kwargs -- A dict of arguments to pass to Interface
- actionsmap -- Either an ActionsMap instance or a dict of
arguments to pass to ActionsMap
"""
from moulinette.actionsmap import ActionsMap
try:
mod = import_module('moulinette.interfaces.%s' % name)
except ImportError:
# TODO: List available interfaces
raise MoulinetteError(errno.EINVAL, _("Unknown interface '%s'" % name))
else:
try:
# Retrieve interface classes
parser = mod.ActionsMapParser
interface = mod.Interface
except AttributeError as e:
raise MoulinetteError(errno.EFAULT, _("Invalid interface '%s': %s") % (name, e))
# Instantiate or retrieve ActionsMap
if isinstance(actionsmap, dict):
amap = ActionsMap(actionsmap.pop('parser', parser), **actionsmap)
elif isinstance(actionsmap, ActionsMap):
amap = actionsmap
else:
raise MoulinetteError(errno.EINVAL, _("Invalid actions map '%r'" % actionsmap))
return interface(amap, **kwargs)
def init_authenticator((vendor, name), kwargs={}):
"""Return a new authenticator instance
Retrieve the given authenticator vendor and return a new instance of
its Authenticator class for the given profile.
Keyword arguments:
- vendor -- The authenticator vendor name
- name -- The authenticator profile name
- kwargs -- A dict of arguments for the authenticator profile
"""
try:
mod = import_module('moulinette.authenticators.%s' % vendor)
except ImportError:
# TODO: List available authenticators vendors
raise MoulinetteError(errno.EINVAL, _("Unknown authenticator vendor '%s'" % vendor))
else:
return mod.Authenticator(name, **kwargs)
def clean_session(session_id, profiles=[]):
"""Clean a session cache
Remove cache for the session 'session_id' and for profiles in
'profiles' or for all of them if the list is empty.
Keyword arguments:
- session_id -- The session id to clean
- profiles -- A list of profiles to clean
"""
sessiondir = pkg.get_cachedir('session')
if not profiles:
profiles = os.listdir(sessiondir)
for p in profiles:
try:
os.unlink(os.path.join(sessiondir, p, '%s.asc' % session_id))
except OSError:
pass
# Moulinette core classes ----------------------------------------------
class MoulinetteError(OSError):
"""Moulinette base exception"""
pass
class MoulinetteLock(object):
"""Locker for a moulinette instance
It provides a lock mechanism for a given moulinette instance. It can
be used in a with statement as it has a context-manager support.
Keyword arguments:
- namespace -- The namespace to lock
- timeout -- The time period before failing if the lock cannot
be acquired
- interval -- The time period before trying again to acquire the
lock
"""
def __init__(self, namespace, timeout=0, interval=.5):
self.namespace = namespace
self.timeout = timeout
self.interval = interval
self._lockfile = '/var/run/moulinette_%s.lock' % namespace
self._locked = False
def acquire(self):
"""Attempt to acquire the lock for the moulinette instance
It will try to write to the lock file only if it doesn't exist.
Otherwise, it will wait and try again until the timeout expires
or the lock file doesn't exist.
"""
start_time = time.time()
while True:
if not os.path.isfile(self._lockfile):
# Create the lock file
(open(self._lockfile, 'w')).close()
break
if (time.time() - start_time) > self.timeout:
raise MoulinetteError(errno.EBUSY, _("An instance is already running for '%s'") \
% self.namespace)
# Wait before checking again
time.sleep(self.interval)
self._locked = True
def release(self):
"""Release the lock of the moulinette instance
It will delete the lock file if the lock has been acquired.
"""
if self._locked:
os.unlink(self._lockfile)
self._locked = False
def __enter__(self):
if not self._locked:
self.acquire()
return self
def __exit__(self, *args):
self.release()
def __del__(self):
self.release()

193
yunohost.py → moulinette/helpers.py
View file

@ -1,67 +1,15 @@
# -*- coding: utf-8 -*-
""" License
Copyright (C) 2013 YunoHost
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program; if not, see http://www.gnu.org/licenses
"""
"""
YunoHost core classes & functions
"""
__credits__ = """
Copyright (C) 2012 YunoHost
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program; if not, see http://www.gnu.org/licenses
"""
__author__ = 'Kload <kload@kload.fr>'
__version__ = '695'
import os
import sys
try:
import ldap
except ImportError:
sys.stderr.write('Error: Yunohost CLI Require LDAP lib\n')
sys.stderr.write('apt-get install python-ldap\n')
sys.exit(1)
import ldap
import ldap.modlist as modlist
import yaml
import json
import re
import getpass
import random
import string
import argparse
import gettext
import getpass
if not __debug__:
import traceback
win = []
@ -132,33 +80,6 @@ def win_msg(astr):
win.append(astr)
def str_to_func(astr):
"""
Call a function from a string name
Keyword arguments:
astr -- Name of function to call
Returns:
Function
"""
try:
module, _, function = astr.rpartition('.')
if module:
__import__(module)
mod = sys.modules[module]
else:
mod = sys.modules['__main__'] # default module
func = getattr(mod, function)
except (AttributeError, ImportError):
#raise YunoHostError(168, _('Function is not defined'))
return None
else:
return func
def validate(pattern, array):
"""
Validate attributes with a pattern
@ -468,115 +389,3 @@ class YunoHostLDAP(Singleton):
else:
raise YunoHostError(17, _('Attribute already exists') + ' "' + attr + '=' + value + '"')
return True
def parse_dict(action_map):
"""
Turn action dictionnary to parser, subparsers and arguments
Keyword arguments:
action_map -- Multi-level dictionnary of categories/actions/arguments list
Returns:
Namespace of args
"""
# Intialize parsers
parsers = subparsers_category = subparsers_action = {}
parsers['general'] = argparse.ArgumentParser()
subparsers = parsers['general'].add_subparsers()
new_args = []
patterns = {}
# Add general arguments
for arg_name, arg_params in action_map['general_arguments'].items():
if 'version' in arg_params:
v = arg_params['version']
arg_params['version'] = v.replace('%version%', __version__)
if 'full' in arg_params:
arg_names = [arg_name, arg_params['full']]
arg_fullname = arg_params['full']
del arg_params['full']
else: arg_names = [arg_name]
parsers['general'].add_argument(*arg_names, **arg_params)
del action_map['general_arguments']
# Split categories into subparsers
for category, category_params in action_map.items():
if 'category_help' not in category_params: category_params['category_help'] = ''
subparsers_category[category] = subparsers.add_parser(category, help=category_params['category_help'])
subparsers_action[category] = subparsers_category[category].add_subparsers()
# Split actions
if 'actions' in category_params:
for action, action_params in category_params['actions'].items():
if 'action_help' not in action_params: action_params['action_help'] = ''
parsers[category + '_' + action] = subparsers_action[category].add_parser(action, help=action_params['action_help'])
# Set the action s related function
parsers[category + '_' + action].set_defaults(
func=str_to_func('yunohost_' + category + '.'
+ category + '_' + action.replace('-', '_')))
# Add arguments
if 'arguments' in action_params:
for arg_name, arg_params in action_params['arguments'].items():
arg_fullname = False
if 'password' in arg_params:
if arg_params['password']: is_password = True
del arg_params['password']
else: is_password = False
if 'full' in arg_params:
arg_names = [arg_name, arg_params['full']]
arg_fullname = arg_params['full']
del arg_params['full']
else: arg_names = [arg_name]
if 'ask' in arg_params:
require_input = True
if '-h' in sys.argv or '--help' in sys.argv:
require_input = False
if (category != sys.argv[1]) or (action != sys.argv[2]):
require_input = False
for name in arg_names:
if name in sys.argv[2:]: require_input = False
if require_input:
if is_password:
if os.isatty(1):
pwd1 = getpass.getpass(colorize(arg_params['ask'] + ': ', 'cyan'))
pwd2 = getpass.getpass(colorize('Retype ' + arg_params['ask'][0].lower() + arg_params['ask'][1:] + ': ', 'cyan'))
if pwd1 != pwd2:
raise YunoHostError(22, _("Passwords don't match"))
sys.exit(1)
else:
raise YunoHostError(22, _("Missing arguments") + ': ' + arg_name)
if arg_name[0] == '-': arg_extend = [arg_name, pwd1]
else: arg_extend = [pwd1]
else:
if os.isatty(1):
arg_value = raw_input(colorize(arg_params['ask'] + ': ', 'cyan'))
else:
raise YunoHostError(22, _("Missing arguments") + ': ' + arg_name)
if arg_name[0] == '-': arg_extend = [arg_name, arg_value]
else: arg_extend = [arg_value]
new_args.extend(arg_extend)
del arg_params['ask']
if 'pattern' in arg_params:
if (category == sys.argv[1]) and (action == sys.argv[2]):
if 'dest' in arg_params: name = arg_params['dest']
elif arg_fullname: name = arg_fullname[2:]
else: name = arg_name
name = name.replace('-', '_')
patterns[name] = arg_params['pattern']
del arg_params['pattern']
parsers[category + '_' + action].add_argument(*arg_names, **arg_params)
args = parsers['general'].parse_args(sys.argv.extend(new_args))
args_dict = vars(args)
for key, value in patterns.items():
validate(value, args_dict[key])
return args

307
moulinette/interfaces/__init__.py Normal file
View file

@ -0,0 +1,307 @@
# -*- coding: utf-8 -*-
import errno
import logging
from moulinette.core import (init_authenticator, MoulinetteError)
# Base Class -----------------------------------------------------------
class BaseActionsMapParser(object):
"""Actions map's base Parser
Each interfaces must implement an ActionsMapParser class derived
from this class which must overrides virtual properties and methods.
It is used to parse the main parts of the actions map (i.e. global
arguments, categories and actions). It implements methods to set/get
the global and actions configuration.
Keyword arguments:
- shandler -- A actionsmap.ActionsMapSignals instance
- parent -- A parent BaseActionsMapParser derived object
"""
def __init__(self, shandler, parent=None):
if parent:
self.shandler = parent.shandler
self._o = parent
else:
self.shandler = shandler
self._o = self
self._global_conf = {}
self._conf = {}
## Virtual properties
# Each parser classes must implement these properties.
"""The name of the interface for which it is the parser"""
interface = None
## Virtual methods
# Each parser classes must implement these methods.
@staticmethod
def format_arg_names(name, full):
"""Format argument name
Format agument name depending on its 'full' parameter and return
a list of strings which will be used as name or option strings
for the argument parser.
Keyword arguments:
- name -- The argument name
- full -- The argument's 'full' parameter
Returns:
A list of option strings
"""
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)
def add_global_parser(self, **kwargs):
"""Add a parser for global arguments
Create and return an argument parser for global arguments.
Returns:
An ArgumentParser based object
"""
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)
def add_category_parser(self, name, **kwargs):
"""Add a parser for a category
Create a new category and return a parser for it.
Keyword arguments:
- name -- The category name
Returns:
A BaseParser based object
"""
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)
def add_action_parser(self, name, tid, **kwargs):
"""Add a parser for an action
Create a new action and return an argument parser for it.
Keyword arguments:
- name -- The action name
- tid -- The tuple identifier of the action
Returns:
An ArgumentParser based object
"""
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)
def parse_args(self, args, **kwargs):
"""Parse arguments
Convert argument variables to objects and assign them as
attributes of the namespace.
Keyword arguments:
- args -- Arguments string or dict (TODO)
Returns:
The populated namespace
"""
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)
## Configuration access
@property
def global_conf(self):
"""Return the global configuration of the parser"""
return self._o._global_conf
def get_global_conf(self, name, profile='default'):
"""Get the global value of a configuration
Return the formated global value of the configuration 'name' for
the given profile. If the configuration doesn't provide profile,
the formated default value is returned.
Keyword arguments:
- name -- The configuration name
- profile -- The profile of the configuration
"""
if name == 'authenticator':
value = self.global_conf[name][profile]
else:
value = self.global_conf[name]
return self._format_conf(name, value)
def set_global_conf(self, configuration):
"""Set global configuration
Set the global configuration to use for the parser.
Keyword arguments:
- configuration -- The global configuration
"""
self._o._global_conf.update(self._validate_conf(configuration, True))
def get_conf(self, action, name):
"""Get the value of an action configuration
Return the formated value of configuration 'name' for the action
identified by 'action'. If the configuration for the action is
not set, the default one is returned.
Keyword arguments:
- action -- An action identifier
- name -- The configuration name
"""
try:
value = self._o._conf[action][name]
except KeyError:
return self.get_global_conf(name)
else:
return self._format_conf(name, value)
def set_conf(self, action, configuration):
"""Set configuration for an action
Set the configuration to use for a given action identified by
'action' which is specific to the parser.
Keyword arguments:
- action -- The action identifier
- configuration -- The configuration for the action
"""
self._o._conf[action] = self._validate_conf(configuration)
def _validate_conf(self, configuration, is_global=False):
"""Validate configuration for the parser
Return the validated configuration for the interface's actions
map parser.
Keyword arguments:
- configuration -- The configuration to pre-format
"""
# TODO: Create a class with a validator method for each configuration
conf = {}
# -- 'authenficate'
try:
ifaces = configuration['authenticate']
except KeyError:
pass
else:
if ifaces == 'all':
conf['authenticate'] = ifaces
elif ifaces == False:
conf['authenticate'] = False
elif isinstance(ifaces, list):
# Store only if authentication is needed
conf['authenticate'] = True if self.interface in ifaces else False
else:
# TODO: Log error instead and tell valid values
raise MoulinetteError(errno.EINVAL, "Invalid value '%r' for configuration 'authenticate'" % ifaces)
# -- 'authenticator'
try:
auth = configuration['authenticator']
except KeyError:
pass
else:
if not is_global and isinstance(auth, str):
try:
# Store needed authenticator profile
conf['authenticator'] = self.global_conf['authenticator'][auth]
except KeyError:
raise MoulinetteError(errno.EINVAL, "Undefined authenticator '%s' in global configuration" % auth)
elif is_global and isinstance(auth, dict):
if len(auth) == 0:
logging.warning('no authenticator defined in global configuration')
else:
auths = {}
for auth_name, auth_conf in auth.items():
# Add authenticator profile as a 3-tuple
# (identifier, configuration, parameters) with
# - identifier: the authenticator vendor and its
# profile name as a 2-tuple
# - configuration: a dict of additional global
# configuration (i.e. 'help')
# - parameters: a dict of arguments for the
# authenticator profile
auths[auth_name] = ((auth_conf.get('vendor'), auth_name),
{ 'help': auth_conf.get('help', None) },
auth_conf.get('parameters', {}))
conf['authenticator'] = auths
else:
# TODO: Log error instead and tell valid values
raise MoulinetteError(errno.EINVAL, "Invalid value '%r' for configuration 'authenticator'" % auth)
# -- 'argument_auth'
try:
arg_auth = configuration['argument_auth']
except KeyError:
pass
else:
if isinstance(arg_auth, bool):
conf['argument_auth'] = arg_auth
else:
# TODO: Log error instead and tell valid values
raise MoulinetteError(errno.EINVAL, "Invalid value '%r' for configuration 'argument_auth'" % arg_auth)
return conf
def _format_conf(self, name, value):
"""Format a configuration value
Return the formated value of the configuration 'name' from its
given value.
Keyword arguments:
- name -- The name of the configuration
- value -- The value to format
"""
if name == 'authenticator' and value:
(identifier, configuration, parameters) = value
# Return global configuration and an authenticator
# instanciator as a 2-tuple
return (configuration,
lambda: init_authenticator(identifier, parameters))
return value
class BaseInterface(object):
"""Moulinette's base Interface
Each interfaces must implement an Interface class derived from this
class which must overrides virtual properties and methods.
It is used to provide a user interface for an actions map.
Keyword arguments:
- actionsmap -- The ActionsMap instance to connect to
"""
# TODO: Add common interface methods and try to standardize default ones
def __init__(self, actionsmap):
raise NotImplementedError("derived class '%s' must override this method" % \
self.__class__.__name__)

487
moulinette/interfaces/api.py Normal file
View file

@ -0,0 +1,487 @@
# -*- coding: utf-8 -*-
import os
import re
import errno
import binascii
import argparse
from json import dumps as json_encode
from bottle import run, request, response, Bottle, HTTPResponse
from moulinette.core import MoulinetteError, clean_session
from moulinette.interfaces import (BaseActionsMapParser, BaseInterface)
# API helpers ----------------------------------------------------------
def random_ascii(length=20):
"""Return a random ascii string"""
return binascii.hexlify(os.urandom(length)).decode('ascii')
class _HTTPArgumentParser(object):
"""Argument parser for HTTP requests
Object for parsing HTTP requests into Python objects. It is based
on argparse.ArgumentParser class and implements some of its methods.
"""
def __init__(self):
# Initialize the ArgumentParser object
self._parser = argparse.ArgumentParser(usage='',
prefix_chars='@',
add_help=False)
self._parser.error = self._error
self._positional = [] # list(arg_name)
self._optional = {} # dict({arg_name: option_strings})
def set_defaults(self, **kwargs):
return self._parser.set_defaults(**kwargs)
def get_default(self, dest):
return self._parser.get_default(dest)
def add_argument(self, *args, **kwargs):
action = self._parser.add_argument(*args, **kwargs)
# Append newly created action
if len(action.option_strings) == 0:
self._positional.append(action.dest)
else:
self._optional[action.dest] = action.option_strings
return action
def parse_args(self, args={}, namespace=None):
arg_strings = []
## Append an argument to the current one
def append(arg_strings, value, option_string=None):
# TODO: Process list arguments
if isinstance(value, bool):
# Append the option string only
if option_string is not None:
arg_strings.append(option_string)
elif isinstance(value, str):
if option_string is not None:
arg_strings.append(option_string)
arg_strings.append(value)
else:
arg_strings.append(value)
return arg_strings
# Iterate over positional arguments
for dest in self._positional:
if dest in args:
arg_strings = append(arg_strings, args[dest])
# Iterate over optional arguments
for dest, opt in self._optional.items():
if dest in args:
arg_strings = append(arg_strings, args[dest], opt[0])
return self._parser.parse_args(arg_strings, namespace)
def _error(self, message):
# TODO: Raise a proper exception
raise MoulinetteError(1, message)
class _ActionsMapPlugin(object):
"""Actions map Bottle Plugin
Process relevant action for the request using the actions map and
manage authentication.
Keyword arguments:
- actionsmap -- An ActionsMap instance
"""
name = 'actionsmap'
api = 2
def __init__(self, actionsmap):
# Connect signals to handlers
actionsmap.connect('authenticate', self._do_authenticate)
self.actionsmap = actionsmap
# TODO: Save and load secrets?
self.secrets = {}
def setup(self, app):
"""Setup plugin on the application
Add routes according to the actions map to the application.
Keyword arguments:
- app -- The application instance
"""
## Login wrapper
def _login(callback):
def wrapper():
kwargs = {}
try:
kwargs['password'] = request.POST['password']
except KeyError:
raise HTTPBadRequestResponse(_("Missing password parameter"))
try:
kwargs['profile'] = request.POST['profile']
except KeyError:
pass
return callback(**kwargs)
return wrapper
## Logout wrapper
def _logout(callback):
def wrapper():
kwargs = {}
try:
kwargs['profile'] = request.POST.get('profile')
except KeyError:
pass
return callback(**kwargs)
return wrapper
# Append authentication routes
app.route('/login', name='login', method='POST',
callback=self.login, skip=['actionsmap'], apply=_login)
app.route('/logout', name='logout', method='GET',
callback=self.logout, skip=['actionsmap'], apply=_logout)
# Append routes from the actions map
for (m, p) in self.actionsmap.parser.routes:
app.route(p, method=m, callback=self.process)
def apply(self, callback, context):
"""Apply plugin to the route callback
Install a wrapper which replace callback and process the
relevant action for the route.
Keyword arguments:
callback -- The route callback
context -- An instance of Route
"""
def wrapper(*args, **kwargs):
# Bring arguments together
params = kwargs
for a in args:
params[a] = True
for k, v in request.params.items():
params[k] = v
# Process the action
return callback((request.method, context.rule), params)
return wrapper
## Routes callbacks
def login(self, password, profile='default'):
"""Log in to an authenticator profile
Attempt to authenticate to a given authenticator profile and
register it with the current session - a new one will be created
if needed.
Keyword arguments:
- password -- A clear text password
- profile -- The authenticator profile name to log in
"""
# Retrieve session values
s_id = request.get_cookie('session.id') or random_ascii()
try:
s_secret = self.secrets[s_id]
except KeyError:
s_hashes = {}
else:
s_hashes = request.get_cookie('session.hashes',
secret=s_secret) or {}
s_hash = random_ascii()
try:
# Attempt to authenticate
auth = self.actionsmap.get_authenticator(profile)
auth(password, token=(s_id, s_hash))
except MoulinetteError as e:
if len(s_hashes) > 0:
try: self.logout(profile)
except: pass
if e.errno == errno.EACCES:
raise HTTPUnauthorizedResponse(e.strerror)
raise HTTPErrorResponse(e.strerror)
else:
# Update dicts with new values
s_hashes[profile] = s_hash
self.secrets[s_id] = s_secret = random_ascii()
response.set_cookie('session.id', s_id, secure=True)
response.set_cookie('session.hashes', s_hashes, secure=True,
secret=s_secret)
raise HTTPOKResponse()
def logout(self, profile=None):
"""Log out from an authenticator profile
Attempt to unregister a given profile - or all by default - from
the current session.
Keyword arguments:
- profile -- The authenticator profile name to log out
"""
s_id = request.get_cookie('session.id')
try:
del self.secrets[s_id]
except KeyError:
raise HTTPUnauthorizedResponse(_("You are not logged in"))
else:
# TODO: Clean the session for profile only
# Delete cookie and clean the session
response.set_cookie('session.hashes', '', max_age=-1)
clean_session(s_id)
raise HTTPOKResponse()
def process(self, _route, arguments={}):
"""Process the relevant action for the route
Call the actions map in order to process the relevant action for
the route with the given arguments and process the returned
value.
Keyword arguments:
- _route -- The action route as a 2-tuple (method, path)
- arguments -- A dict of arguments for the route
"""
try:
ret = self.actionsmap.process(arguments, route=_route)
except MoulinetteError as e:
raise HTTPErrorResponse(e.strerror)
else:
return ret
## Signals handlers
def _do_authenticate(self, authenticator, help):
"""Process the authentication
Handle the actionsmap._AMapSignals.authenticate signal.
"""
s_id = request.get_cookie('session.id')
try:
s_secret = self.secrets[s_id]
s_hash = request.get_cookie('session.hashes',
secret=s_secret)[authenticator.name]
except KeyError:
if authenticator.name == 'default':
msg = _("Needing authentication")
else:
msg = _("Needing authentication to profile '%s'") % authenticator.name
raise HTTPUnauthorizedResponse(msg)
else:
return authenticator(token=(s_id, s_hash))
# HTTP Responses -------------------------------------------------------
class HTTPOKResponse(HTTPResponse):
def __init__(self, output=''):
super(HTTPOKResponse, self).__init__(output, 200)
class HTTPBadRequestResponse(HTTPResponse):
def __init__(self, output=''):
super(HTTPBadRequestResponse, self).__init__(output, 400)
class HTTPUnauthorizedResponse(HTTPResponse):
def __init__(self, output=''):
super(HTTPUnauthorizedResponse, self).__init__(output, 401)
class HTTPErrorResponse(HTTPResponse):
def __init__(self, output=''):
super(HTTPErrorResponse, self).__init__(output, 500)
# API Classes Implementation -------------------------------------------
class ActionsMapParser(BaseActionsMapParser):
"""Actions map's Parser for the API
Provide actions map parsing methods for a CLI usage. The parser for
the arguments is represented by a argparse.ArgumentParser object.
"""
def __init__(self, shandler, parent=None):
super(ActionsMapParser, self).__init__(shandler, parent)
self._parsers = {} # dict({(method, path): _HTTPArgumentParser})
@property
def routes(self):
"""Get current routes"""
return self._parsers.keys()
## Implement virtual properties
name = 'api'
## Implement virtual methods
@staticmethod
def format_arg_names(name, full):
if name[0] != '-':
return [name]
if full:
return [full.replace('--', '@', 1)]
if name.startswith('--'):
return [name.replace('--', '@', 1)]
return [name.replace('-', '@', 1)]
def add_global_parser(self, **kwargs):
raise AttributeError("global arguments are not managed")
def add_category_parser(self, name, **kwargs):
return self
def add_action_parser(self, name, tid, api=None, **kwargs):
"""Add a parser for an action
Keyword arguments:
- api -- The action route (e.g. 'GET /' )
Returns:
A new _HTTPArgumentParser object for the route
"""
try:
# Validate action route
m = re.match('(GET|POST|PUT|DELETE) (/\S+)', api)
except TypeError:
raise AttributeError("the action '%s' doesn't provide api access" % name)
if not m:
# TODO: Log error
raise ValueError("the action '%s' doesn't provide api access" % name)
# Check if a parser already exists for the route
key = (m.group(1), m.group(2))
if key in self.routes:
raise AttributeError("a parser for '%s' already exists" % key)
# Create and append parser
parser = _HTTPArgumentParser()
self._parsers[key] = (tid, parser)
# Return the created parser
return parser
def parse_args(self, args, route, **kwargs):
"""Parse arguments
Keyword arguments:
- route -- The action route as a 2-tuple (method, path)
"""
try:
# Retrieve the tid and the parser for the route
tid, parser = self._parsers[route]
except KeyError:
raise MoulinetteError(errno.EINVAL, "No parser found for route '%s'" % route)
ret = argparse.Namespace()
# Perform authentication if needed
if self.get_conf(tid, 'authenticate'):
auth_conf, klass = self.get_conf(tid, 'authenticator')
# TODO: Catch errors
auth = self.shandler.authenticate(klass(), **auth_conf)
if not auth.is_authenticated:
# TODO: Set proper error code
raise MoulinetteError(errno.EACCES, _("This action need authentication"))
if self.get_conf(tid, 'argument_auth') and \
self.get_conf(tid, 'authenticate') == 'all':
ret.auth = auth
return parser.parse_args(args, ret)
class Interface(BaseInterface):
"""Application Programming Interface for the moulinette
Initialize a HTTP server which serves the API connected to a given
actions map.
Keyword arguments:
- actionsmap -- The ActionsMap instance to connect to
- routes -- A dict of additional routes to add in the form of
{(method, path): callback}
"""
def __init__(self, actionsmap, routes={}):
# TODO: Return OK to 'OPTIONS' xhr requests (l173)
app = Bottle(autojson=False)
## Wrapper which sets proper header
def apiheader(callback):
def wrapper(*args, **kwargs):
response.content_type = 'application/json'
response.set_header('Access-Control-Allow-Origin', '*')
return json_encode(callback(*args, **kwargs))
return wrapper
# Install plugins
app.install(apiheader)
app.install(_ActionsMapPlugin(actionsmap))
# Append default routes
app.route(['/api', '/api/<category:re:[a-z]+>'], method='GET',
callback=self.doc, skip=['actionsmap'])
# Append additional routes
# TODO: Add optional authentication to those routes?
for (m, p), c in routes.items():
app.route(p, method=m, callback=c, skip=['actionsmap'])
self._app = app
def run(self, _port):
"""Run the moulinette
Start a server instance on the given port to serve moulinette
actions.
Keyword arguments:
- _port -- Port number to run on
"""
try:
run(self._app, port=_port)
except IOError as e:
if e.args[0] == errno.EADDRINUSE:
raise MoulinetteError(errno.EADDRINUSE, _("A server is already running"))
raise
## Routes handlers
def doc(self, category=None):
"""
Get API documentation for a category (all by default)
Keyword argument:
category -- Name of the category
"""
if category is None:
with open('%s/../doc/resources.json' % pkg.datadir) as f:
return f.read()
try:
with open('%s/../doc/%s.json' % (pkg.datadir, category)) as f:
return f.read()
except IOError:
return None

209
moulinette/interfaces/cli.py Normal file
View file

@ -0,0 +1,209 @@
# -*- coding: utf-8 -*-
import errno
import getpass
import argparse
from moulinette.core import MoulinetteError
from moulinette.interfaces import (BaseActionsMapParser, BaseInterface)
# CLI helpers ----------------------------------------------------------
colors_codes = {
'red' : 31,
'green' : 32,
'yellow': 33,
'cyan' : 34,
'purple': 35
}
def colorize(astr, color):
"""Colorize a string
Return a colorized string for printing in shell with style ;)
Keyword arguments:
- astr -- String to colorize
- color -- Name of the color
"""
return '\033[{:d}m\033[1m{:s}\033[m'.format(colors_codes[color], astr)
def pretty_print_dict(d, depth=0):
"""Print a dictionary recursively
Print a dictionary recursively with colors to the standard output.
Keyword arguments:
- d -- The dictionary to print
- depth -- The recursive depth of the dictionary
"""
for k,v in sorted(d.items(), key=lambda x: x[0]):
k = colorize(str(k), 'purple')
if isinstance(v, list) and len(v) == 1:
v = v[0]
if isinstance(v, dict):
print((" ") * depth + ("%s: " % str(k)))
pretty_print_dict(v, depth+1)
elif isinstance(v, list):
print((" ") * depth + ("%s: " % str(k)))
for key, value in enumerate(v):
if isinstance(value, tuple):
pretty_print_dict({value[0]: value[1]}, depth+1)
elif isinstance(value, dict):
pretty_print_dict({key: value}, depth+1)
else:
print((" ") * (depth+1) + "- " +str(value))
else:
if not isinstance(v, basestring):
v = str(v)
print((" ") * depth + "%s: %s" % (str(k), v))
# CLI Classes Implementation -------------------------------------------
class ActionsMapParser(BaseActionsMapParser):
"""Actions map's Parser for the CLI
Provide actions map parsing methods for a CLI usage. The parser for
the arguments is represented by a argparse.ArgumentParser object.
Keyword arguments:
- parser -- The argparse.ArgumentParser object to use
"""
def __init__(self, shandler, parent=None, parser=None):
super(ActionsMapParser, self).__init__(shandler, parent)
self._parser = parser or argparse.ArgumentParser()
self._subparsers = self._parser.add_subparsers()
## Implement virtual properties
interface = 'cli'
## Implement virtual methods
@staticmethod
def format_arg_names(name, full):
if name[0] == '-' and full:
return [name, full]
return [name]
def add_global_parser(self, **kwargs):
return self._parser
def add_category_parser(self, name, category_help=None, **kwargs):
"""Add a parser for a category
Keyword arguments:
- category_help -- A brief description for the category
Returns:
A new ActionsMapParser object for the category
"""
parser = self._subparsers.add_parser(name, help=category_help)
return self.__class__(None, self, parser)
def add_action_parser(self, name, tid, action_help=None, **kwargs):
"""Add a parser for an action
Keyword arguments:
- action_help -- A brief description for the action
Returns:
A new argparse.ArgumentParser object for the action
"""
return self._subparsers.add_parser(name, help=action_help)
def parse_args(self, args, **kwargs):
ret = self._parser.parse_args(args)
# Perform authentication if needed
if self.get_conf(ret._tid, 'authenticate'):
auth_conf, klass = self.get_conf(ret._tid, 'authenticator')
# TODO: Catch errors
auth = self.shandler.authenticate(klass(), **auth_conf)
if not auth.is_authenticated:
# TODO: Set proper error code
raise MoulinetteError(errno.EACCES, _("This action need authentication"))
if self.get_conf(ret._tid, 'argument_auth') and \
self.get_conf(ret._tid, 'authenticate') == 'all':
ret.auth = auth
return ret
class Interface(BaseInterface):
"""Command-line Interface for the moulinette
Initialize an interface connected to the standard input/output
stream and to a given actions map.
Keyword arguments:
- actionsmap -- The ActionsMap instance to connect to
"""
def __init__(self, actionsmap):
# Connect signals to handlers
actionsmap.connect('authenticate', self._do_authenticate)
actionsmap.connect('prompt', self._do_prompt)
self.actionsmap = actionsmap
def run(self, args):
"""Run the moulinette
Process the action corresponding to the given arguments 'args'
and print the result.
Keyword arguments:
- args -- A list of argument strings
"""
try:
ret = self.actionsmap.process(args, timeout=5)
except KeyboardInterrupt, EOFError:
raise MoulinetteError(errno.EINTR, _("Interrupted"))
if isinstance(ret, dict):
pretty_print_dict(ret)
elif ret:
print(ret)
## Signals handlers
def _do_authenticate(self, authenticator, help):
"""Process the authentication
Handle the actionsmap._AMapSignals.authenticate signal.
"""
# TODO: Allow token authentication?
msg = help or _("Password")
return authenticator(password=self._do_prompt(msg, True, False))
def _do_prompt(self, message, is_password, confirm):
"""Prompt for a value
Handle the actionsmap._AMapSignals.prompt signal.
"""
if is_password:
prompt = lambda m: getpass.getpass(colorize(_('%s: ') % m, 'cyan'))
else:
prompt = lambda m: raw_input(colorize(_('%s: ') % m, 'cyan'))
value = prompt(message)
if confirm:
if prompt(_('Retype %s: ') % message) != value:
raise MoulinetteError(errno.EINVAL, _("Values don't match"))
return value

15
moulinette/package.py Normal file
View file

@ -0,0 +1,15 @@
# -*- coding: utf-8 -*-
# Public constants defined during build
"""Package's data directory (e.g. /usr/share/moulinette)"""
datadir = '/usr/share/moulinette'
"""Package's library directory (e.g. /usr/lib/moulinette)"""
libdir = '/usr/lib/moulinette'
"""Locale directory for the package (e.g. /usr/lib/moulinette/locale)"""
localedir = '/usr/lib/moulinette/locale'
"""Cache directory for the package (e.g. /var/cache/moulinette)"""
cachedir = '/var/cache/moulinette'

15
moulinette/package.py.in Normal file
View file

@ -0,0 +1,15 @@
# -*- coding: utf-8 -*-
# Public constants defined during build
"""Package's data directory (e.g. /usr/share/moulinette)"""
datadir = %PKGDATADIR%
"""Package's library directory (e.g. /usr/lib/moulinette)"""
libdir = %PKGLIBDIR%
"""Locale directory for the package (e.g. /usr/lib/moulinette/locale)"""
localedir = %PKGLOCALEDIR%
"""Cache directory for the package (e.g. /var/cache/moulinette)"""
cachedir = %PKGCACHEDIR%

3
txrestapi/.gitignore vendored
View file

@ -1,3 +0,0 @@
_trial_temp
txrestapi.egg-info
txrestapi/_trial_temp

146
txrestapi/README.rst
View file

@ -1,146 +0,0 @@
============
Introduction
============
``txrestapi`` makes it easier to create Twisted REST API services. Normally, one
would create ``Resource`` subclasses defining each segment of a path; this is
cubersome to implement and results in output that isn't very readable.
``txrestapi`` provides an ``APIResource`` class allowing complex mapping of path to
callback (a la Django) with a readable decorator.
===============================
Basic URL callback registration
===============================
First, let's create a bare API service::
>>> from txrestapi.resource import APIResource
>>> api = APIResource()
and a web server to serve it::
>>> from twisted.web.server import Site
>>> from twisted.internet import reactor
>>> site = Site(api, timeout=None)
and a function to make it easy for us to make requests (only for doctest
purposes; normally you would of course use ``reactor.listenTCP(8080, site)``)::
>>> from twisted.web.server import Request
>>> class FakeChannel(object):
... transport = None
>>> def makeRequest(method, path):
... req = Request(FakeChannel(), None)
... req.prepath = req.postpath = None
... req.method = method; req.path = path
... resource = site.getChildWithDefault(path, req)
... return resource.render(req)
We can now register callbacks for paths we care about. We can provide different
callbacks for different methods; they must accept ``request`` as the first
argument::
>>> def get_callback(request): return 'GET callback'
>>> api.register('GET', '^/path/to/method', get_callback)
>>> def post_callback(request): return 'POST callback'
>>> api.register('POST', '^/path/to/method', post_callback)
Then, when we make a call, the request is routed to the proper callback::
>>> print makeRequest('GET', '/path/to/method')
GET callback
>>> print makeRequest('POST', '/path/to/method')
POST callback
We can register multiple callbacks for different requests; the first one that
matches wins::
>>> def default_callback(request):
... return 'Default callback'
>>> api.register('GET', '^/.*$', default_callback) # Matches everything
>>> print makeRequest('GET', '/path/to/method')
GET callback
>>> print makeRequest('GET', '/path/to/different/method')
Default callback
Our default callback, however, will only match GET requests. For a true default
callback, we can either register callbacks for each method individually, or we
can use ALL::
>>> api.register('ALL', '^/.*$', default_callback)
>>> print makeRequest('PUT', '/path/to/method')
Default callback
>>> print makeRequest('DELETE', '/path/to/method')
Default callback
>>> print makeRequest('GET', '/path/to/method')
GET callback
Let's unregister all references to the default callback so it doesn't interfere
with later tests (default callbacks should, of course, always be registered
last, so they don't get called before other callbacks)::
>>> api.unregister(callback=default_callback)
=============
URL Arguments
=============
Since callbacks accept ``request``, they have access to POST data or query
arguments, but we can also pull arguments out of the URL by using named groups
in the regular expression (similar to Django). These will be passed into the
callback as keyword arguments::
>>> def get_info(request, id):
... return 'Information for id %s' % id
>>> api.register('GET', '/(?P<id>[^/]+)/info$', get_info)
>>> print makeRequest('GET', '/someid/info')
Information for id someid
Bear in mind all arguments will come in as strings, so code should be
accordingly defensive.
================
Decorator syntax
================
Registration via the ``register()`` method is somewhat awkward, so decorators
are provided making it much more straightforward. ::
>>> from txrestapi.methods import GET, POST, PUT, ALL
>>> class MyResource(APIResource):
...
... @GET('^/(?P<id>[^/]+)/info')
... def get_info(self, request, id):
... return 'Info for id %s' % id
...
... @PUT('^/(?P<id>[^/]+)/update')
... @POST('^/(?P<id>[^/]+)/update')
... def set_info(self, request, id):
... return "Setting info for id %s" % id
...
... @ALL('^/')
... def default_view(self, request):
... return "I match any URL"
Again, registrations occur top to bottom, so methods should be written from
most specific to least. Also notice that one can use the decorator syntax as
one would expect to register a method as the target for two URLs ::
>>> site = Site(MyResource(), timeout=None)
>>> print makeRequest('GET', '/anid/info')
Info for id anid
>>> print makeRequest('PUT', '/anid/update')
Setting info for id anid
>>> print makeRequest('POST', '/anid/update')
Setting info for id anid
>>> print makeRequest('DELETE', '/anid/delete')
I match any URL
======================
Callback return values
======================
You can return Resource objects from a callback if you wish, allowing you to
have APIs that send you to other kinds of resources, or even other APIs.
Normally, however, you'll most likely want to return strings, which will be
wrapped in a Resource object for convenience.

1
txrestapi/__init__.py
View file

@ -1 +0,0 @@
#

29
txrestapi/methods.py
View file

@ -1,29 +0,0 @@
from zope.interface.advice import addClassAdvisor
def method_factory_factory(method):
def factory(regex):
_f = {}
def decorator(f):
_f[f.__name__] = f
return f
def advisor(cls):
def wrapped(f):
def __init__(self, *args, **kwargs):
f(self, *args, **kwargs)
for func_name in _f:
orig = _f[func_name]
func = getattr(self, func_name)
if func.im_func==orig:
self.register(method, regex, func)
return __init__
cls.__init__ = wrapped(cls.__init__)
return cls
addClassAdvisor(advisor)
return decorator
return factory
ALL = method_factory_factory('ALL')
GET = method_factory_factory('GET')
POST = method_factory_factory('POST')
PUT = method_factory_factory('PUT')
DELETE = method_factory_factory('DELETE')

65
txrestapi/resource.py
View file

@ -1,65 +0,0 @@
import re
from itertools import ifilter
from functools import wraps
from twisted.web.resource import Resource, NoResource
class _FakeResource(Resource):
_result = ''
isLeaf = True
def __init__(self, result):
Resource.__init__(self)
self._result = result
def render(self, request):
return self._result
def maybeResource(f):
@wraps(f)
def inner(*args, **kwargs):
result = f(*args, **kwargs)
if not isinstance(result, Resource):
result = _FakeResource(result)
return result
return inner
class APIResource(Resource):
_registry = None
def __init__(self, *args, **kwargs):
Resource.__init__(self, *args, **kwargs)
self._registry = []
def _get_callback(self, request):
filterf = lambda t:t[0] in (request.method, 'ALL')
path_to_check = getattr(request, '_remaining_path', request.path)
for m, r, cb in ifilter(filterf, self._registry):
result = r.search(path_to_check)
if result:
request._remaining_path = path_to_check[result.span()[1]:]
return cb, result.groupdict()
return None, None
def register(self, method, regex, callback):
self._registry.append((method, re.compile(regex), callback))
def unregister(self, method=None, regex=None, callback=None):
if regex is not None: regex = re.compile(regex)
for m, r, cb in self._registry[:]:
if not method or (method and m==method):
if not regex or (regex and r==regex):
if not callback or (callback and cb==callback):
self._registry.remove((m, r, cb))
def getChild(self, name, request):
r = self.children.get(name, None)
if r is None:
# Go into the thing
callback, args = self._get_callback(request)
if callback is None:
return NoResource()
else:
return maybeResource(callback)(request, **args)
else:
return r

7
txrestapi/service.py
View file

@ -1,7 +0,0 @@
from twisted.web.server import Site
from .resource import APIResource
class RESTfulService(Site):
def __init__(self, port=8080):
self.root = APIResource()

194
txrestapi/tests.py
View file

@ -1,194 +0,0 @@
import txrestapi
__package__="txrestapi"
import re
import os.path
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.web.resource import Resource, NoResource
from twisted.web.server import Request, Site
from twisted.web.client import getPage
from twisted.trial import unittest
from .resource import APIResource
from .methods import GET, PUT
class FakeChannel(object):
transport = None
def getRequest(method, url):
req = Request(FakeChannel(), None)
req.method = method
req.path = url
return req
class APIResourceTest(unittest.TestCase):
def test_returns_normal_resources(self):
r = APIResource()
a = Resource()
r.putChild('a', a)
req = Request(FakeChannel(), None)
a_ = r.getChild('a', req)
self.assertEqual(a, a_)
def test_registry(self):
compiled = re.compile('regex')
r = APIResource()
r.register('GET', 'regex', None)
self.assertEqual([x[0] for x in r._registry], ['GET'])
self.assertEqual(r._registry[0], ('GET', compiled, None))
def test_method_matching(self):
r = APIResource()
r.register('GET', 'regex', 1)
r.register('PUT', 'regex', 2)
r.register('GET', 'another', 3)
req = getRequest('GET', 'regex')
result = r._get_callback(req)
self.assert_(result)
self.assertEqual(result[0], 1)
req = getRequest('PUT', 'regex')
result = r._get_callback(req)
self.assert_(result)
self.assertEqual(result[0], 2)
req = getRequest('GET', 'another')
result = r._get_callback(req)
self.assert_(result)
self.assertEqual(result[0], 3)
req = getRequest('PUT', 'another')
result = r._get_callback(req)
self.assertEqual(result, (None, None))
def test_callback(self):
marker = object()
def cb(request):
return marker
r = APIResource()
r.register('GET', 'regex', cb)
req = getRequest('GET', 'regex')
result = r.getChild('regex', req)
self.assertEqual(result.render(req), marker)
def test_longerpath(self):
marker = object()
r = APIResource()
def cb(request):
return marker
r.register('GET', '/regex/a/b/c', cb)
req = getRequest('GET', '/regex/a/b/c')
result = r.getChild('regex', req)
self.assertEqual(result.render(req), marker)
def test_args(self):
r = APIResource()
def cb(request, **kwargs):
return kwargs
r.register('GET', '/(?P<a>[^/]*)/a/(?P<b>[^/]*)/c', cb)
req = getRequest('GET', '/regex/a/b/c')
result = r.getChild('regex', req)
self.assertEqual(sorted(result.render(req).keys()), ['a', 'b'])
def test_order(self):
r = APIResource()
def cb1(request, **kwargs):
kwargs.update({'cb1':True})
return kwargs
def cb(request, **kwargs):
return kwargs
# Register two regexes that will match
r.register('GET', '/(?P<a>[^/]*)/a/(?P<b>[^/]*)/c', cb1)
r.register('GET', '/(?P<a>[^/]*)/a/(?P<b>[^/]*)', cb)
req = getRequest('GET', '/regex/a/b/c')
result = r.getChild('regex', req)
# Make sure the first one got it
self.assert_('cb1' in result.render(req))
def test_no_resource(self):
r = APIResource()
r.register('GET', '^/(?P<a>[^/]*)/a/(?P<b>[^/]*)$', None)
req = getRequest('GET', '/definitely/not/a/match')
result = r.getChild('regex', req)
self.assert_(isinstance(result, NoResource))
def test_all(self):
r = APIResource()
def get_cb(r): return 'GET'
def put_cb(r): return 'PUT'
def all_cb(r): return 'ALL'
r.register('GET', '^path', get_cb)
r.register('ALL', '^path', all_cb)
r.register('PUT', '^path', put_cb)
# Test that the ALL registration picks it up before the PUT one
for method in ('GET', 'PUT', 'ALL'):
req = getRequest(method, 'path')
result = r.getChild('path', req)
self.assertEqual(result.render(req), 'ALL' if method=='PUT' else method)
class TestResource(Resource):
isLeaf = True
def render(self, request):
return 'aresource'
class TestAPI(APIResource):
@GET('^/(?P<a>test[^/]*)/?')
def _on_test_get(self, request, a):
return 'GET %s' % a
@PUT('^/(?P<a>test[^/]*)/?')
def _on_test_put(self, request, a):
return 'PUT %s' % a
@GET('^/gettest')
def _on_gettest(self, request):
return TestResource()
class DecoratorsTest(unittest.TestCase):
def _listen(self, site):
return reactor.listenTCP(0, site, interface="127.0.0.1")
def setUp(self):
r = TestAPI()
site = Site(r, timeout=None)
self.port = self._listen(site)
self.portno = self.port.getHost().port
def tearDown(self):
return self.port.stopListening()
def getURL(self, path):
return "http://127.0.0.1:%d/%s" % (self.portno, path)
@inlineCallbacks
def test_get(self):
url = self.getURL('test_thing/')
result = yield getPage(url, method='GET')
self.assertEqual(result, 'GET test_thing')
@inlineCallbacks
def test_put(self):
url = self.getURL('test_thing/')
result = yield getPage(url, method='PUT')
self.assertEqual(result, 'PUT test_thing')
@inlineCallbacks
def test_resource_wrapper(self):
url = self.getURL('gettest')
result = yield getPage(url, method='GET')
self.assertEqual(result, 'aresource')
def test_suite():
import unittest as ut
suite = unittest.TestSuite()
suite.addTest(ut.makeSuite(DecoratorsTest))
suite.addTest(ut.makeSuite(APIResourceTest))
suite.addTest(unittest.doctest.DocFileSuite(os.path.join('..', 'README.rst')))
return suite

134
yunohost
View file

@ -1,134 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import sys
import argparse
import gettext
import getpass
try:
import yaml
except ImportError:
sys.stderr.write('Error: Yunohost CLI Require yaml lib\n')
sys.stderr.write('apt-get install python-yaml\n')
sys.exit(1)
import json
if not __debug__:
import traceback
gettext.install('YunoHost')
try:
from yunohost import YunoHostError, YunoHostLDAP, str_to_func, colorize, pretty_print_dict, display_error, validate, win, parse_dict
except ImportError:
sys.stderr.write('Error: Yunohost CLI Require YunoHost lib\n')
sys.exit(1)
def main():
"""
Main instructions
Parse the action_dict and execute the action-specific function,
then print json or pretty result if executed in a tty :)
Returns:
int -- 0 or error code
"""
if len(sys.argv) < 2:
sys.argv.append('-h')
with open('action_map.yml') as f:
action_map = yaml.load(f)
admin_password_provided = False
json_print = False
write_ldap = True
postinstall = False
for key, arg in enumerate(sys.argv):
if arg == '--admin-password':
admin_password_provided = True
admin_password = sys.argv[key+1]
sys.argv.pop(key)
sys.argv.pop(key)
if arg == '--no-ldap':
write_ldap = False
sys.argv.pop(key)
if arg == '--json':
json_print = True
sys.argv.pop(key)
try:
try:
with open('/etc/yunohost/installed') as f: pass
except IOError:
postinstall = True
if len(sys.argv) < 3 or sys.argv[1] != 'tools' or sys.argv[2] != 'postinstall':
raise YunoHostError(17, _("YunoHost is not correctly installed, please execute 'yunohost tools postinstall'"))
args = parse_dict(action_map)
args_dict = vars(args).copy()
for key in args_dict.keys():
sanitized_key = key.replace('-', '_')
if sanitized_key is not key:
args_dict[sanitized_key] = args_dict[key]
del args_dict[key]
del args_dict['func']
try:
with open('/etc/yunohost/passwd') as f:
admin_password = f.read()
admin_password_provided = True
except IOError: pass
if postinstall:
result = args.func(**args_dict)
elif admin_password_provided:
with YunoHostLDAP(password=admin_password):
result = args.func(**args_dict)
elif os.isatty(1) and write_ldap:
admin_password = getpass.getpass(colorize(_('Admin Password: '), 'yellow'))
with YunoHostLDAP(password=admin_password):
try:
with open('/var/run/yunohost.pid', 'r'):
raise YunoHostError(1, _("A YunoHost command is already running"))
except IOError:
with open('/var/run/yunohost.pid', 'w') as f:
f.write('ldap')
os.system('chmod 400 /var/run/yunohost.pid')
with open('/etc/yunohost/passwd', 'w') as f:
f.write(admin_password)
os.system('chmod 400 /etc/yunohost/passwd')
try:
result = args.func(**args_dict)
except KeyboardInterrupt, EOFError:
raise YunoHostError(125, _("Interrupted"))
finally:
os.remove('/etc/yunohost/passwd')
os.remove('/var/run/yunohost.pid')
else:
with YunoHostLDAP(anonymous=True):
result = args.func(**args_dict)
#except TypeError, error:
#if not __debug__ :
#traceback.print_exc()
#print(_("Not (yet) implemented function"))
#return 1
except YunoHostError, error:
display_error(error, json_print)
return error.code
else:
if json_print or not os.isatty(1) and result is not None:
if len(win) > 0:
result['success'] = win
print(json.dumps(result))
elif result is not None:
pretty_print_dict(result)
else:
pass
return 0
if __name__ == '__main__':
sys.exit(main())

273
yunohost.tac
View file

@ -1,273 +0,0 @@
# -*- mode: python -*-
import os
import sys
import gettext
import ldap
import yaml
import json
sys.path.append('/usr/share/pyshared')
from twisted.python.log import ILogObserver, FileLogObserver, startLogging, msg
from twisted.python.logfile import DailyLogFile
from twisted.web.server import Site, http
from twisted.internet import reactor
from twisted.application import internet,service
from txrestapi.resource import APIResource
from yunohost import YunoHostError, YunoHostLDAP, str_to_func, colorize, pretty_print_dict, display_error, validate, win, parse_dict
import yunohost
if not __debug__:
import traceback
gettext.install('YunoHost')
dev = False
installed = True
action_dict = {}
api = APIResource()
def http_exec(request, **kwargs):
global installed
request.setHeader('Access-Control-Allow-Origin', '*') # Allow cross-domain requests
request.setHeader('Content-Type', 'application/json') # Return JSON anyway
# Return OK to 'OPTIONS' xhr requests
if request.method == 'OPTIONS':
request.setResponseCode(200, 'OK')
request.setHeader('Access-Control-Allow-Headers', 'Authorization, Content-Type')
request.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS')
return ''
# Simple HTTP auth
elif installed:
authorized = False
pwd = request.getPassword()
if request.getUser() == 'admin' and pwd != '':
authorized = True
if dev and 'api_key' in request.args:
pwd = request.args['api_key'][0]
authorized = True
if authorized:
try: YunoHostLDAP(password=pwd)
except YunoHostError: authorized = False
if not authorized:
request.setResponseCode(401, 'Unauthorized')
request.setHeader('Access-Control-Allow-Origin', '*')
request.setHeader('www-authenticate', 'Basic realm="Restricted Area"')
return 'Unauthorized'
path = request.path
if request.method == 'PUT':
given_args = http.parse_qs(request.content.read(), 1)
else:
given_args = request.args
if kwargs:
for k, v in kwargs.iteritems():
dynamic_key = path.split('/')[-1]
path = path.replace(dynamic_key, '{'+ k +'}')
given_args[k] = [v]
#msg(given_args)
# Sanitize arguments
dict = action_dict[request.method +' '+ path]
if 'arguments' in dict: possible_args = dict['arguments']
else: possible_args = {}
for arg, params in possible_args.items():
sanitized_key = arg.replace('-', '_')
if sanitized_key is not arg:
possible_args[sanitized_key] = possible_args[arg]
del possible_args[arg]
arg = sanitized_key
if arg[0] == '_':
if 'nargs' not in params:
possible_args[arg]['nargs'] = '*'
if 'full' in params:
new_key = params['full'][2:]
else:
new_key = arg[2:]
new_key = new_key.replace('-', '_')
possible_args[new_key] = possible_args[arg]
del possible_args[arg]
try:
# Validate arguments
validated_args = {}
for key, value in given_args.items():
if key in possible_args:
# Validate args
if 'pattern' in possible_args[key]:
validate(possible_args[key]['pattern'], value)
if 'nargs' not in possible_args[key] or ('nargs' != '*' and 'nargs' != '+'):
value = value[0]
if 'choices' in possible_args[key] and value not in possible_args[key]['choices']:
raise YunoHostError(22, _('Invalid argument') + ' ' + value)
if 'action' in possible_args[key] and possible_args[key]['action'] == 'store_true':
yes = ['true', 'True', 'yes', 'Yes']
value = value in yes
validated_args[key] = value
func = str_to_func(dict['function'])
if func is None:
raise YunoHostError(168, _('Function not yet implemented : ') + dict['function'].split('.')[1])
# Execute requested function
try:
with open('/var/run/yunohost.pid', 'r'):
raise YunoHostError(1, _("A YunoHost command is already running"))
except IOError:
if dict['function'].split('.')[1] != 'tools_postinstall':
try:
with open('/etc/yunohost/installed'): pass
except IOError:
raise YunoHostError(1, _("You must run postinstall before any other actions"))
with open('/var/run/yunohost.pid', 'w') as f:
f.write('ldap')
os.system('chmod 400 /var/run/yunohost.pid')
with open('/etc/yunohost/passwd', 'w') as f:
f.write(request.getPassword())
os.system('chmod 400 /etc/yunohost/passwd')
try:
result = func(**validated_args)
except KeyboardInterrupt, EOFError:
raise YunoHostError(125, _("Interrupted"))
finally:
try:
os.remove('/etc/yunohost/passwd')
os.remove('/var/run/yunohost.pid')
except: pass
if result is None:
result = {}
if len(yunohost.win) > 0:
result['win'] = yunohost.win
yunohost.win = []
# Build response
if request.method == 'POST':
request.setResponseCode(201, 'Created')
if not installed:
installed = True
elif request.method == 'DELETE':
request.setResponseCode(204, 'No Content')
else:
request.setResponseCode(200, 'OK')
except YunoHostError, error:
# Set response code with function's raised code
server_errors = [1, 111, 168, 169]
client_errors = [13, 17, 22, 87, 122, 125, 167]
if error.code in client_errors:
request.setResponseCode(400, 'Bad Request')
else:
request.setResponseCode(500, 'Internal Server Error')
result = { 'error' : error.message }
return json.dumps(result)
def api_doc(request):
request.setHeader('Access-Control-Allow-Origin', '*') # Allow cross-domain requests
request.setHeader('Content-Type', 'application/json') # Return JSON anyway
# Return OK to 'OPTIONS' xhr requests
if request.method == 'OPTIONS':
request.setResponseCode(200, 'OK')
request.setHeader('Access-Control-Allow-Headers', 'Authorization')
return ''
if request.path == '/api':
with open('doc/resources.json') as f:
return f.read()
category = request.path.split('/')[2]
try:
with open('doc/'+ category +'.json') as f:
return f.read()
except IOError:
return ''
def favicon(request):
request.setHeader('Access-Control-Allow-Origin', '*') # Allow cross-domain requests
request.setResponseCode(404, 'Not Found')
return ''
def is_installed(request):
global installed
try:
with open('/etc/yunohost/installed'):
installed = True
except IOError:
installed = False
request.setHeader('Access-Control-Allow-Origin', '*') # Allow cross-domain requests
request.setResponseCode(200, 'OK')
return json.dumps({ 'installed': installed })
def main():
global action_dict
global api
global installed
# Generate API doc
os.system('python ./generate_api_doc.py')
# Register API doc service
api.register('ALL', '/api', api_doc)
# favicon.ico error
api.register('ALL', '/favicon.ico', favicon)
# Load & parse yaml file
with open('action_map.yml') as f:
action_map = yaml.load(f)
# Register only postinstall action if YunoHost isn't completely set up
try:
with open('/etc/yunohost/installed'):
installed = True
except IOError:
installed = False
del action_map['general_arguments']
for category, category_params in action_map.items():
api.register('ALL', '/api/'+ category, api_doc)
for action, action_params in category_params['actions'].items():
if 'action_help' not in action_params:
action_params['action_help'] = ''
if 'api' not in action_params:
action_params['api'] = 'GET /'+ category +'/'+ action
method, path = action_params['api'].split(' ')
# Register route
if '{' in path:
path = path.replace('{', '(?P<').replace('}', '>[^/]+)')
api.register(method, path, http_exec)
api.register('OPTIONS', path, http_exec)
action_dict[action_params['api']] = {
'function': 'yunohost_'+ category +'.'+ category +'_'+ action.replace('-', '_'),
'help' : action_params['action_help']
}
if 'arguments' in action_params:
action_dict[action_params['api']]['arguments'] = action_params['arguments']
api.register('ALL', '/installed', is_installed)
if __name__ == '__main__':
if '--dev' in sys.argv:
dev = True
startLogging(sys.stdout)
else:
startLogging(open('/var/log/yunohost.log', 'a+')) # Log actions to file
main()
reactor.listenTCP(6787, Site(api, timeout=None))
reactor.run()
else:
application = service.Application("YunoHost API")
logfile = DailyLogFile("yunohost.log", "/var/log")
application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
main()
internet.TCPServer(6787, Site(api, timeout=None)).setServiceParent(application)