From 3f12eab1dd39d67d5c8e4b79b96fae96e42a85ac Mon Sep 17 00:00:00 2001 From: Jimmy Monin Date: Fri, 4 Aug 2017 22:38:02 +0200 Subject: [PATCH] Upgrade to upstream version 1.7.0 and extend weblog monitoring to every domain --- README.md | 466 ++++++++++++++++++++++----------------------- manifest.json | 2 +- scripts/_common.sh | 2 +- scripts/install | 3 + scripts/upgrade | 3 + 5 files changed, 241 insertions(+), 235 deletions(-) diff --git a/README.md b/README.md index c1fb508..76847a1 100644 --- a/README.md +++ b/README.md @@ -1,233 +1,233 @@ -NetData for YunoHost ---------------------- - - -[NetData](http://my-netdata.io/) is a system for **distributed real-time performance and health monitoring**. -It provides **unparalleled insights, in real-time**, of everything happening on the -system it runs (including applications such as web and database servers), using -**modern interactive web dashboards**. - -_netdata is **fast** and **efficient**, designed to permanently run on all systems -(**physical** & **virtual** servers, **containers**, **IoT** devices), without -disrupting their core function._ - -**Shipped version:** 1.6.0 - -**Customization brought by the package:** - -* grant MySQL statistics access via a `netdata` user -* nginx root log statistics via putting `netdata` user in the `adm` group -* Dovecot statistics via giving access to Dovecot stats stocket to `netdata` user (works only with Dovecot 2.2.16+) - -**Further recommendations:** -We don't allow YunoHost packages to make sensible changes to system files. So here are further customizations you can make to allow more monitoring: - -* Nginx: - * requests/connections: follow [these recommandations](https://github.com/firehol/netdata/tree/master/python.d#nginx) to enable `/stab_status` (for example by putting the `location` section in `/etc/nginx/conf.d/yunohost_admin.conf` - * weblogs: you can monitor all your nginx weblogs for errors; follow [these recommendations](https://github.com/firehol/netdata/tree/master/python.d#nginx_log) -* phpfpm: follow [these recommandations](https://github.com/firehol/netdata/tree/master/python.d#phpfpm) - -It has been tested on x86_64 and ARM. - -## Features - -

- -

- - - **Stunning interactive bootstrap dashboards**
- mouse and touch friendly, in 2 themes: dark, light - - - **Amazingly fast**
- responds to all queries in less than 0.5 ms per metric, - even on low-end hardware - - - **Highly efficient**
- collects thousands of metrics per server per second, - with just 1% CPU utilization of a single core, a few MB of RAM and no disk I/O at all - - - **Sophisticated alarming**
- supports dynamic thresholds, hysteresis, alarm templates, - multiple role-based notification methods (such as email, slack.com, - pushover.net, pushbullet.com telegram.org, twilio.com, messagebird.com) - - - **Extensible**
- you can monitor anything you can get a metric for, - using its Plugin API (anything can be a netdata plugin, - BASH, python, perl, node.js, java, Go, ruby, etc) - - - **Embeddable**
- it can run anywhere a Linux kernel runs (even IoT) - and its charts can be embedded on your web pages too - - - **Customizable**
- custom dashboards can be built using simple HTML (no javascript necessary) - - - **Zero configuration**
- auto-detects everything, it can collect up to 5000 metrics - per server out of the box - - - **Zero dependencies**
- it is even its own web server, for its static web files and its web API - - - **Zero maintenance**
- you just run it, it does the rest - - - **scales to infinity**
- requiring minimal central resources - - - **back-ends supported**
- can archive its metrics on `graphite` or `opentsdb`, in the same or lower detail - (lower: to prevent it from congesting these servers due to the amount of data collected) - -![netdata](https://cloud.githubusercontent.com/assets/2662304/14092712/93b039ea-f551-11e5-822c-beadbf2b2a2e.gif) - ---- - -## What does it monitor? - -netdata monitors several thousands of metrics per device. -All these metrics are collected and visualized in real-time. - -> _Almost all metrics are auto-detected, without any configuration._ - -This is a list of what it currently monitors: - -- **CPU**
- usage, interrupts, softirqs, frequency, total and per core - -- **Memory**
- RAM, swap and kernel memory usage, KSM (Kernel Samepage Merging), NUMA - -- **Disks**
- per disk: I/O, operations, backlog, utilization, space - - ![sda](https://cloud.githubusercontent.com/assets/2662304/14093195/c882bbf4-f554-11e5-8863-1788d643d2c0.gif) - -- **Network interfaces**
- per interface: bandwidth, packets, errors, drops - - ![dsl0](https://cloud.githubusercontent.com/assets/2662304/14093128/4d566494-f554-11e5-8ee4-5392e0ac51f0.gif) - -- **IPv4 networking**
- bandwidth, packets, errors, fragments, - tcp: connections, packets, errors, handshake, - udp: packets, errors, - broadcast: bandwidth, packets, - multicast: bandwidth, packets - -- **IPv6 networking**
- bandwidth, packets, errors, fragments, ECT, - udp: packets, errors, - udplite: packets, errors, - broadcast: bandwidth, - multicast: bandwidth, packets, - icmp: messages, errors, echos, router, neighbor, MLDv2, group membership, - break down by type - -- **Interprocess Communication - IPC**
- such as semaphores and semaphores arrays - -- **netfilter / iptables Linux firewall**
- connections, connection tracker events, errors - -- **Linux DDoS protection**
- SYNPROXY metrics - -- **fping** latencies
- for any number of hosts, showing latency, packets and packet loss - - ![image](https://cloud.githubusercontent.com/assets/2662304/20464811/9517d2b4-af57-11e6-8361-f6cc57541cd7.png) - - -- **Processes**
- running, blocked, forks, active - -- **Entropy**
- random numbers pool, using in cryptography - -- **NFS file servers and clients**
- NFS v2, v3, v4: I/O, cache, read ahead, RPC calls - -- **Network QoS**
- the only tool that visualizes network `tc` classes in realtime - - ![qos-tc-classes](https://cloud.githubusercontent.com/assets/2662304/14093004/68966020-f553-11e5-98fe-ffee2086fafd.gif) - -- **Linux Control Groups**
- containers: systemd, lxc, docker - -- **Applications**
- by grouping the process tree and reporting CPU, memory, disk reads, - disk writes, swap, threads, pipes, sockets - per group - - ![apps](https://cloud.githubusercontent.com/assets/2662304/14093565/67c4002c-f557-11e5-86bd-0154f5135def.gif) - -- **Users and User Groups resource usage**
- by summarizing the process tree per user and group, - reporting: CPU, memory, disk reads, disk writes, swap, threads, pipes, sockets - -- **Apache and lighttpd web servers**
- `mod-status` (v2.2, v2.4) and cache log statistics, for multiple servers - -- **Nginx web servers**
- `stub-status`, for multiple servers - -- **Tomcat**
- accesses, threads, free memory, volume - -- **mySQL databases**
- multiple servers, each showing: bandwidth, queries/s, handlers, locks, issues, - tmp operations, connections, binlog metrics, threads, innodb metrics, and more - -- **Postgres databases**
- multiple servers, each showing: per database statistics (connections, tuples - read - written - returned, transactions, locks), backend processes, indexes, - tables, write ahead, background writer and more - -- **Redis databases**
- multiple servers, each showing: operations, hit rate, memory, keys, clients, slaves - -- **memcached databases**
- multiple servers, each showing: bandwidth, connections, items - -- **ISC Bind name servers**
- multiple servers, each showing: clients, requests, queries, updates, failures and several per view metrics - -- **Postfix email servers**
- message queue (entries, size) - -- **exim email servers**
- message queue (emails queued) - -- **Dovecot** POP3/IMAP servers
- -- **IPFS**
- bandwidth, peers - -- **Squid proxy servers**
- multiple servers, each showing: clients bandwidth and requests, servers bandwidth and requests - -- **Hardware sensors**
- temperature, voltage, fans, power, humidity - -- **NUT and APC UPSes**
- load, charge, battery voltage, temperature, utility metrics, output metrics - -- **PHP-FPM**
- multiple instances, each reporting connections, requests, performance - -- **hddtemp**
- disk temperatures - -- **SNMP devices**
- can be monitored too (although you will need to configure these) - -And you can extend it, by writing plugins that collect data from any source, using any computer language. - - -## Links - - * Report a bug: https://github.com/YunoHost-Apps/netdata_ynh/issues - * NetData website: http://my-netdata.io/ - * YunoHost website: https://yunohost.org/ +NetData for YunoHost +--------------------- + + +[NetData](http://my-netdata.io/) is a system for **distributed real-time performance and health monitoring**. +It provides **unparalleled insights, in real-time**, of everything happening on the +system it runs (including applications such as web and database servers), using +**modern interactive web dashboards**. + +_netdata is **fast** and **efficient**, designed to permanently run on all systems +(**physical** & **virtual** servers, **containers**, **IoT** devices), without +disrupting their core function._ + +**Shipped version:** 1.7.0 + +**Customization brought by the package:** + +* grant MySQL statistics access via a `netdata` user +* nginx root log statistics via putting `netdata` user in the `adm` group +* Dovecot statistics via giving access to Dovecot stats stocket to `netdata` user (works only with Dovecot 2.2.16+) + +**Further recommendations:** +We don't allow YunoHost packages to make sensible changes to system files. So here are further customizations you can make to allow more monitoring: + +* Nginx: + * requests/connections: follow [these recommandations](https://github.com/firehol/netdata/tree/master/python.d#nginx) to enable `/stab_status` (for example by putting the `location` section in `/etc/nginx/conf.d/yunohost_admin.conf` + * weblogs: you can monitor all your nginx weblogs for errors; follow [these recommendations](https://github.com/firehol/netdata/tree/master/python.d#nginx_log) +* phpfpm: follow [these recommandations](https://github.com/firehol/netdata/tree/master/python.d#phpfpm) + +It has been tested on x86_64 and ARM. + +## Features + +

+ +

+ + - **Stunning interactive bootstrap dashboards**
+ mouse and touch friendly, in 2 themes: dark, light + + - **Amazingly fast**
+ responds to all queries in less than 0.5 ms per metric, + even on low-end hardware + + - **Highly efficient**
+ collects thousands of metrics per server per second, + with just 1% CPU utilization of a single core, a few MB of RAM and no disk I/O at all + + - **Sophisticated alarming**
+ supports dynamic thresholds, hysteresis, alarm templates, + multiple role-based notification methods (such as email, slack.com, + pushover.net, pushbullet.com telegram.org, twilio.com, messagebird.com) + + - **Extensible**
+ you can monitor anything you can get a metric for, + using its Plugin API (anything can be a netdata plugin, + BASH, python, perl, node.js, java, Go, ruby, etc) + + - **Embeddable**
+ it can run anywhere a Linux kernel runs (even IoT) + and its charts can be embedded on your web pages too + + - **Customizable**
+ custom dashboards can be built using simple HTML (no javascript necessary) + + - **Zero configuration**
+ auto-detects everything, it can collect up to 5000 metrics + per server out of the box + + - **Zero dependencies**
+ it is even its own web server, for its static web files and its web API + + - **Zero maintenance**
+ you just run it, it does the rest + + - **scales to infinity**
+ requiring minimal central resources + + - **back-ends supported**
+ can archive its metrics on `graphite` or `opentsdb`, in the same or lower detail + (lower: to prevent it from congesting these servers due to the amount of data collected) + +![netdata](https://cloud.githubusercontent.com/assets/2662304/14092712/93b039ea-f551-11e5-822c-beadbf2b2a2e.gif) + +--- + +## What does it monitor? + +netdata monitors several thousands of metrics per device. +All these metrics are collected and visualized in real-time. + +> _Almost all metrics are auto-detected, without any configuration._ + +This is a list of what it currently monitors: + +- **CPU**
+ usage, interrupts, softirqs, frequency, total and per core + +- **Memory**
+ RAM, swap and kernel memory usage, KSM (Kernel Samepage Merging), NUMA + +- **Disks**
+ per disk: I/O, operations, backlog, utilization, space + + ![sda](https://cloud.githubusercontent.com/assets/2662304/14093195/c882bbf4-f554-11e5-8863-1788d643d2c0.gif) + +- **Network interfaces**
+ per interface: bandwidth, packets, errors, drops + + ![dsl0](https://cloud.githubusercontent.com/assets/2662304/14093128/4d566494-f554-11e5-8ee4-5392e0ac51f0.gif) + +- **IPv4 networking**
+ bandwidth, packets, errors, fragments, + tcp: connections, packets, errors, handshake, + udp: packets, errors, + broadcast: bandwidth, packets, + multicast: bandwidth, packets + +- **IPv6 networking**
+ bandwidth, packets, errors, fragments, ECT, + udp: packets, errors, + udplite: packets, errors, + broadcast: bandwidth, + multicast: bandwidth, packets, + icmp: messages, errors, echos, router, neighbor, MLDv2, group membership, + break down by type + +- **Interprocess Communication - IPC**
+ such as semaphores and semaphores arrays + +- **netfilter / iptables Linux firewall**
+ connections, connection tracker events, errors + +- **Linux DDoS protection**
+ SYNPROXY metrics + +- **fping** latencies
+ for any number of hosts, showing latency, packets and packet loss + + ![image](https://cloud.githubusercontent.com/assets/2662304/20464811/9517d2b4-af57-11e6-8361-f6cc57541cd7.png) + + +- **Processes**
+ running, blocked, forks, active + +- **Entropy**
+ random numbers pool, using in cryptography + +- **NFS file servers and clients**
+ NFS v2, v3, v4: I/O, cache, read ahead, RPC calls + +- **Network QoS**
+ the only tool that visualizes network `tc` classes in realtime + + ![qos-tc-classes](https://cloud.githubusercontent.com/assets/2662304/14093004/68966020-f553-11e5-98fe-ffee2086fafd.gif) + +- **Linux Control Groups**
+ containers: systemd, lxc, docker + +- **Applications**
+ by grouping the process tree and reporting CPU, memory, disk reads, + disk writes, swap, threads, pipes, sockets - per group + + ![apps](https://cloud.githubusercontent.com/assets/2662304/14093565/67c4002c-f557-11e5-86bd-0154f5135def.gif) + +- **Users and User Groups resource usage**
+ by summarizing the process tree per user and group, + reporting: CPU, memory, disk reads, disk writes, swap, threads, pipes, sockets + +- **Apache and lighttpd web servers**
+ `mod-status` (v2.2, v2.4) and cache log statistics, for multiple servers + +- **Nginx web servers**
+ `stub-status`, for multiple servers + +- **Tomcat**
+ accesses, threads, free memory, volume + +- **mySQL databases**
+ multiple servers, each showing: bandwidth, queries/s, handlers, locks, issues, + tmp operations, connections, binlog metrics, threads, innodb metrics, and more + +- **Postgres databases**
+ multiple servers, each showing: per database statistics (connections, tuples + read - written - returned, transactions, locks), backend processes, indexes, + tables, write ahead, background writer and more + +- **Redis databases**
+ multiple servers, each showing: operations, hit rate, memory, keys, clients, slaves + +- **memcached databases**
+ multiple servers, each showing: bandwidth, connections, items + +- **ISC Bind name servers**
+ multiple servers, each showing: clients, requests, queries, updates, failures and several per view metrics + +- **Postfix email servers**
+ message queue (entries, size) + +- **exim email servers**
+ message queue (emails queued) + +- **Dovecot** POP3/IMAP servers
+ +- **IPFS**
+ bandwidth, peers + +- **Squid proxy servers**
+ multiple servers, each showing: clients bandwidth and requests, servers bandwidth and requests + +- **Hardware sensors**
+ temperature, voltage, fans, power, humidity + +- **NUT and APC UPSes**
+ load, charge, battery voltage, temperature, utility metrics, output metrics + +- **PHP-FPM**
+ multiple instances, each reporting connections, requests, performance + +- **hddtemp**
+ disk temperatures + +- **SNMP devices**
+ can be monitored too (although you will need to configure these) + +And you can extend it, by writing plugins that collect data from any source, using any computer language. + + +## Links + + * Report a bug: https://github.com/YunoHost-Apps/netdata_ynh/issues + * NetData website: http://my-netdata.io/ + * YunoHost website: https://yunohost.org/ diff --git a/manifest.json b/manifest.json index 1c92810..97cb9dd 100644 --- a/manifest.json +++ b/manifest.json @@ -6,7 +6,7 @@ "en": "real-time performance and health monitoring", "fr": "Monitoring serveur en temps reel" }, - "version": "1.6.0", + "version": "1.7.0", "url": "http://my-netdata.io/", "license": "GPL-3.0", "maintainer": { diff --git a/scripts/_common.sh b/scripts/_common.sh index 193af18..dbf17c6 100644 --- a/scripts/_common.sh +++ b/scripts/_common.sh @@ -4,7 +4,7 @@ # # Application version -VERSION="1.6.0" +VERSION="1.7.0" # Remote URL to fetch application source archive APPLICATION_SOURCE_URL="https://github.com/firehol/netdata/releases/download/v${VERSION}/netdata-${VERSION}.tar.gz" diff --git a/scripts/install b/scripts/install index 73062a0..c5bbf88 100644 --- a/scripts/install +++ b/scripts/install @@ -53,6 +53,9 @@ sudo setfacl -m u:netdata:rw /var/run/dovecot/stats # Add netdata to the adm group to access web logs sudo usermod -a -G adm netdata +# Extend weblog path to process logs for every domain +sed -i "s@path: '/var/log/nginx/access.log'@path: '/var/log/nginx/*-access.log'@g" /opt/netdata/etc/netdata/python.d/web_log.conf + # Declare service for YunoHost monitoring sudo yunohost service add netdata --log "/opt/netdata/var/log/netdata/error.log" "/opt/netdata/var/log/netdata/access.log" "/opt/netdata/var/log/netdata/debug.log" diff --git a/scripts/upgrade b/scripts/upgrade index 640938f..948ae90 100644 --- a/scripts/upgrade +++ b/scripts/upgrade @@ -58,6 +58,9 @@ sudo setfacl -m u:netdata:rw /var/run/dovecot/stats # Add netdata to the adm group to access web logs sudo usermod -a -G adm netdata +# Extend weblog path to process logs for every domain +sed -i "s@path: '/var/log/nginx/access.log'@path: '/var/log/nginx/*-access.log'@g" /opt/netdata/etc/netdata/python.d/web_log.conf + # Declare service for YunoHost monitoring sudo yunohost service add netdata --log "/opt/netdata/var/log/netdata/error.log" "/opt/netdata/var/log/netdata/access.log" "/opt/netdata/var/log/netdata/debug.log"