Wikibooks

FreedomBox for Communities/Monitoring Server

< FreedomBox for Communities

The monitoring client on the FreedomBox server of each of the communities will need to send the collected metrics to a central location. This central location, the monitoring server, needs to store the data sent, visualize the data for administrator and alert them when something is off. This section describes the setup on the server. It is assumed that the server is also running a FreedomBox. However, with minor changes to the instructions any server running Debian GNU/Linux Buster or above can be used as a server.

Setup Graphite Carbon

edit

Graphite Carbon server is responsible collecting the data from the various FreedomBox servers and storing them in a time-series database called Whisper. Run the following commands to setup Graphite Carbon.

  1. Install Graphite Carbon. 
    apt install graphite-carbon
  2. Open up ports in the firewall for receiving data from FreedomBox Servers. 
    firewall-cmd --zone=external --add-port=2003/tcp
firewall-cmd --zone=external --add-port=2003/tcp --permanent
  3. After this, graphite-carbon server will be running and listening on port 2003. This can be checked using the following commands: 
    systemctl status carbon-cache
ss -nlpt | grep carbon-cache

Setup Graphite Web

edit

Graphite Web service is responsible for serving the data stored by Graphite Carbon to consumers that can visualize the data such as Grafana. Run the following commands to setup Graphite Web.

  1. Install Graphite Web 
    apt install graphite-web libapache2-mode-wsgi-py3
  2. Generate a secret key for Django framework used by Graphite Web. 
    python3 -c 'import random; result = "".join([random.choice("abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)") for i in range(64)]); print(result)'
  3. Note the key generated from previous command and set it as secret key in the file /etc/graphite/local_settings.py. 
    SECRET_KEY="<output from the key generation command>"
  4. Create a database for Graphite Web to store its own data. 
    sudo --user _graphite graphite-manage migrate --run-syncdb
chown _graphite:_graphite /var/lib/graphite/graphite.db
chmod 0600 /var/lib/graphite/graphite.db
  5. Create Apache configuration for graphite-web and start serving. 
    cat << EOF > /etc/apache2/sites-available/graphite-web.conf
Listen 8080

<VirtualHost *:8080>
        WSGIDaemonProcess _graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 user=_graphite group=_graphite
        WSGIProcessGroup _graphite
        WSGIImportScript /usr/share/graphite-web/graphite.wsgi process-group=_graphite application-group=%{GLOBAL}
        WSGIScriptAlias / /usr/share/graphite-web/graphite.wsgi

        Alias /static/ /usr/share/graphite-web/static/
        <Location "/static/">
                SetHandler None
        </Location>
</VirtualHost>
EOF
a2ensite graphite-web
systemctl restart apache2

Setup Grafana

edit

Grafana consumes data from Graphite Web and produces beautiful visualizations. It can also be configured for alerting when certain event occurs in the data. Run the following commands to setup Grafana.

  1. Install Grafana from it's project's Debian repositories. 
    apt-get install -y apt-transport-https
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
cat << EOF > /etc/apt/sources.list.d/grafana.list
deb https://packages.grafana.com/oss/deb stable main
EOF
apt update
apt install grafana
  2. Configure Grafana to be served from the a existing domain under a sub-path. Edit the file /etc/grafana/grafana.ini and ensure the following two directives are configured as shown. 
    root_url = https://<domain_name>/monitor/
serve_from_sub_path = true
  3. Configure Grafana to start every time the machine is booted and start Grafana. 
    systemctl daemon-reload
systemctl enable grafana-server.service
systemctl start grafana-server.service
  4. Create Apache configuration to proxy Grafana from a sub-path and start serving. 
    cat << EOF > /etc/apache2/conf-available/grafana.conf
<Location /monitor/>
    ProxyPass http://localhost:3000/monitor/
</Location>
EOF
a2enconf grafana
systemctl reload apache2
  5. Setup credentials for Grafana access. Visit https://<domain_name>/monitor/ and login with the following credentials. You will be prompted to change the password. Set a strong password.
    • Username: admin
    • Password: admin
  6. Create user accounts for each for users who will using the monitoring setup. In the left menu, go to Server Admin, Users. Click on the New User button and provide details such as name, email, username and password to create a new user account.
  7. Connect Grafana to the Graphite Carbon database via Graphite Web we have configured in the previous sections. Go to Configuration, Data Sources and select Carbon data source. In the configuration dialog that shows up, provide the following details: The press Save & Test. You should see a message confirming the data source is accessible.

Creating a Dashboard for Each Community

edit

In Grafana interface, you need to create a dashboard for monitoring each of the communities you are the administering. The following instructions provide an overview of how this can be done:

  1. Login to Grafana interface.
  2. In the menu, select Create, Dashboard. A dashboard is created and a prompt is shown to add a new panel. Select a Graph.
  3. A new dashboard is created with a simple graph.
  4. Save dashboard by clicking the save icon. In the save dialog, provide a name for the dashboard based on the community for which the dashboard is being created. Click Save.
  5. To start showing proper data in the panel, click on Panel Title and select Edit. In the Metric tab, click on select metric drop down. You should see the name of the domain you assigned to your community as configured in collectd. Then select a metric such as cpu-0 and then cpu-user. You should start seeing the plot of the metric in the graph. Save the dashboard.

Parameters to Monitor

edit

It is recommended that you create a dashboard to your taste but monitor the following parameters at least for each community:

  • CPUs
    • Series: <community> • * • cpu-user
    • Functions: averageSeries() • alias(User)
    • Series: <community> • * • cpu-nice
    • Functions: averageSeries() • alias(Nice)
    • Series: <community> • * • cpu-system
    • Functions: averageSeries() • alias(System)
    • Series: <community> • * • cpu-wait
    • Functions: averageSeries() • alias(Wait)
  • Memory
    • Series: <community> • memory • percent-used
    • Functions: alias(% Used)
  • Load
    • Series: <community> • load • load • shorterm
    • Functions: alias(5min)
    • Series: <community> • load • load • midterm
    • Functions: alias(10min)
    • Series: <community> • load • load • longterm
    • Functions: alias(15min)
  • Disk
    • Series: <community> • <disk> • disk_ops • read
    • Functions: alias(Read ops)
    • Series: <community> • <disk> • disk_ops • write
    • Functions: alias(Write ops)
    • Series: <community> • <disk> • disk_octets • read
    • Functions: alias(Read Bytes)
    • Series: <community> • <disk> • disk_octets • write
    • Functions: alias(Write Bytes)
  • Disk I/O Time
    • Series: <community> • <disk> • pending_operations
    • Functions: alias(Pending Operations)
    • Series: <community> • <disk> • disk_io_time • io_time
    • Functions: alias(I/O Time)
    • Series: <community> • <disk> • disk_io_time • weighted_io_time
    • Functions: alias(Weighted I/O Time)
  • <mountpoint> Partition (for each partition)
    • Series: <community> • df-<mountpoint> • percent_bytes-used
    • Functions: alias(Used)
    • Series: <community> • df-<mountpoint> • percent_bytes-reserved
    • Functions: alias(Reserved)
  • <LAN/WAN> Network Bandwidth (for each interface)
    • Series: <community> • interface-<interface> • if_octets • rx
    • Functions: alias(Received)
    • Series: <community> • interface-<interface> • if_octets • tx
    • Functions: alias(Transmitted)
  • Connections Tracked
    • Series: <community> • conntrack • conntrack
    • Functions: alias(Connections Tracked)
  • Wi-Fi Access Points
    • alias(sumSeries(isNonNull(<community>.ping.ping-10_42_0_[1-9][0-9])), 'Wi-Fi Access Points')
  • Internet Uplink
    • alias(isNonNull(<community>.ping.ping-10_42_0_4), 'Internet Router')
    • alias(isNonNull(<community>.ping.ping-10_42_0_5), 'PTP Link - Router A')
    • alias(isNonNull(<community>.ping.ping-10_42_0_6), 'PTP Link - Router B')
  • Internet Ping
    • Series: <community> • ping • ping-1_1_1_1
    • Functions: alias(Internet Ping)
  • Processes
    • Series: <community> • processes • ps_state-running
    • Functions: alias(Running)
    • Series: <community> • processes • ps_state-sleeping
    • Functions: alias(Sleeping)
    • Series: <community> • processes • ps_state-paging
    • Functions: alias(Paging)
    • Series: <community> • processes • ps_state-blocked
    • Functions: alias(Blocked)
    • Series: <community> • processes • ps_state-zombies
    • Functions: alias(Zombies)
    • Series: <community> • processes • ps_state-stopped
    • Functions: alias(Stopped)
  • Uptime
    • Series: <community> • uptime • uptime
    • Functions: alias(Time Since Reboot)
  • Users
    • Series: <community> • users • users
    • Functions: alias(Logged-In Users)
  • collectd Queue
    • Series: <community> • collectd-cache • cache_size
    • Functions: alias(Cache Size)
    • Series: <community> • collectd-write_queue • derive-dropped
    • Functions: alias(Dropped Metrics)
    • Series: <community> • collectd-write_queue • queue_length
    • Functions: alias(Queue Length)
Retrieved from "https://en.wikibooks.org/w/index.php?title=FreedomBox_for_Communities/Monitoring_Server&oldid=3574503"