System Monitoring with Xymon/Administration Guide

All things related system administration will be documented here.

Design Overview

Xymon Protocol

There is a version of Xymon protocol in ASCII text format from Xymon author.

Architecture of a Xymon System Monitoring Environment

TBC

Picking an OS for Xymon Server

These are some notes and advice from Xymon users.

Linux

Red Hat Enterprise Linux / CentOS
Debian

Oracle Solaris 10

Pros

Plus 1: Turbocharged TCP/IP.
Plus 2: dtrace
Plus 3: Self Heal
Plus 4: You can configure root and disk to use zfs and have zfs snapshot enabled.

Cons

Minus 1: Xymon depended on other open source software that doesn't come with Oracle Solaris by default. Following are three sources where you can get the software in binary or source code format.

http://www.blastwave.org
http://www.sunfreeware.com has lots of open source.
http://www.thewrittenword.com

List of software required to meet all dependecies and order of installation:

common-1.4.5-SunOS5.8-sparc-CSW.pkg.gz
pcre-4.5-SunOS5.8-sparc-CSW.pkg.gz
fping-2.4,REV=2004.10.12_rev=b2_to_ipv6-SunOS5.8-sparc-CSW.pkg.gz
zlib-1.2.3,REV=2007.05.12-SunOS5.8-sparc-CSW.pkg.gz
png-1.2.18-SunOS5.8-sparc-CSW.pkg.gz
libiconv-1.9.2-SunOS5.8-sparc-CSW.pkg.gz
expat-1.95.7-SunOS5.8-sparc-CSW.pkg.gz
ggettext-0.14.1,REV=2005.06.29-SunOS5.8-sparc-CSW.pkg.gz
libpopt-1.7,REV=2004.05.15-SunOS5.8-sparc-CSW.pkg.gz
chkconfig-1.2.24h,REV=2006.12.12-SunOS5.8-sparc-CSW.pkg.gz
libpopt-1.7,REV=2004.05.15-SunOS5.8-sparc-CSW.pkg.gz
openssl-0.9.8,REV=2007.05.10_rev=e-SunOS5.8-sparc-CSW.pkg.gz
imaprt-2004,REV=2006.09.02_rev=g-SunOS5.8-sparc-CSW.pkg.gz
freetype2-2.1.10,REV=2005.12.11-SunOS5.8-sparc-CSW.pkg.gz
libart-2.3.16-SunOS5.8-sparc-CSW.pkg.gz
berkeleydb44-4.4.20,REV=2007.01.27-SunOS5.8-sparc-CSW.pkg.gz
ncurses-5.5,REV=2006.02.10-SunOS5.8-sparc-CSW.pkg.gz
readline-5.0,REV=2005.06.07-SunOS5.8-sparc-CSW.pkg.gz
gbc-1.06-SunOS5.8-sparc-CSW.pkg.gz
gdbm-1.8.3,REV=2006.01.01-SunOS5.8-sparc-CSW.pkg.gz
perl-5.8.8,REV=2007.03.16-SunOS5.8-sparc-CSW.pkg.gz
cvs-1.11.22-sol10-sparc-local.gz
rrdtool-1.2.19,REV=2007.02.07-SunOS5.8-sparc-CSW.pkg.gz
libnet-1.0.2,REV=2004.04.08_rev=a-SunOS5.8-sparc-CSW.pkg.gz
berkeleydb4-4.2.52,REV=2005.04.28_rev=p4-SunOS5.8-sparc-CSW.pkg.gz
sasl-2.1.22,REV=2007.06.19-SunOS5.8-sparc-CSW.pkg.gz
openldap_rt-2.3.35,REV=2007.04.14-SunOS5.8-sparc-CSW.pkg.gz
xymon-4.2.0,REV=2007.04.12-SunOS5.8-sparc-CSW.pkg.gz
xymon_client-4.2.0,REV=2007.04.12-SunOS5.8-sparc-CSW.pkg.gz

Notes

To avoid "xymond status-board not available" error message in bbgen webpage, add "set ip:do_tcp_fusion = 0x0" into /etc/system to disable TCP fusion.
1. References: http://www.hswn.dk/hobbiton/2007/04/msg00187.html
2. Solaris 5.10 kernel patch 120011-14-1, it fix this bug "6449337 kmem exhaustion caused by tcp fusion flow control logic error" .

Xymon Server: Solaris Intel 11/06 U3 VMware appliance on a 2GB flash pen drive

Following are main procedures for this to-go Xymon server.

VMware server 1.0.1 to create Solaris 10 VMware session.
Create a 1.9G partition, select custom install.
modify the partition table to take out /export/home,only leave /swap and /.
decrease default 512M swap size to 300M.
select "Core group" (about 573M in size).
Install httpd server
Install xymon server

Xymon Server and Development: Solaris Intel 11/06 U3 VMware appliance on a 4GB flash pen drive

VMware server 1.0.1 to create Solaris 10 VMware session.
- Need to use vmware player 1.0.3 so dhcp will work.

Xymon Server Test site

Solaris Intel 11/06 U3 VMware appliance on a 4GB flash pen drive

Operational difference between Xymon and BB BTF

Servers

This is a comparison table on how Xymon server is different from BB when performing an administration task.

Operation	Xymon 4.2.0 and above	Big Brother BTF (Better Than Free, 1.9c version above)
start/stop server	~/xymon.sh start/stop	~/runbb.sh start/stop
Delete a host	$XYMONHOME/bin/xymon 127.0.0.1 "drop HOSTNAME [test]"	$BBHOME/bin/bbrm
add a host	1. add hostnames into hosts.cfg	1. add hostnames into bb-hosts
Log data path	1.	1.

Clients

This is a comparison on how Xymon is different from BB when performing an administration task.

Operation	Xymon 4.2.0 above	Big Brother BTF(Better Than Free, 1.9c version above)
addin external module	~xymon/client/etc/xymonclient.cfg	$BBHOME/etc/bb-extab

References

http://www.hswn.dk/hobbit/help/about.html

Capacity Planning

rule of Thumb is 5MB disk space on Xymon server per machine being monitored

Installation

Windows

Client

Run the BBWin 0.13 installer.
Under HKEY_LOCAL_MACHINE\SOFTWARE\BBWin (32-bit) or HKLM\SOFTWARE\Wow6432Node\BBWin (64-bit) in the registry set the computer name (as it is in the bbhosts file)

Make the top of the config file in C:\Program Files\BBWin\etc (or C:\Program Files (x86)\BBWin\etc on Windows x64 systems) look like this:

<setting name="bbdisplay" value="xymon server name" />

<!-- bbwin mode local or central -->
<setting name="mode" value="central" />
<setting name="configclass" value="win32" />

Delete or comment out the default lines:

<cpu>
     <setting name="default" warnlevel="85%" paniclevel="95%" delay="3" />

   ...snip ...

<disk>
     <setting name="default" warnlevel="85%" paniclevel="95%" />

This causes these thresholds to be set at the server side. Any settings here will override the settings in the server's analysis.cfg file. It is much easier to manage these settings centrally.
Start the service at the server.
Then edit /home/xymon/server/etc/analysis.cfg and add:

#Hostname entries from bbwin clients.
#
HOST=[[new host name, as it appears in the bbhosts file]]
        LOAD 65 75       # Load thresholds are in %
        DISK C 80 90
        DISK D 90 95
        MEMPHYS 75 101
        MEMSWAP 75 85
        MEMACT  75 85
        PROC BBWin.exe 1 1

Server

/xymon/server/etc/client-local.cfg:

[win32]
eventlog:Security
ignore Success
eventlog:System
ignore Information
eventlog:Application
ignore Information

filtering in: /xymon/server/etc/analysis.cfg

CLASS=win32
        LOAD 80 90 # Load thresholds are in %
        PROC BBWin.exe 1 1
        PORT STATE=LISTENING MIN=0 TRACK=Listen TEXT=Listen
        LOG %.*  %error -.* COLOR=yellow
        LOG eventlog:Security  %failure.* COLOR=yellow
        LOG eventlog:Application  %warning.* COLOR=yellow
        LOG eventlog:System  %error.* COLOR=yellow

Instead you can use the following, but every update to the eventlog is send to the xymon server (instead of local filteret first).

CLASS=win32
        LOAD 80 90 # Load thresholds are in %
        PROC BBWin.exe 1 1
        PORT STATE=LISTENING MIN=0 TRACK=Listen TEXT=Listen
        LOG %.*  %^error.* COLOR=red #IGNORE=TermServDevices \(
        LOG %.*  %^warning.* COLOR=yellow IGNORE=%.*TermServDevices.*
        LOG %.*  %^failure.* COLOR=yellow

Unix-like

AIX
Debian (Ubuntu)
FreeBSD
HP-UX
IRIX
Mandriva (xymon 4.2.3 is available in contrib as of 2009.0, prior to that hobbit was available in contrib)
NSLU2 Unslung OS
RedHat Linux / RedHat Enterprise Linux / Fedora Core (http://rpm.razorsedge.org/ or http://staff.telkomsa.net/packages/)
~~Solaris~~ As of 20 September 2012, Blastwave has ceased its operation, and its Web site is no longer accessible.

Client

xymon:/home/xymon/client/bin # ls -lrt
total 2020
-rwxr-xr-x 1 xymon 1000   1915 Jul 31  2011 xymonclient-unixware.sh
-rwxr-xr-x 1 xymon 1000   3389 Jul 31  2011 xymonclient-sunos.sh
-rwxr-xr-x 1 xymon 1000   1849 Jul 31  2011 xymonclient-sco_sv.sh
-rwxr-xr-x 1 xymon 1000   1708 Jul 31  2011 xymonclient-osf1.sh
-rwxr-xr-x 1 xymon 1000   1914 Jul 31  2011 xymonclient-openbsd.sh
-rwxr-xr-x 1 xymon 1000   1917 Jul 31  2011 xymonclient-netbsd.sh
-rwxr-xr-x 1 xymon 1000   2821 Jul 31  2011 xymonclient-linux.sh
-rwxr-xr-x 1 xymon 1000   1842 Jul 31  2011 xymonclient-irix.sh
-rwxr-xr-x 1 xymon 1000   2421 Jul 31  2011 xymonclient-hp-ux.sh
-rwxr-xr-x 1 xymon 1000   2092 Jul 31  2011 xymonclient-freebsd.sh
-rwxr-xr-x 1 xymon 1000   1550 Jul 31  2011 xymonclient-darwin.sh
-rwxr-xr-x 1 xymon 1000   1979 Jul 31  2011 xymonclient-aix.sh
-rwxr-xr-x 1 xymon 1000   3252 Dec 12 22:15 xymonclient.sh
-rwxr-xr-x 1 xymon root 187072 Feb  8 09:33 xymonlaunch
-rwxr-xr-x 1 xymon root 288748 Feb  8 09:33 xymongrep
-rwxr-xr-x 1 xymon root 210216 Feb  8 09:33 xymondigest
-rwxr-xr-x 1 xymon root 153410 Feb  8 09:33 xymoncmd
-rwxr-xr-x 1 xymon root 151751 Feb  8 09:33 xymoncfg
-rwxr-xr-x 1 xymon root 180799 Feb  8 09:33 xymon
-rwxr-xr-x 1 xymon root 179969 Feb  8 09:33 orcaxymon
-rwxr-xr-x 1 xymon root 171691 Feb  8 09:33 msgcache
-rwxr-xr-x 1 xymon root 240486 Feb  8 09:33 logfetch
-rwxr-xr-x 1 xymon root 188930 Feb  8 09:33 clientupdate

xymon:/home/xymon/client/bin # ./xymon
Xymon version 4.3.7
Usage: ./xymon [--debug] [--merge] [--proxy=http://ip.of.the.proxy:port/] RECIPIENT DATA
  RECIPIENT: IP-address, hostname or URL
  DATA: Message to send, or "-" to read from stdin

Server

Building from package source using TWW HPMS

TWW Hyper Package Management system can help a software developer or system administrator to create different native package formats for different OS. The package source for compiling and packaging hobbit client and server software are in XML format that can be repeated reliably with TWW's sb and pb tools.

Hobbit server and Hobbit client package source is GPL licensed on TWW's support ftp server.

Building from src RPM

Sometimes it's better to build your own RPMs specifically for your environment. If you are using RH Enterprise or CentOS, the Fedora Core or generic RPM may not install correctly. You could also run into this problem if you have versions of dependent libraries that are not compatible with the system that the RPM was built on.

In order to build the src RPM, you'll need several packages:

openssl-devel, openldap-devel, and pcre-devel from the CentOS CDs.
- You may also have to make a link from /usr/include/pcre/pcre.h to /usr/include/pcre.h
rrdtool-devel
- I recommend getting this from the DAG repository
fping
- Also available from the DAG repository

RPMs from a matching version of RHEL usually work on CentOS with no problem (for example RPMs for EL 4 work fine on CentOS 4)

Once you have all the dependencies installed, download the src RPM from SourceForge. Once you have that, just run rpmbuild --rebuild hobbit-xxxx.src.rpm. For example:

rpmbuild --rebuild hobbit-4.1.0-1.src.rpm

The rpmbuild command should compile and build the RPM for you. You can watch the compiler output for any problems. After it is done, you should have new RPMs in the /usr/src/redhat/RPMS/i386 directory (assuming your architecture is i386). This process will build both server and client RPMs for your system. The server RPM also includes the client, so it is not necessary to install both of them.

SUSE

Dependencies for installation include apache2, apache2-utils, gcc, libstdc++-devel, net-snmp, pcre, pcre-devel, rrdtool and rrdtool-devel. Download the latest Xymon source from http://sourceforge.net/projects/xymon/files/Xymon/. Ensure that mod_rewrite is enabled in apache2, from YAST -> Network Services -> HTTP Server -> Server Modules.

$ useradd -m xymon
$ ./configure.server
[...]
*Where do you want the Xymon installation [/home/xymon] ? 
[...]
What group-ID does your webserver use [nobody] ? www
[...]
$ make
[...]
Now run 'make install' as root
$ make install
[...]
Installation complete.

cp /home/xymon/server/etc/xymon-apache.conf to /etc/apache2/conf.d/
htpasswd2 -c /home/xymon/server/etc/xymonpasswd <choose an administrative user name>
Ensure that fping can be executed by user xymon, either via appropriate sudo permissions, or by chmodding fping to setuid root.
Start the apache2 service.
/home/xymon/server/bin/xymon.sh start

Ubuntu

With Synaptic, install the PCRE and RRDtool libraries^[1]. Then, download xymon and unpack it.

Launch a terminal (CTRL + t) and enter the commands below, in order to install the software in your HTTP directory. Example with Apache:

$ adduser xymon
$ cd /home/Desktop/xymon
$ ./configure.server
[...]
Where do you want the Xymon installation [/home/xymon] ? /var/www/xymon
[...]
What group-ID does your webserver use [nobody] ? xymon
[...]
$ make
[...]
Now run 'make install' as root
$ make install
[...]
Installation complete.

You must configure your webserver for the Xymon webpages and CGI-scripts.
A sample Apache configuration is in /var/www/xymon/server/etc/xymon-apache.conf
If you have your Administration CGI scripts in a separate directory,
then you must also setup the password-file with the htpasswd command.

To start Xymon, as the xymon user run '/var/www/xymon/server/bin/xymon.sh start'
To view the Xymon webpages, go to http://localhost/xymon

If it hasn't already been done, it's necessary to configure Apache to execute the CGI programs:

$ vim /etc/apache2/httpd.conf
# Add the following lines without the sharps and save:
<Directory /var/www/*>
Options +ExecCGI
AddHandler cgi-script .cgi
</Directory> 
$ /etc/init.d/apache2 restart
$ su xymon /home/xymon/server/bin/xymon.sh start
Xymon started

Finally, test the software: http://localhost/xymon/server/bin/confreport.cgi

Hobbit in HA

There are two approaches to implement High Availability for Xymon servers,HA-LAN and HA-WAN. Pick one of them according to your network structure.

HA-LAN approach

This approach is using clustering software to do fail over using a set of Xymon servers. Each OS has their own version of clustering software. We know for Linux we can use Linux-HA plus DRBD. For Solaris, we have Sun Cluster Software.

The cons of this approach is the High Availability is at the scale of LAN not WAN level. The server in clustering need to reside at same LAN subnet. If the clustering site went down then we will end up with xymon messages has no place to send message to.

HA-LAN using LinuxHA and DRBD

HA-LAN using Solaris Sun Cluster software plus TrueCopy

HA-WAN approach

For networks that span over states or countries, failing over a primary xymon server to standby server over WAN network is not an easy networking task.

Following HA-WAN architecture can do fail-over without involve network team to do dns or routing changes.

                
          hobbit.test.com                     hobbit2.test.com
                   | Primary                         | Standby Xymon server
                   |  <-----  heart beat ----->      | 
      LAN1         |                                 |     LAN2                        
     --------------------------             -------------------------
     ^           ^           ^                ^   ^          ^
     |           |           |                |   |          |
     |  ---------------------------------------   |          |
     |  |        |     ----------------------------          |
     |  |        |     |     |--------------------------     |
     |  |        |     |                                |    |
 hobbitc A     hobbitc B                              hobbitc C 
    LAN 3         LAN 4                                LAN 5 

LAN1: California
LAN2: Brazil
LAN3: Argentina
LAN4: Mexico
LAN5: Japan

Requirements

a script that can detect failing of hobbit.test.com services.

Notes

hobbit2.test.com pager module is disabled.
Hobbit2.test.com and hobbit.test.com reside on different sites connected by WAN.
Hobbit clients does not lock on to hobbit.test.com alone.
Each hobbit client send messages to both hobbit.test.com and hobbit2.test.com
Hobbit2.test.com has every thing hobbit.test.com has and become active as hobbit2.test.com to send out alerts for hobbit.test.com.
There is no need to do ip failover of hobbit.test.com to hobbit2.test.com.

Pros

No need to alter existing network configuration.

Cons

Increase network bandwidth by sending same message to two different servers.

HA-WAN 2 approach

From Patrick: we have 3 data centres and each data centre contains a xymon server. All clients in a data centre only report to their local xymon server. However the xymon servers can communicate with each other using BBDISPLAYS (its a little more complicated than that as we utilise a bbproxy in each DC to take the messages and spray them to all 3 xymons).

                       
          hobbit1.test.com                     hobbit2.test.com
                   | Primary                         | Standby Xymon server
                   |  <-----  bbproxy    ----->      | 
      LAN1         |                                 |     LAN2                        
     --------------------------             -------------------------
     ^          ^     ^                                ^
     |          |     |                                |
     |          |     |                                |
     |          |     |                                |
     |          |     |                                |    
 hobbitc A     hobbitc B                              hobbitc C 

LAN1= has hobbitc A,B
LAN2= has hobbitc C

HA-WAN3 approach

This is a two node hobbit loosely-coupled cluster across WAN. It has following challenge need to be resolved.

hobbit.test.com DNS need to failover to hobbit2 from hobbit1 when hobbit1 is down.
The web page on hobbit1 and hobbit2 are not in sync.
Maintence records are not in sync between two servers.
RRD databases on two hobbit servers are not in sync after either one server is down for a while.


                           hobbit.test.com
                                 -> hobbitdynamic.test.com (using CISCO DD software).
                                      -> hobbit1.test.com
                                      -> hobbit2.test.com

                
          hobbit1.test.com                     hobbit2.test.com
                   | Primary                             | Standby Xymon server                              
                   |  <----- 1985 heart beat ----->      | 
                   |  <----- 1986 history    ----->      | 
                   |  <----- 1987 heart beat ----->      | 
      LAN1         |                                     |     LAN2                        
     --------------------------             -------------------------
     ^           ^           ^                ^   ^          ^
     |           |           |                |   |          |
     |  ---------------------------------------   |          |
     |  |        |     ----------------------------          |
     |  |        |     |     |--------------------------     |
     |  |        |     |                                |    |
 hobbitc A     hobbitc B                              hobbitc C 
    LAN 3         LAN 4                                LAN 5 

LAN1: California
LAN2: Brazil
LAN3: Argentina
LAN4: Mexico
LAN5: Japan

Requirements

a script that can detect failing of hobbit.test.com services.

Notes

hobbit2.test.com pager module is disabled.
Hobbit2.test.com and hobbit.test.com reside on different sites connected by WAN.
Hobbit clients does not lock on to hobbit.test.com alone.
Each hobbit client send messages to both hobbit.test.com and hobbit2.test.com
Hobbit2.test.com has every thing hobbit.test.com has and become active as hobbit2.test.com to send out alerts for hobbit.test.com.
There is no need to do ip failover of hobbit.test.com to hobbit2.test.com.

Pros

No need to alter existing network configuration.

Cons

Increase network bandwidth by sending same message to two different servers.

Hobbit HA on LAN

           
          hobbit.test.com                       hobbit2.test.com
                   |       HA Software                 |
                   |    <-  heart beat ->              | 
                   |                                   | LAN1: 192.168.1.0
  ----------------------------------------------------------------
     ^          ^    ^
     |          |    |
     |          |    ---------------------------
     |          |                              |
     |          |                              |
     |          |     
 hobbitc A     hobbitc B                   hobbitc C 
 LAN 2          LAN 3                        LAN4

LAN1: California
LAN2: Brazil
LAN3: Argentina
LAN4: Mexico

Notes

HA Software = Sun Cluster 3.2 + Sun AVS
hobbit2.test.com and hobbit.test.com reside on same subnet(same site).
Cluster software (Sun Cluster 3.2) is used to do hobbit.test.com fail over.
Each hobbit client send messages to hobbit.test.com only.
hobbit2.test.com has every thing hobbit.test.com has.
hobbit2.test.com is monitoring hobbit.test.com and will assume hobbit.test.com's identity.
identity: MAC address and IP address of hobbit.test.com

Pros

Close to real-time fail-over.

Cons

Fail over happens only on LAN, not WAN.

SunCluster

Free and opensourced clustering software from Sun. Commercial technical support is available.

Using two sol-nv-b68-x86 VMware sessions with Sun Cluster express 07/07.

References

FST HA

An opensource Clustering solution specifically for Solaris.

FSTha vs other HA

Small Text

Hobbit Configuration and tuning

Encryption and Tunnelling

Hobbit(bb)/XyMon port 1984 encryption Using Stunnel

References: http://www.stunnel.org/

Plain text bb message will be a bottleneck to make Hobbit a enterprise solution which require high security standard. Following is an attempt to make your CIO smile on hobbit solution. Note: It is possible to use reverse SSH tunnels, using Padraig Lennon's ssh_tunnels.sh script. instead of Stunnel server and client. See more details in Monitor Hobbit clients in a DMZ using reverse SSH tunnels

Machine A : has both HB Server and Stunnel server running.
Machine B : is a BB client.
Machine C : is a hobbit client with stunnel client enabled. hb client will send bb message via encrypted port 1999.
Machine D : is a HB client.
Note: old bb port is one way, hb's bb protocol's is bi-directional.

      Machine A (192.168.1.111)                                          

    ---------------------------
     HB Server process         |   <---------port 1984 <---------  BB client (Machine B)
         |                     |
         |1984                 |   <---------port 1984 --------->  HB client (Machine D)
         |                     |                                   
   Stunnel Server process 1999 |   <-------- port 1999 ----------> 1999 Stunnel Client
   ----------------------------                                    |            (Machine C 192.168.1.141)
                                                                   |
                                                                   --1984 ---HB client

Configure stunnel server to run in hobbit server

stunnel config file on server to direct 1999 into local 1984 port.

accept = 1999, we accept any incoming bb message on port 1999.
connect = 127.0.0.1:1984, redirect 1999 to 1984 on hb server itself.

bash-3.00# cat /opt/stunnel420/etc/stunnel/stunnel.conf
<snip>
[hobbit-server]
accept  = 1999
connect = 1984
<snip>
bash-3.00#

starting stunnel server on machine A. we can see hobbit-server port redirection is ok.

bash-3.00# /etc/init.d/stunnel420 start
Starting universal SSL tunnel: stunnel2007.04.29 06:47:50 LOG7[1898:1]: RAND_status claims sufficient entropy for the PRNG
2007.04.29 06:47:50 LOG7[1898:1]: PRNG seeded successfully
2007.04.29 06:47:50 LOG7[1898:1]: Certificate: /opt/stunnel420/etc/stunnel/stunnel.pem
2007.04.29 06:47:50 LOG7[1898:1]: Certificate loaded
2007.04.29 06:47:50 LOG7[1898:1]: Key file: /opt/moto/stunnel420/etc/stunnel/stunnel.pem
2007.04.29 06:47:50 LOG7[1898:1]: Private key loaded
2007.04.29 06:47:50 LOG7[1898:1]: SSL context initialized for service pop3s
2007.04.29 06:47:50 LOG7[1898:1]: Certificate: /opt/stunnel420/etc/stunnel/stunnel.pem
2007.04.29 06:47:50 LOG7[1898:1]: Certificate loaded
2007.04.29 06:47:50 LOG7[1898:1]: Key file: /opt/stunnel420/etc/stunnel/stunnel.pem
2007.04.29 06:47:50 LOG7[1898:1]: Private key loaded
2007.04.29 06:47:50 LOG7[1898:1]: SSL context initialized for service hobbit-server
.
bash-3.00#

make sure stunnel is running.

bash-3.00# ps -eaf |grep stunnel
  nobody  1984     1   0 06:55:00 ?           0:00 /opt/stunnel420/sbin/stunnel
    root  2133  1811   0 07:04:32 pts/2       0:00 grep stunnel
bash-3.00#

Testing port 1999 on hb server directly, typing garbage message "asdf" and then control+d to quit.

bash-3.00# telnet machineA.test.com 1999
Trying 192.168.1.111...
Connected to machineA.test.com.
Escape character is '^]'.
asdf
Connection to machineA.test.com closed by foreign host.
bash-3.00#

We can see port 1999 has incoming message from 192.168.1.141(machine c)in stunnel log file on machine A.

bash-3.00# tail -10f /opt/stunnel420/etc/stunnel/stunnel.log
2007.04.29 06:55:00 LOG5[1983:1]: 125 clients allowed
2007.04.29 06:55:00 LOG7[1983:1]: FD 4 in non-blocking mode
2007.04.29 06:55:00 LOG7[1983:1]: FD 5 in non-blocking mode
2007.04.29 06:55:00 LOG7[1983:1]: FD 6 in non-blocking mode
2007.04.29 06:55:00 LOG7[1983:1]: SO_REUSEADDR option set on accept socket
2007.04.29 06:55:00 LOG7[1983:1]: pop3s bound to 0.0.0.0:995
2007.04.29 06:55:00 LOG7[1983:1]: FD 7 in non-blocking mode
2007.04.29 06:55:00 LOG7[1983:1]: SO_REUSEADDR option set on accept socket
2007.04.29 06:55:00 LOG7[1983:1]: hobbit-server bound to 0.0.0.0:1999
2007.04.29 06:55:00 LOG7[1984:1]: Created pid file /stunnel.pid
2007.04.29 06:55:35 LOG7[1984:1]: hobbit-server accepted FD=0 from 192.168.1.141:38764
2007.04.29 06:55:35 LOG7[1984:2]: hobbit-server started
2007.04.29 06:55:35 LOG7[1984:2]: FD 0 in non-blocking mode
2007.04.29 06:55:35 LOG7[1984:2]: TCP_NODELAY option set on local socket
2007.04.29 06:55:35 LOG5[1984:2]: hobbit-server accepted connection from 192.168.1.141:38764
2007.04.29 06:55:35 LOG7[1984:2]: SSL state (accept): before/accept initialization
2007.04.29 06:55:39 LOG3[1984:2]: SSL_accept: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
2007.04.29 06:55:39 LOG5[1984:2]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket
2007.04.29 06:55:39 LOG7[1984:2]: hobbit-server finished (0 left)

Configuring hb client to use port 1999

add hobbitclientLocalIP into hobbitclient.cfg file. We want hobbit client send bb message to itself.

bash-3.00# grep ^BBDISPLAYS   /etc/opt/hobbitclient42/hobbitclient.cfg
BBDISPLAYS="myotherhobbitserver.my.com hobbitclientLocalIP"                   # IP of multiple Hobbit servers. BBDISP must be "0.0.0.0".
bash-3.00#
bash-3.00# egrep -v '^;|^$'  /opt/stunnel420/etc/stunnel/stunnel.conf
cert = /opt/stunnel420/etc/stunnel/stunnel.pem
sslVersion = SSLv3
chroot = /opt/stunnel420/var/lib/stunnel/
setuid = nobody
setgid = nogroup
pid = /stunnel.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
debug = 7
output = stunnel.log
client = yes
[hobbitclient]
connect  =  hbServerRemoteIP:1999
accept   =  hbLocalIP:1984
bash-3.00#

A successful hobbit client stunneling to hobbit server using port 1999.

bash-3.00# grep 06:50   stunnel.log
2007.08.19 00:06:50 LOG7[14842:1]: hobbitclient accepted FD=0 from HobbitclientIP:63758
2007.08.19 00:06:50 LOG7[14842:3]: hobbitclient started
2007.08.19 00:06:50 LOG7[14842:3]: FD 0 in non-blocking mode
2007.08.19 00:06:50 LOG7[14842:3]: TCP_NODELAY option set on local socket
2007.08.19 00:06:50 LOG5[14842:3]: hobbitclient accepted connection from HobbitclientIP:63758
2007.08.19 00:06:50 LOG7[14842:3]: FD 1 in non-blocking mode
2007.08.19 00:06:50 LOG7[14842:3]: hobbitclient connecting HobbitServerIP:1999
2007.08.19 00:06:50 LOG7[14842:3]: connect_wait: waiting 10 seconds
2007.08.19 00:06:50 LOG7[14842:3]: connect_wait: connected
2007.08.19 00:06:50 LOG5[14842:3]: hobbitclient connected remote server from HobbitclientIP:63759
2007.08.19 00:06:50 LOG7[14842:3]: Remote FD=1 initialized
2007.08.19 00:06:50 LOG7[14842:3]: TCP_NODELAY option set on remote socket
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): before/connect initialization
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 write client hello A
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 read server hello A
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 read finished A
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 write change cipher spec A
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 write finished A
2007.08.19 00:06:50 LOG7[14842:3]: SSL state (connect): SSLv3 flush data
2007.08.19 00:06:50 LOG7[14842:3]:    1 items in the session cache
2007.08.19 00:06:50 LOG7[14842:3]:    2 client connects (SSL_connect())
2007.08.19 00:06:50 LOG7[14842:3]:    2 client connects that finished
2007.08.19 00:06:50 LOG7[14842:3]:    0 client renegotiations requested
2007.08.19 00:06:50 LOG7[14842:3]:    0 server connects (SSL_accept())
2007.08.19 00:06:50 LOG7[14842:3]:    0 server connects that finished
2007.08.19 00:06:50 LOG7[14842:3]:    0 server renegotiations requested
2007.08.19 00:06:50 LOG7[14842:3]:    1 session cache hits
2007.08.19 00:06:50 LOG7[14842:3]:    0 session cache misses
2007.08.19 00:06:50 LOG7[14842:3]:    0 session cache timeouts
2007.08.19 00:06:50 LOG6[14842:3]: SSL connected: previous session reused
2007.08.19 00:06:50 LOG7[14842:3]: Socket closed on read
2007.08.19 00:06:50 LOG7[14842:3]: SSL write shutdown
2007.08.19 00:06:50 LOG7[14842:3]: SSL alert (write): warning: close notify
2007.08.19 00:06:50 LOG6[14842:3]: SSL socket closed on SSL_shutdown
2007.08.19 00:06:50 LOG7[14842:3]: Socket write shutdown
2007.08.19 00:06:50 LOG5[14842:3]: Connection closed: 30068 bytes sent to SSL, 0 bytes sent to socket
2007.08.19 00:06:50 LOG7[14842:3]: hobbitclient finished (0 left)
bash-3.00#

Using HTTPS Transport

A posting at http://lists.xymon.com/archive/2011-October/032866.html describes a technique where Xymon clients can submit client messages using a web connection. It requires a CGI script to be installed on the Xymon server. This method can be used to connect via web proxies, and authentication can be achieved by configuring the web server to enforce client-side certificates or user/password logins.

Encryption via Secure-Shell (ssh) Tunnel

Xymon can be configured to use the IP address of an ssh tunnel, and thus its traffic will be encrypted. This section describes two ways to establish a tunnel between the Xymon server and Xymon client.

Persistent Tunnel

This method is essentially creating a kind of VPN between the Xymon server and the client. Once established, the Xymon client is configured with XYMSRV set to 127.0.0.1, and all updates are sent down the tunnel.

The simplest way to setup a persistent tunnel is with a tool such as Autossh. There's also a Xymon-specific add-on for establishing tunnels called ssh_tunnel.

Ephemeral ssh Tunnel

An ephemeral tunnel is a temporary tunnel created only when Xymon data need to be collected. Secure shell tunnels make use of key authentication so that passwords are not required. They can be established by ssh connection made in either direction, depending on requirements. In both cases, XYMSRV is set to 127.0.0.1.

Xymon Server to Client

For a server-to-client connection, the Xymon server runs an ssh connection to the client with a remote tunnel on port 1984, sets up some variables, and runs the Xymon client scripts. An example is shown here.

ssh -R1984:127.0.0.1:1984 -o batchmode=yes xymon@xymon-client '/usr/lib/xymon/client/bin/xymoncmd sh -c "XYMSRV=127.0.0.1 /usr/lib/xymon/client/bin/xymonclient.sh"'

This command can be put into tasks.cfg, run every 5 minutes.

Xymon Client to Server

For a client-to-server connection, the Xymon client establishes a connection to the server with a local tunnel on port 1984, and runs the Xymon client scripts. An example is shown here.

ssh -f -L1984:127.0.0.1:1984 xymon@xymon-server sleep 15 && /usr/lib/xymon/client/bin/xymoncmd sh -c "XYMSRV=127.0.0.1 /usr/lib/xymon/client/bin/xymonclient.sh"

This command should be run every 5 minutes on the Xymon client, and can be run from cron or from clientlaunch.cfg.

32 bit vs 64 bit binary for hobbit on Solaris

This article describe this subject in great detail.

Configuration

LDAP Authentication

Example httpd.conf (Apache 2.0.x with LDAP authenticated against Active Directory):

Substitute LDAPSERVER.DOMAIN.COM with your LDAP server

<USERNAME>: use account with permission to view LDAP directory

<PASSWORD>: password for account (You should limit what this account can do)

<Directory "/var/hobbit/cgi-secure">
    AllowOverride None
    Options ExecCGI Includes
    Order allow,deny
    Allow from all

    AuthType Basic
    AuthName "Hobbit Administration"
    AuthLDAPEnabled on
    AuthLDAPURL ldap://LDAPSERVER.DOMAIN.COM:389/dc=DOMAIN,dc=COM?sAMAccountName?sub?(objectClass=person)
    AuthLDAPBindDN "cn=<USERNAME>,cn=Users,dc=DOMAIN,dc=COM"
    AuthLDAPBindPassword <PASSWORD>
    require valid-user

</Directory>

Same for a Novell-edir ldap server:

<Directory "/usr/lib/hobbit/cgi-secure">
    AllowOverride None
    Options ExecCGI Includes
    Order allow,deny
    Allow from all

    AuthName "Hobbit-Admin"
    AuthType Basic
    AuthLDAPURL ldap://LDAPSERVER.DOMAIN.COM/o=TREE,ou=Users?cn?sub?(groupMembership=cn=your_group,ou=groups,o=TREE)
    require valid-user
</Directory>

Alerts setting

Pager

Using sms_client [smsclient.org]

Create a shell-script (/usr/bin/hobbitsms) like this:

#!/bin/bash
if [ $RECOVERED != 1 ]; then
echo $RCPT \"HOBBIT : $BBHOSTSVC  is $BBCOLORLEVEL\" >> /var/log/hobbit/page.log
/usr/bin/sms_client $RCPT "HOBBIT : $BBHOSTSVC  is $BBCOLORLEVEL"
else
echo $RCPT \"HOBBIT : $BBHOSTSVC  is weer OK\" >> /var/log/hobbit/page.log
/usr/bin/sms_client $RCPT "HOBBIT : $BBHOSTSVC  is OK"
fi

Edit hobbit-alerts.cfg and add the lines for the alerts you want to receive:

      SCRIPT /usr/bin/hobbitsms hobbit DURATION>5 FORMAT=SMS REPEAT=180 COLOR=red TIME=W:0730:1800 RECOVERED

Pager.

Using snpp sendpage.org

Create a shell-script (/usr/bin/hobbitsnpp) like this:

#!/bin/bash
/usr/bin/snpp -n $RCPT <<SCRIPTEOF
$BBALPHAMSG
SCRIPTEOF

Email.

Tuning

How to shorten Xymon Server nslook up time ?

Xymon server do lots nslookup for every five minutes on the machines that need to be pinged.

Install a local dns cache server. I use djbdns for it

How to shorten the ping test time ?

Hobbit and Remedy Ticket System

Overview

Remedy ticket system has a web interface for opening up a ticket to a particular ticket queue.

The Perl approach is to use the following software to automate the ticket request when an alert occurs.

perl
LWP
trouble_ticket.tgz on http://www.deadcat.net
an entrance URL on remedy server web interface.
A perl subroutine to open up remedy ticket.

Open Remedy ticket on hobbit alerts

Open Remedy ticket on demand

Migration from BB

Cost (efforts) of Migration

System and Inventory Monitoring

System monitoring and inventory monitoring can achieved by an external module to report a system's inventory's information.(TBC)

Trouble Shooting Guide

Q. When I click on a status icon I get the message "Status not available". What should I check?

A. First make sure that the server is actually running.

ps -ef | grep hobbitd

You should see several processes similar to:

hobbit   32717 32716  0 Nov07 ?        00:01:07 hobbitd --pidfile....
hobbit   32726 32716  0 Nov07 ?        00:00:03 hobbitd_channel --channel=page...
hobbit   32727 32716  0 Nov07 ?        00:01:58 hobbitd_channel --channel=status...
hobbit   32728 32716  0 Nov07 ?        00:00:01 hobbitd_channel --channel=data...
hobbit   32725 32716  0 Nov07 ?        00:00:00 hobbitd_channel --channel=stachg...

If the server is failing to start, start looking at the hobbit logs directory. Check here for one location

/var/log/hobbit

Q. After installing the Hobbit client, my msgs tests are "clear" (sometimes referred to as "white")

A. As of the time of this writing, the Hobbit client does NOT have msgs functionality like the BB client does. This can be added by installing the bb-msgs.sh file from the BB client as an external test. Even so, the Hobbit server will turn the test to "clear" instead of the expected status. To correct his issue, you'll have to edit the hobbitlaunch.cfg file (usually found in /etc/hobbit/ or /usr/lib/hobbit/server/etc/) to add --no-clear-msgs to the client channel and restart the server:

CMD hobbitd_channel --channel=client hobbitd_client --no-clear-msgs --log=$BBSERVERLOGS/clientdata.log ...

References

↑ http://www.xymon.com/xymon/help/install.html