Square Bubble for IIB & MQ Operating Guide

Start

Choose from the available installation type:

Square Bubble for IIB & MQ ships with System V init.d and systemd scripts for RHEL 6 & 7 respectively, to control the running state of the program. To start, issue the following command as root or more preferably using sudo to restrict the user to only run a limited set of commands in such a privileged mode:

  • For RHEL 6:
    [...] # service sqrbbl start
  • For RHEL 7:
    [...] # systemctl start sqrbbl

From the installation directory, using the user that untarred the package, edit the sqrbbl-setup file if java is not on the local PATH then use the following command:

[...sqrbbl-...] $ bin/start-sqrbbl

This command will return immediately as it launches square bubble in the background. To check for errors look at the files in the log directory (especially sqrbbl-stdouterr.log).


Stop

Choose from the available installation type:

To stop, issue the following command as root (or via sudo):

  • For RHEL6:

    [...] # service sqrbbl stop

  • For RHEL7:

    [...] # systemctl stop sqrbbl

From the installation directory, using the user that untarred the package, edit the sqrbbl-setup file to set up the JAVA_HOME variable, then use the following command:

[...sqrbbl-...] $ bin/stop-sqrbbl

If you are using the IBM JDK or have not set up the JAVA_HOME variable to point to a jdk where ${JAVA_HOME}/lib/tools.jar is present you can kill the process. From the command terminal used to start square bubble, discover the pid of the running agent by use of the following command:

[...sqrbbl-...] $ ps -H

This command will return the process hierarchy from the current shell. One such process should be java. You can kill this process using the kill <pid> command.

If you are not in the same shell, use the following command

[...] $ ps -ef | grep java | grep sqrbbl

No return means that square bubble is not running. More than 1 return would suggest multiple instance are running.

A further alternative is to use Jolokia as a deployed javaagent at startup (configure in the sqrbbl-setup script) then use the configured URL to stop, such as http://localhost:7777/jolokia/exec/com.syntegrity.innovations.sqrbbl:type=Main/stop. Change the host and port as appropriate to your deployment.


Status

To query the running status, issue the following command as root (or via sudo):

  • For RHEL6:

    [...] # service sqrbbl status

  • For RHEL7:

    [...] # systemctl status sqrbbl

From the command terminal used to start square bubble, discover the pid of the running agent by use the following command:

[...sqrbbl-...] $ ps -H

This command will return the process hierarchy from the current shell. One such process should be java. If this is so, square bubble is running, if not square bubble is not.

If you are not running from the command terminal that square bubble was started from, try the following command:

[...] $ ps -ef | grep java | grep sqrbbl

No return means that square bubble is not running. More than 1 return would suggest multiple instance are running.

In addition, square bubble supports the JMX ethos to allow greater details of what is happening within the JVM. This is available using any of the following:

There are many more options to access JMX.


Logs

To view the logs, issue the following command (as any user, unless your system administrator has altered the permissions):

[...] $ cat /var/log/sqrbbl/sqrbbl.log

Or to see lines being added:

[...] $ tail -f /var/log/sqrbbl/sqrbbl.log

On RHEL7 you can also query the systemd journal by using the command:

[...] $ journalctl _UID=`id -u sqrbbl`

From the installation directory, you can cat or tail the log files in the log directory. e.g.

[...sqrbbl-...] $ cat log/sqrbbl-stdouterr.log
[...sqrbbl-...] $ tail -f log/sqrbbl.log

The sqrbbl-stdouterr.log is for start up messages directed to STDOUT and all messages directed to STDERR.

The sqrbbl.log file is the main agent log where activity is reported.


Validation

To validate the configuration, you need to have java installed and available on the PATH for user sqrbbl. This can also be achieved by modifying the /etc/sqrbbl/sqrbbl-setup file.

You can test this from root using the command:

[...] # su -c ". /etc/sqrbbl/sqrbbl-setup;java -version" -s /bin/bash sqrbbl

If you see something like "bash: java: command not found" then java has not been set up.

To run the validation script, from root, use the following command:

[...] # su -c "/opt/syntegrityinnovations/sqrbbl/bin/validate" -s /bin/bash sqrbbl

All messages will be displayed in the console.

To validate the configuration, you need to have java installed and available on the PATH for user that installed square bubble . This can also be achieved by modifying the <install dir>/sqrbbl-setup file.

You can test this from the install dir using the command:

[...] $ . sqrbbl-setup;java -version

If you see something like "bash: java: command not found" then java has not been set up.

To run the validation script, use the following command:

[...] $ bin/validate

All messages will be displayed in the console.

There are some additional switches that can be added to the validate command to modify the behaviour, which includes:


Additional jars

If you need to have additional jar files available to square bubble, as is required for the use of MQ security exits, you can now simply place them in the lib directory. For the rpm install, this is in /etc/sqrbbl/lib, for .tar.gz or .zip install, the lib directory should be created in the installation directory.

Additional jars are loaded after the Square Bubble jar.


Licensing

You need a valid license to run this software. Licenses can be purchased here and are to be copied into the directory /etc/sqrbbl/license.d either before the software is started or whilst it is running should additional license be acquired.

Each license file has attributes for:

Servers in this context are running OS images which are distinguished by IP address (or the DNS resolved IP address). All loopback addresses (localhost etc) are treated as the same server. This is irrespective of the number of services (IIB Integration nodes, servers etc and MQ Queue Managers) that are running. Should the configuration exceed the number of servers licensed, the software will not start. You can then either purchase another license or reduce the servers being monitored in your configuration (sources).

We use the inotify watcher to receive notifications whenever the directory, or its contents are modified. There are a number of events that we watch for:

The software will not start or perform any validation unless a valid license is made available. It will also stop once all current licenses expire, or there are insufficient licensed servers at the current time. This may also result from inadvertently removing one or more current license files.

There is another case where the entire license directory may be removed inadvertently. In this case the software will continue as if nothing happened however once the directory is restored, it will attempt to reload licenses as if started anew. Therefore, if this situation does arise, make sure you have all the required licenses in the directory before you restore or rename it.

Licenses may be deployed in more than one running instance but only if the total instances that use the same license maintain the limit on the overall number of servers being monitored. Should a license for 1 server exist, it MUST only be used in a single instance at any given point in time.

License usage is reported in the output destinations (files, splunk etc), and we can detect if they are being used incorrectly. If you are unsure whether you are operating within the license terms, please contact us at innovations@syntegrity.com.au immediately, for assistance.


MQ Connections

Connecting to MQ can be a challenge in its own right however we have developed a profile to make this easier and without compromising the integrity of your trusted MQ service.

We have identified the minimal authorisations needed to enable Square Bubble to extract the data required by the dashboards. The include the following :

setmqaut -m $1 -n SYSTEM.ADMIN.COMMAND.QUEUE -t q -p $3 +browse +dsp +get +inq +put +set
setmqaut -m $1 -n SYSTEM.DEFAULT.MODEL.QUEUE -t q -p $3 +browse +dsp +get +inq +put +set
setmqaut -m $1 -t qmgr -p $3 +connect +dsp +inq
setmqaut -m $1 -n '**' -t listener -p $3 +dsp
setmqaut -m $1 -n '**' -t service -p $3 +dsp
setmqaut -m $1 -n '**' -t clntconn -p $3 +dsp
setmqaut -m $1 -n '**' -t q -p $3 +dsp +inq
setmqaut -m $1 -n '**' -t chl -p $3 +dsp
        

Where $1 is the Queue Manager name and $3 is the user principal.

The user principal can be either a local user (non-privileged) or a signed certificate which could also be registered in an LDAP server.

This will work for all MQ only data types except QSTATS. To enable QSTATS you need to add another definition for the queues you wish to allow Square Bubble to run the REST QUEUE STATISTICS against, sush as:

 setmqaut -m $1 -n 'MYQUEUE*' -t q -p $3 +dsp +inq +dsp +chq 

If your square bubble config is not limited to those which have the +chq authorisation, you will receive an authorisation error, and no data. We would recommend you limit this authorisation to maintain system integrity.

We have also provided sample scripts based on a standard 8.0.0.4 installation of MQ that will create a Queue Manager that has these authorisations included so you can start testing right away. These scripts are not intended to run in a production environment and as such will require modification to suit the needs of your environment. The scripts take 3 parameters:

  1. Queue Manager name, to be created e.g. QM_SQRBBL
  2. Listener port e.g. 1414
  3. User principal (local user name) e.g. sqrbbl_user (which needs to be created on the local host and the password made available to square bubble)

They are available for:


Use of SSL/TLS

We support the use of secure sockets layer which will encrypt data passing between the agent and the output destination or source. We support only the latest protocols from TLS V1 onwards, as is common with browsers and of course JVMs.

As we use Java, this is a relatively straightforward affair however if you are not familiar with the use of certificates and private/public key pairs, we would suggest you read the following:

Trust

To trust certificates from a source or destination you will need the CA or CA chain in either the cacerts or another trust store. Assemble the collection of trusted CA certificates that you want to deploy. The trusted CA certificates can be obtained from public CAs or private CAs. The trusted CA certificates can be in any format that is compatible with the Java keystore utility, such as PEM format. All you need are the certificates themselves, the private keys and passwords are not required.

Given a CA certificate, cacert.pem, in PEM format, you can add the certificate to a JKS truststore (or create a new truststore) by entering the following command:

keytool -import -file cacert.pem -alias CAAlias -keystore truststore.ts -storepass StorePass

Where CAAlias is a convenient tag that enables you to access this particular CA certificate using the keytool utility. The file, truststore.ts, is a keystore file containing CA certificates. If this file does not already exist, the keytool utility will create it. The StorePass password provides access to the keystore file, truststore.ts.

This process can be used for all required CA certificates or self signed certificates (the latter is not recommended for production purposes).

To configure Square Bubble to use the truststore, which enables 1-way authentication, you need to add the appropriate switches to the sqrbbl-setup file, located in /etc/sqrbbl for rpm install or the <Install dir> for other installs:

#!/bin/sh
# run time config for sqrbbl
# use this config script to set PATH to point to the correct version of java, example below
JAVA_HOME=/usr/java/jre1.7.0_80
export JAVA_HOME
export PATH=${JAVA_HOME}/bin:${PATH}
...
# and set any additional java options required, as follows
START_JAVA_OPTS="-server -Xmx256m -Djavax.net.ssl.trustStore=<path>/truststore.jks -Djavax.net.ssl.trustStorePassword=<password>"
...

See also section Hiding passwords from command line below.

Client Authentication

To enable client, or mutual (two-way), certificate-based authentication, you need to enable the trust (as documented in the previous section) and add (or generate) private keys for each client identity. In a corporate environment or other large organisations, this may be created for you, in which case you will need to import the private key. This can be done by using openssl to generate a keystore and then import that to a java keystore. For example from a provided certificate (provided.crt):

openssl pkcs12 -export -name myservercert -in provided.crt -inkey mykey -out keystore.p12

Followed by:

keytool -importkeystore -destkeystore mykeystore.jks -srckeystore keystore.p12 -srcstoretype pkcs12 -alias mycert

To use the generated keystore and trustore, you need to modify the sqrbbl-setup file in /etc/sqrbbl/ to identify where these files are. The following is a complete example:

#!/bin/sh
# run time config for sqrbbl
# use this config script to set PATH to point to the correct version of java, example below
JAVA_HOME=/usr/java/jre1.7.0_80
export JAVA_HOME
export PATH=${JAVA_HOME}/bin:${PATH}
...
# and set any additional java options required, as follows
START_JAVA_OPTS="-server -Xmx256m -Djavax.net.ssl.trustStore=/home/user/mqssl/mytruststore.jks -Djavax.net.ssl.trustStorePassword=p@ssw0rd -Djavax.net.ssl.keyStore=/home/user/mqssl/mykeystore.jks -Djavax.net.ssl.keyStorePassword=p@ssw0rd"
...

If you have imported the required CA certificates into the cacert, you can omit the truststore options.

See also section Hiding passwords from command line below.

MQ setup

MQ can be a little tricky to get working with SSL/TLS however the following resources should help you:

The videos make extensive use of the IBM Websphere MQ Explorer which we would recommend you use to help setup your MQ environment.


Hiding passwords from command line

The examples above are not recommended in a production environment that may be shared with other applications or users. To prevent the passwords being visible, from the command line, we have added an option to include additional properties from a file. This is done by adding the -props option to the SQRBBL_OPTS var in sqrbbl_setup. So using the example the the Client Authentications section above, the sqrbbl-setup file will include:

#!/bin/sh
# run time config for sqrbbl
# use this config script to set PATH to point to the correct version of java, example below
JAVA_HOME=/usr/java/jre1.7.0_80
export JAVA_HOME
export PATH=${JAVA_HOME}/bin:${PATH}

# set square bubble options (other than config & licensedir)
SQRBBL_OPTS="-startsec 0 -props /etc/sqrbbl/sqrbbl-props"

# and set any additional java options required, as follows
START_JAVA_OPTS="-server -Xmx256m"
...

The additional properties file (/etc/sqrbbl/sqrbbl-props) should then have appropriate permissions and contents as follows:

# Additional system properties required by Square Bubble
javax.net.ssl.trustStore=/home/user/mqssl/mytruststore.jks
javax.net.ssl.trustStorePassword=p@ssw0rd
javax.net.ssl.keyStore=/home/user/mqssl/mykeystore.jks
javax.net.ssl.keyStorePassword=p@ssw0rd

The properties should conform to the java properties file standard, as documented in the Java Properties load method.

Each property is added to or overrides the system properties.


MQ CCDT

This is currently experimental

We have shipped a script which simplifies the creation of MQ Client Channel Definition Table, used to simplify and control client connections into MQ. More details about MQ CCDT can be found here.

The script requires the MQ client to be installed on the local machine and that the profile is initialized to make the MQ client tools available on the path of the shell being used. This can be done by running the command:

[...] $ . setmqenv -s

The script can be found /opt/syntegrityinnovations/sqrbbl/bin/sbDefQmgr. This script takes the following command line arguments:

Switch Description Default
-m QMgrName No default - required
-c ConnectionName No default - required
-n ChannelName "SQRBBL.QMgrName"
-t TLS cipher Suite name TLS_RSA_WITH_AES_128_CBC_SHA256
-x X509 certificate file name /etc/sqrbbl/sqrbbl.cer
-b Channel Table name AMQCLCHL.TAB
-l Channel Library name (directory) /etc/sqrbbl
-q QueueManager Certificate DN or file name empty (allow any queue manager DN)
-u User on QMgr for SquareBubble to use (should not be mqm) sqrbbl