Square Bubble for IIB & MQ Configuration Guide

Introduction

Our goal for configuration is to make it both very flexible but also simple. We use XML as it can be validated readily and is explicit in its definition.

We also allow restricting the data to only that which is required thus reducing the load placed upon the network and associated processing chain.

Square Bubble for IIB/MQ can also be deployed in a variety of different topologies ranging from being co-located with each monitored server or as a single instance monitoring a set of sources.

This page is only concerned with the configuration which applies to the XML configuration file (usually located at /etc/sqrbbl/sqrbbl-config.xml). We recommend you start with the Quick Start section and delve deeper as you require more control. For other aspects of configuration related to the JVM, licensing or use of SSL/TLS, refer to the Operating Guide.


Quick Start

In this section we will attempt to get you up and running very quickly, without getting into the various options to restrict the data being generated. This is covered in the later sections. We will take a lot of shortcuts which are undoubtedly not advisable in a production environment, but it is a good place to start.

Each IIB/MQ service to be monitored can be configured over these interfaces:

The labels in brackets are used to indicate the type of data that is to be extracted.

IIB V9 & MQ 7.5

We will start with an example IIB V9 & MQ 7.5 deployment. We will extract all available data and output it to splunk on the local host. This will simplify a number of configuration details, which can be further explored. We will also presume that these are the only services deployed and therefore the default ports will be used.

The configuration for this environment will be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Config>
    <Source host="localhost" name="DVNODE01" port="4414" type="IIBREST" interval="60"/>
    <Source host="localhost" name="DVNODE01-MQ" port="1414" type="MQ" channel="SYSTEM.DEF.SVRCONN" interval="60"/>
    <OutputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>
</Config>

We have cheated a little here but not too much. The main cheat is that we have used a pre-defined channel (SYSTEM.DEF.SVRCONN) to connect to MQ. This will need to be configured in order to get access to the data via MQ. You can use an existing channel (for example if one was set up to use MQ Explorer, which we would recommend), or create one by following this guide .

This configuration will also run the MQ RESET QUEUE STATS command, which may not be desirable if other monitoring services are running. You can switch this off however using the following configuration:

<?xml version="1.0" encoding="UTF-8"?>
<Config>
    <Source host="localhost" name="DVNODE01" port="4414" type="IIBREST" interval="60"/>
    <Source host="localhost" name="DVNODE01-MQ" port="1414" type="MQ" channel="SYSTEM.DEF.SVRCONN" interval="60">
        <Excludes>
            <Exclude>QSTATS</Exclude>
        </Excludes>
    </Source>
    <OutputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>
</Config>

IIB V10

This sample covers IIB V10 without MQ with output being sent to Elasticsearch. FLOWSTATS and RESOURCES data are acquired via the embedded MQTT server. All elements (IIB, Square Bubble and Elasticsearch) are deployed on the same host.

<?xml version="1.0" encoding="UTF-8"?>
<Config>
    <Source host="localhost" name="IIB10NODE01" port="4414" type="IIBREST" interval="60"/>
    <Source host="localhost" name="IIB10NODE01-MQTT" port= "11883" type="MQTT">
    <OutputDestination type="ELASTIC" host="localhost" port="9200">
</Config>

IIB V10 & MQ V8

This sample covers IIB V10 with MQ V8. FLOWSTATS and RESOURCES data can be acquired via either the embedded MQTT server or the associated MQ queue manager. In this example we will use the MQTT server otherwise we could use the IIB V9 & MQ 7.5 sample above. Again we will assume all elements are installed on the same host. The minimal configuration would be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Config>
    <Source host="localhost" name="IIB10NODE01" port="4414" type="IIBREST" interval="60"/>
    <Source host="localhost" name="IIB10NODE01-MQTT" port="11883" type="MQTT">
    <Source host="localhost" name="IIB10NODE01-MQ" port="1414" type="MQ"
        channel="SYSTEM.DEF.SVRCONN" interval="60">
        <Excludes>
            <Exclude>FLOWSTATS</Exclude>
            <Exclude>RESOURCES</Exclude>
        </Excludes>
    </Source>
    <OutputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>
</Config>

In this sample, we have to exclude the FLOWSTATS and RESOURCES data from the MQ source as this would otherwise be collected (as per the default).


Configuration Reference

This section, and the remainder of this page, is intended to be a reference for the configuration of Square Bubble for IIB & MQ. It introduces the schema of data items available and terms used to express which field(s) are to be included, excluded or subject to match filtering.

To configure Square Bubble for IIB & MQ, you need to create an XML document which defines the inputs and outputs that are required when running.

Square Bubble for IIB/MQ has options to allow the configuration to be validated (as covered in the Operating Guide) and tested in place to minimise possible configuration errors.

The used of XML is validated by a schema (XSD), which is fixed for the version of Square Bubble for IIB/MQ. The use of the schema will prevent a lot of common mistakes and typographic errors but it will not guarantee the data provided is valid, for example, a hostname may be provided which is valid syntactically but may not be correct. For this reason, a further level of validation is provided to ensure the semantic aspects of the configuration are valid.

The configuration includes 3 sections, which are:

  1. Common, which is an optional section used to simplify the configuration of many sources. This has an Includes and Excludes sub-sections to define the data to be included/excluded as appropriate. Note that excludes override includes.
  2. Source, defines one or more sources of data including the connections details (such as hostname and port) and the data to be included/excluded which is extended from the Common (should that be present).
  3. OutputDestination, which defines one or more destinations for the data. The format of the data (such as JSON) and the connection details are also recorded as well as the data from which source(s) they are to be taken from.

Only the latter 2 sections are required i.e. you need a minimum of one source and one destination.


Source Configuration

The Source configuration defines the source of data that will be collected by Square Bubble. It has the following attributes:

Attribute Name Description Required
host Host name or IP address where the service is running Yes
port Port the service is listening on Yes
name Name used to reference the service within the XML document. The value must be unique within the document Yes
type The interface type used to connect to the service. Can be one of MQ, MQTT or IIBREST Yes
channel The MQ channel used to connect to the MQ Queue Manager No, unless type is MQ
qmgr The name of the Queue Manager No
interval The interval between runs of the process to acquire data from the service. This is only used for Data Types that are acquired using a synchronous interface (see Data Types for more details) No, defaults to 30 seconds
username Authenticated user, for use with IIBREST or MQ No
password Authenticated users password, for use with IIBREST or MQ No
secure Connection make use of SSL/TLS No, defaults to false
cipher To be used in SSL/TLS connections to MQ. The values come from SSL/TLS CipherSpecs and CipherSuites in IBM MQ classes for Java, e.g. SSL_RSA_WITH_3DES_EDE_CBC_SHA (*) Yes if secure="true" and type="MQ"
securityexit Name of the security exit to be used in MQ connections as described in the IBM MQ documentation Assigning a channel exit in IBM MQ classes for Java No
brokerVersion Version of WMB/IIB that is present for an MQ Source which is collecting FLOWSTATS or RESOURCES. It only needs to be set for an MQ instance (Version 7.5 and above) connected to WMB 7 or 8. Otherwise it will default to the correct version. No, only applicable to type="MQ"

(*) the SSLv3 protocol and ciphers are not supported in the latest Java 7 and above versions or by Square Bubble for IIB/MQ.

For further details as to the use of SSL/TLS, refer to the Operating Guide

Includes and Excludes

This applies to both Common and Source sections.

Includes and Excludes are expressed as an optional set of data items with an optional set of filters. Data items can include either:

Omitting Includes means all available data types, which is the same as specifying ALL, and is applicable only for a Source. This is different to Common, where, the omission of Includes, is treated as an empty set.

Specifying just a data type implies all sub-types and fields available.

Excludes override Includes.

Filters

Filters are applicable only to fields. They can be matched to a value or a wildcarded value. Wildcards are denoted by asterisk (*) and can only be applied at the end of a value string. For example to match all queues whose name begins FLOW, the following filter would be used:

...
        <Filter>QUEUES.Q_NAME=FLOW*</Filter>
...

Filter matching can also be negated by using the exclamation mark (!) before the equal sign. As an example, to include queues whose name does not begin with SYSTEM., use the following:

...
        <Filter>QUEUES.Q_NAME!=SYSTEM.*</Filter>
...

Using filters in the include section defines expression(s) that must be matched in order to allow the data to be passed to an output destination. From Version 1.1.0, this has been changed to a logical OR operation, thus any of the expression may match rather than all must. This was added to allow multiple name filters to be used as this can reduce the load placed on MQ.

Using filters in the exclude section defines expression(s) that must not be matched in order to allow the data to be passed to an output destination. They are subject to a Negated logical AND (NAND) operation. And remember that excludes override includes.

Filters which contain invalid field names will be ignored, although a warning be logged.


Data Types

The following major data types are supported, and can be acquired using the listed protocol/interface type:

Data Type Description Protocol/Interface Type Synchronous Has sub-types
QMGR MQ Queue Manager status MQ Yes No
CONNECTIONS MQ Queue Manager Connections MQ Yes No
CHANNELS MQ Queue Manager Channels MQ Yes No
QUEUES MQ Queue Status MQ Yes No
QSTATS MQ Queue Statistics MQ Yes No
LISTENERS MQ Listeners Status MQ Yes No
SERVICES MQ Services Status MQ Yes No
BROKER IIB Broker/Node status IIBREST Yes Yes
RESOURCES IIB Resource statistics MQ MQTT No Yes
FLOWSTATS IIB Flowstats MQ MQTT No Yes

Square Bubble for IIB/MQ has the MQ, MQTT and IIBREST client libraries included. So there is no further configuration required other than the JRE (specifically the JVM) to be used to run in.

Sub-types

Sub-types enable the data within a data type to be further sub-divided. Use of sub-types will allow you to restrict the data output to only that which is required.

Sub-types are mixed case to differentiate themselves from the major data type.

The BROKER data type has the following sub-types:

The RESOURCES data type the following sub-types available:

FLOWSTATS also has sub-types available, which includes:

The list above is not definitive as IBM may remove or add items in future product versions. Some are only available on some platforms, such as Windows. This is why they are not validated rigorously. Instead, we enable the names to be used dynamically.

Fields

Fields are the lowest level of data item available. As with sub-types, they are not validated rigorously. Instead, we enable the names being dynamically used. This will enable different versions of the monitored software to be monitored without incurring excessive data coupling. The following tables list the fields available by type, be aware however that the names may be changed by IBM in future versions.

QMGR

The QMGR data type provides the MQ Queue Manager status and has the following fields:

CONNECTIONS

The connections data type has the following fields:

Be aware that these fields can vary between versions of MQ.

CHANNELS

The channels data type provides details of the MQ channels with the following fields:

QUEUES

The queues data type provides the status and details about MQ Queues. It includes the following fields:

QSTATS

Added in Version 1.1.0, the qstats data type provides statistical data about MQ Queues. It includes the following fields:

Be aware that the statistics are reset to zero once the call to retrieve this data has completed. Therefore, you should ensure only a single process should be performing this action to avoid data loss or corruption.

LISTENERS

The listeners data type provides the status and details about MQ listeners. It includes the following fields:

SERVICES

The services data type provides the status and details about MQ services. It includes the following fields:

BROKER

The broker data type provides the status of the elements within the IBM Integration Bus. It includes the following fields:

set fields
common type
runMode
name
type=Node sourceName
hostName
hostPort
AdminSecurity
version
shortDesc
longDesc
platformName
FixpackCapability
operationMode
platformArchitecture
platformVersion
queueManager
buildLevel
AdminAgentPID
type=Server uuid
isRunning
shortDesc
longDesc
processId
traceLevel
soapNodesUseEmbeddedListener
compiledXPathCacheSizeEntries
consoleMode
httpNodesUseEmbeddedListener
inactiveUserExitList
activeUserExitList
traceNodeLevel
userTraceLevel
node
type=Application version
uuid
isRunning
shortDesc
startMode
longDesc
defaultDotNetAppDomain
traceNodeLevel
traceLevel
userTraceLevel
modifyTime
deployTime
barFileName
node
server
type=MessageFlow uuid
commitCount
traceLevel
additionalInstances
startMode
coordinatedTransaction
commitInterval
version
isRunning
StatsSnapPublicationOn
StatsSnapOutputFormat
userTraceLevel
modifyTime
deployTime
barFileName
node
server
application

RESOURCES

The resources data type provides a wealth of information as provided by the IIB. Further details are provided in the IBM Documentation at Resource statistics data.

These are the fields available by sub-type:

sub-type Fields
common type (=sub-type)
name
BrokerLabel
ExecutionGroupName
timestamp
CICS RequestSuccess
RequestFailures
RequestSecurityFailures
ConnectionAttemptFailures
ConnectDirect

InboundTransfers
InboundBytes
NoHitTransfers
OutboundTransfers
OutboundBytes

CORBA OutboundInvocations
OutboundSuccessfulInvocations
OutboundCorbaExceptions
DecisionServices SuccessfulDecisions
FailedDecisions
RulesMatched
File

FilesRead
RecordsRead
BytesRead
FilesCreated
RecordsWritten
BytesWritten

FTEAgent

InboundTransfers
InboundBytes
OutboundTransfers
OutboundBytes

FTP

FTPGets
BytesReceived
FTPPuts
BytesSent
Protocol

GlobalCache

TotalMapActions
MapReads
MapWrites
MapRemoves
FailedActions
MapsUsed
ConnectionFailures
Connects

JDBCConnectionPools

NameOfJDBCProvider
MaxSizeOfPool
ActualSizeOfPool
CumulativeRequests
CumulativeDelayedRequests
CumulativeTimedOutRequests
MaxDelayInMilliseconds

JMS

NumberOfOpenJMSConnections
NumberOfClosedJMSConnections
NumberOfOpenJMSSessions
NumberOfClosedJMSSessions
NumberOfMessagesReceived
NumberOfMessagesSent
NumberOfMessagesBrowsed
NumberOfJMSConnectionFailures

JVM

InitialMemoryInMB
UsedMemoryInMB
CommittedMemoryInMB
MaxMemoryInMB
CumulativeGCTimeInSeconds
CumulativeNumberOfGCCollections

ODBC

ExecuteSuccess
ExecuteFailure
ActiveConnections
ClosedConnections
ConnectionErrors

Parsers

Threads
ApproxMemKB
MaxReadKB
MaxWrittenKB
Fields
Reads
FailedReads
Writes
FailedWrites

Security

TotalCacheEntries
TotalOperations
TotalSuccessfulOperations
TotalOperationsServicedByCache

Sockets

TotalSockets
TotalMessages
TotalDataSent_KB
TotalDataReceived_KB
SentMessageSize_0-1KB
SentMessageSize_1KB-10KB
SentMessageSize_10KB-100KB
SentMessageSize_100KB-1MB
SentMessageSize_1MB-10MB
SentMessageSize_Over10MB
ReceivedMessageSize_0-1KB
ReceivedMessageSize_1KB-10KB
ReceivedMessageSize_10KB-100KB
ReceivedMessageSize_100KB-1MB
ReceivedMessageSize_1MB-10MB
ReceivedMessageSize_Over10MB

SOAPInput

InboundMessagesTotal
RepliesSentTotal
InboundMessagesMadeFlow
InboundMessagesFaultedBeforeFlow
SuccessfulRepliesSent
FaultRepliesSent
PolicySetApplied

TCPIPClientNodes
TCPIPServerNodes

OpenConnections
ClosedConnections
MessagesReceived
MessagesSent
BytesReceived
BytesSent
FailedConnections (Client only)
FailedSSLConnections (Server only)

The lists were compiled using IIB V9 on a Linux platform. Be aware that the list may be subject to change between versions and deployment characteristics of the IBM Integration Bus.

FLOWSTATS

The FLOWSTATS data type provides the Message Flow Statistics and Accounting data as documented here, Message flow statistics and accounting data.

There are 3 subtypes associated with flowstats, and they have the following fields:

sub-type fields
(common to all) timestamp
type=sub-type
BrokerLabel
ExecutionGroupName
ApplicationName
MessageFlowName
TotalCPUTime
TotalElapsedTime
MessageFlow

BrokerUUID
ExecutionGroupUUID
StartDate
StartTime
GMTStartTime
EndDate
EndTime
GMTEndTime
MaximumElapsedTime
MinimumElapsedTime
MaximumCPUTime
MinimumCPUTime
CPUTimeWaitingForInputMessage
ElapsedTimeWaitingForInputMessage
TotalInputMessages
TotalSizeOfInputMessages
MaximumSizeOfInputMessages
MinimumSizeOfInputMessages
NumberOfThreadsInPool
TimesMaximumNumberOfThreadsReached
TotalNumberOfMQErrors
TotalNumberOfMessagesWithErrors
TotalNumberOfErrorsProcessingMessages
TotalNumberOfTimeOutsWaitingForRepliesToAggregateMessages
TotalNumberOfCommits
TotalNumberOfBackouts
AccountingOrigin

ThreadStatistics

Number
TotalNumberOfInputMessages
CPUTimeWaitingForInputMessage
ElapsedTimeWaitingForInputMessage
TotalSizeOfInputMessages
MaximumSizeOfInputMessages
MinimumSizeOfInputMessages

Nodes

Label
Type
MaximumElapsedTime
MinimumElapsedTime
MaximumCPUTime
MinimumCPUTime
CountOfInvocations
NumberOfInputTerminals
NumberOfOutputTerminals

The Node sub-type does have further details (Terminal Statistics) however we do not currently process them. If you require these further data fields, please contact us and we can discuss how to enable them.


Output Destinations

Data can be output to a number of destinations and can be taken from a number of sources. This is achieved by defining one or more OutputDestination sections in the XML. Each OutputDestination has the following attributes/child elements:

Attribute Description Default Required
type Type of output which is SPLUNK, SPLUNKHEC, ELASTIC or FILE none Yes
format Required output format JSON for FILE, ELASTIC and SPLUNKHEC, KEYVALUE for SPLUNK No
fromSource Is a list of sources that data will be taken from All sources No
port The TCP port used by the receiver none Yes for SPLUNK, SPLUNKHEC and ELASTIC
host The host name or IP address of the receiver none Yes for SPLUNK, SPLUNKHEC and ELASTIC
username The Username used to authenticate to the receiver none Optional, only supported for types ELASTIC, SPLUNK and SPLUNKHEC
password The Password used to authenticate to the receiver. none Optional, only supported for types ELASTIC, SPLUNK and SPLUNKHEC
secure The connection is made using TLS/SSL False Optional, only supported for types ELASTIC, SPLUNK and SPLUNKHEC
threads The number of threads used to send data to the destination. 3 for SPLUNK or SPLUNKHEC Optional, only supported for types SPLUNK and SPLUNKHEC
token The token used to authenticate to the Splunk HTTP Event Collector. none Yes if type is SPLUNKHEC

Note that a number of attributes are only used by specific types however they can be specified for others. In this situation they are ignored. For example attributes username and password are permitted for type FILE but are not used.

SPLUNKHEC (available from Version 1.2.0) requires a Splunk HTTP Event Collector to be set up to add data to the sqrbbl index (available once the Square Bubble Splunk app has been installed). This can also be used with HTTPS however you will need to acquire the appropriate certificate and add this to a truststore as defined in the Operating Guide. You Splunk administrator should be able to provide this for you.

The following child stanzas are also supported:

Child Description Default Required
Filename The filename to be used to receive data. The value also supports the use of substitution variables (see below) none Only for FILE type
OutputData A set of major data types to be output from the defined sources (in attribute fromSource above) All data No

At least one OutputDestination must be defined.

The optional OutputData section can be added, which is used to filter the available data by data type. This is only supported for major data type i.e. no further reduction of data is available here.. For Example, to send or write only BROKER data, you would include the following in your OutputDestination:

...
    <OutputDestination ...
        <OutputData>
            <Include>BROKER</Include>
        </OutputData>
    </OutputDestination>
...

Note this will naturally ignore all sources that do not provide BROKER data.

Filename substitution variables

To enable reuse of the configuration files in different environments, filenames may include variables that will be substituted at runtime to suit the target environment. Variables are denoted using the ${variablename} syntax. The variable values are taken from the java system properties. There are a number of system properties set up by default but you can add variable using the -D switch on the java command. Some of the standard system properties include:

A more complete list be can found here (for Oracle JVM).

In addition we have added a special variable ${date}, which will resolve to the date in YYYY-MM-dd format

As an example a filename could be defined as:

...
        <Filename>${java.io.tmpdir}${file.separator}${user.name}-iib-status-${date}.json</Filename>
...

which will resolve, for example on a linux platform using the sqrbbl user, to /tmp/sqrbbl-iib-status-2016-06-01.json.


Examples

Single Source using MQ

The simplest example is to acquire all available data from a single source. For example:

<?xml version="1.0" encoding="UTF-8"?>

<Config>

<Source name="DVNODE01" type="MQ" channel="SYSTEM.DEF.SVRCONN" host="localhost" port="1414" qmgr="DVQM01"/>

<OutputDestination type="SPLUNK" host="localhost" port="8089" username="admin" password="changeme">
</OutputDestination>

</Config>

This will acquire flowstats and resource data if enabled on the associated MQ queue manager

MQ filtered

There are a lot of data items you may wish to exclude such as queues that are called SYSTEM.*. The following example covers a number of filters to reduce the possible noise:

<?xml version="1.0" encoding="UTF-8"?>
<Config>

    <Source host="localhost" name="DVNODE01" port="4417" type="IIBREST" interval="60"/>

    <!-- MQ -->
    <Source host="localhost" name="DVNODE01-MQ" port="1417" type="MQ" channel="SYSTEM.DEF.SVRCONN" interval="60">
    <Excludes>
        <Exclude>CONNECTIONS.Q_MGR_UOW_ID</Exclude>
        <Exclude>CONNECTIONS.EXTERNAL_UOW_ID</Exclude>
        <Exclude>CONNECTIONS.UOW_LOG_EXTENT_NAME</Exclude>
        <Exclude>CONNECTIONS.UOW_LOG_START_DATE</Exclude>
        <Exclude>CONNECTIONS.MONITOR_KB</Exclude>
        <Exclude>QMGR.LOG_PATH</Exclude>
        <Exclude>QMGR.MEDIA_LOG_EXTENT_NAME</Exclude>
        <Exclude>QMGR.INSTALLATION_DESC</Exclude>
        <Exclude>QMGR.RESTART_LOG_EXTENT_NAME</Exclude>
        <Exclude>QMGR.CURRENT_LOG_EXTENT_NAME</Exclude>
        <Exclude>QUEUES.Q_STATUS_TYPE</Exclude>
        <Exclude>LISTENERS.MONITOR_KB</Exclude> <!-- PID on linux, of limited value -->
        <Filter>QUEUES.Q_NAME=SYSTEM.*</Filter>
        <Filter>QUEUES.Q_NAME=AMQ.*</Filter>
    </Excludes>
    </Source>

    <!-- First file gets all. -->
    <OutputDestination type="FILE">
        <FileName>/tmp/iib-All.json</FileName>
    </OutputDestination>

    <!-- Second file gets FLOWSTATS only -->
    <OutputDestination fromSource="DVNODE01-MQ" type="FILE">
        <FileName>/tmp/iib-flowstats.json</FileName>
        <OutputData>
            <Include>FLOWSTATS</Include>
        </OutputData>
    </OutputDestination>

</Config>

This example also shows FLOWSTATS data being written to both files and all data being written to the first file.

IIB V10 MQTT

You can acquire flowstats and resource data from the MQTT server (assuming it is configured to do so), which is only available from Version 10. For example:

<?xml version="1.0" encoding="UTF-8"?>
<Config>

<Source name="DVNODE01" type="MQTT" host="localhost" port="11884" />

<OutputDestination fromSource="DVNODE01" type="SPLUNK" host="localhost" port="8089" username="admin" password="changeme" />
</Config>
        

Note that the inclusion of RESOURCES and FLOWSTATS is implicit for an MQTT based node so they could have been omitted (assuming of course they were not excluded in the Common section).

IIB V9 & V10 REST

BROKER status data is available using the REST interface, and can be acquired as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Config>

<Source name="localhost" name="TESTNODE2-IIBREST" port="4415"
        type="IIBREST" interval="60" />

<OutputDestination fromSource="TESTNODE2-IIBREST" type="SPLUNKHEC" host="localhost" port="8088" token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" />
</Config>