Square Bubble Version 2 Configuration Guide

Introduction

Our goal for configuration is to make it both very flexible but also simple. We use XML as it can be validated readily and is explicit in its definition.

We also allow restricting the data to only that which is required thus reducing the load placed upon the network and associated processing chain.

Square Bubble Version 2 can also be deployed in a variety of different topologies ranging from being co-located with each monitored server or as a single instance monitoring a set of sources (what is often described as an agent-less topology).

This page is only concerned with the configuration which applies to the XML configuration file (usually located at /etc/sqrbbl/sqrbbl-config.xml). We recommend you start with the Quick Start section and delve deeper as you require more control. For other aspects of configuration related to the JVM, licensing or use of SSL/TLS, refer to the Operating Guide.

If you have an existing installation of Square Bubble for IIB & MQ (Version 1) and want to migrate your configuration to the Square Bubble version 2 format, follow the instructions in the Migration Guide


Quick Start

In this section we will attempt to get you up and running very quickly, without getting into the various options to restrict the data being generated. This is covered in the later sections. We will take a lot of shortcuts which are undoubtedly not advisable in a production environment, but it is a good place to start.

Datapower

DataPower is monitored over its REST or SOMA interfaces and can also be monitored using a syslog interface. As such, there are 2 source types that can be used:

This is an example DataPower deployment. We will extract all available data and output it to splunk on the local host. This will simplify a number of configuration details, which can be further explored. We will also presume that these are deployed locally (including the use of docker) and therefore the default ports will be used.

The configuration for this environment will be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <datapower host="datapower1" name="Datapower" port="5444" interval="60" username="a" password="b" />

    <syslog port="1515" host="datapower1" name="DatapowerSyslog" bindAddress="0.0.0.0" />

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

This configuration will connect to a DataPower appliance on host datapower1, port 5444, authenticating with the given user name and password over the REST interface. (See Source Type: datapower for a full list of attributes and data that can be gathered from a running DataPower instance). The information collected will then be sent to Splunk where it can be visualised with Square Bubble's dashboards. In addition Square Bubble can receive and process syslog data from DataPower. You will need to configure a logging target (with a target type syslog-tcp) to match the config (as above). Square Bubble is capable of collecting any syslog message, however, we recommend you include the following event subscriptions:

Category Minimum Priority Description of use
latency Information Used to provide enhanced latency data which is more comprehensible, per configured domain. See latency fields below
cert-monitor Critical Certificate monitor warnings, in particular when certificates are about to expire and when they have expired. Only required per device.
all Error Provides all errors generated by the domain.

IIB

IIB V10

This sample covers IIB V10 without MQ with output being sent to a file. FLOWSTATS and RESOURCES data are acquired via the embedded MQTT server. All elements (IIB, Square Bubble and File System) are deployed on the same host.

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <iibrest host="localhost" name="IIB10NODE01" port="4414" interval="60"/>

    <mqtt host="localhost" name="IIB10NODE01-MQTT" port= "11883"/>

    <outputDestination type="FILE">
        <fileName>/tmp/iib-data.json</fileName>
    </outputDestination>

</config>

IIB V10 & MQ V8-9

This sample covers IIB V10 with MQ V8-9. FLOWSTATS and RESOURCES data can be acquired via either the embedded MQTT server or the associated MQ queue manager. We will assume all elements are installed on the same host. The minimal configuration would be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <iibrest host="localhost" name="IIB10NODE01" port="4414" interval="60"/>

    <mq host="localhost" name="IIB10NODE01-MQ" port="1414"
        channel="SYSTEM.DEF.SVRCONN" interval="60">
    </mq>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

In this sample, we assume FLOWSTATS and RESOURCES data are being published to the MQ source. The RESET Q STATS command will also be run in this configuration which is not a recommended approach, instead you should be using Accounting, Statistics or System topics (MQ V9 only) to obtain this data.

MQ

MQ has both a synchronous and asynchronous means of collecting data, which we will cover separately, although they may be combined

Synchronous

This is a sample to retrieve data using synchronous requests:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mq host="localhost" name="DVQM01" port="1414" channel="SYSTEM.DEF.SVRCONN" interval="60">
        <excludes>
            <exclude>FLOWSTATS</exclude>
            <exclude>RESOURCES</exclude>
            <exclude>QSTATS</exclude>
        </excludes>
    </mq>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

This configuration will not provide the number of enqueues and dequeues per queue. This can be achieved by using System Topics (V9), Accounting messages, Statistics messages or using the RESET Q STATS command (QSTATS). The latter is NOT recommended.

Asynchronous IIB data

This is a sample to retrieve IIB related data using asynchronous requests:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mq host="localhost" name="DVQM01" port="1414" channel="SYSTEM.DEF.SVRCONN">
        <includes>
            <include>FLOWSTATS</include>
            <include>RESOURCES</include>
        </includes>
    </mq>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

Asynchronous MQ events

This is a sample to retrieve MQ event data using asynchronous requests from the default queue SYSTEM.ADMIN.QMGR.EVENT:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mq host="localhost" name="DVQM01" port="1414" channel="SYSTEM.DEF.SVRCONN">
        <includes>
            <include>MQEVENT</include>
        </includes>
        <adminQueue>SYSTEM.ADMIN.QMGR.EVENT</adminQueue>
    </mq>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

MQ Audit

This is a sample to retrieve auditable MQ events using asynchronous requests from a combined queue called SQRBBL.ADMIN.AUDIT.EVENT, which uses an administrative subscription:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mq host="localhost" name="DVQM01" port="1414" channel="SYSTEM.DEF.SVRCONN"
            uow="true" failWait="60000">
        <includes>
            <include>MQEVENT</include>
        </includes>
        <adminQueue>SQRBBL.ADMIN.AUDIT.EVENT</adminQueue>
    </mq>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

The use of a unit of work (uow) means that if there is a problem during the processing of the message (such as the output destination is not available), the process will backout and wait for a period of time (failWait, in milliseconds) before retrying.

MQ V9+ System topics

This is a sample to retrieve all monitoring data from System Topics using a pre-defined queue called SQRBBL.MQMON.ALL.QUEUE, which uses unmanaged subscriptions:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mq host="localhost" name="DVQM01" port="1414" channel="SYSTEM.DEF.SVRCONN">
        <includes>
            <include>MQMONALL</include>
        </includes>
        <mqmonQueue>SQRBBL.MQMON.ALL.QUEUE</mqmonQueue>
    </mq>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

Configuration Reference

This section, and the remainder of this page, is intended to be a reference for the configuration of Square Bubble Version 2. It introduces the schema of data items available and terms used to express which field(s) are to be included, excluded or subject to match filtering.

To configure Square Bubble, you need to create an XML document which defines the inputs and outputs that are required when running.

Square Bubble has options to allow the configuration to be validated (as covered in the Operating Guide ) and tested in place to minimise possible configuration errors.

The XML is validated by a schema (XSD), which is fixed for the version of Square Bubble. The use of the schema will prevent a lot of common mistakes and typographic errors but it will not guarantee the data provided is valid, for example, a hostname may be provided which is valid syntactically but may not be correct. For this reason, a further level of validation is provided to ensure the semantic aspects of the configuration are valid.

The configuration includes 2 sections, which are:

  1. Source, There are a number of types of sources (iibrest, mqtt, mq, datapower and syslog). A configuration must have at least one source. Sources define the data sources, including connection details and the data to be included/excluded.
  2. outputDestination: Defines one or more destinations for the data. The format of the data (such as JSON) and the connection details are also recorded as well as the data from which source(s) they are to be taken from.

You need a minimum of one source and one destination.


Source Configuration

The Source configuration defines the source of data that will be collected by Square Bubble. Each source type defines the interface that will be used to connect to a service.

The following source types are available to monitor:

Each source type has the ability to accept different data types. The data types provide a coarse grained, but easy way of selecting the information that you want to ingest. The sections below define all data types available to each source.

Sub-types, where used, enable the data within a data type to be further sub-divided. Use of sub-types will allow you to restrict the data output to only that which is required.

Fields are the lowest level of data item available to provide fine grained control over what data is output. As with sub-types, they are not validated rigorously. Instead, we enable the names to be dynamically used. This enables different versions of the monitored software to be monitored without incurring excessive data coupling.


Source Type: datapower

Attributes
Attribute Description Required
name Name used to reference the source. The value must be unique within the document Yes
host Hostname or IP of the Datapower device being monitored. This should be the IP/Host that Datapower's network interface is bound to. Yes
port Datapower's REST/SOMA management interface port. Yes
useSoap set to true for SOMA (XML Management Interface) otherwise the REST Management interface will be used No
interval The interval between runs of the process to acquire data from the service. No, defaults to 60 seconds
username Authenticated user No
password Authenticated users password No
Data Types

DEVICE represents DataPower's device-related statistics. It has the following sub-types and fields. Please note that these are meant as a guide and may change depending on the configuration and versions being used.

Sub-type Fields
Status AuditReserve, BackupMode, Description, EntitlementNumber, ProductID, ProductMode, ProductOID, SerialNumber, Services, SystemLogFixedFormat, SystemName, mAdminState, deviceName
CPU tenSeconds, oneMinute, tenMinutes, oneHour, oneDay, deviceName
Memory Usage, TotalMemory, UsedMemory, FreeMemory, ReqMemory, HoldMemory, ReservedMemory, InstalledMemory, deviceName
SystemUsage Interval, Load, WorkList, deviceName
Task TaskID, TaskName, Load, WorkList, CPU, Memory, FileCount, deviceName
Filesystem FreeEncrypted, TotalEncrypted, FreeTemporary, TotalTemporary, FreeInternal, TotalInternal, deviceName
TCP localIP, localPort, remoteIP, remotePort, state, serviceDomain, serviceClass, serviceName, deviceName

DOMAIN represents DataPower's domain-specific statistics. It has the following sub-types and fields. Please note that these are meant as a guide and may change depending on the configuration and versions being used.

Sub-type Fields
Status Domain, SaveNeeded, TraceEnabled, DebugEnabled, ProbeEnabled, DiagEnabled, CurrentCommand, QuiesceState, InterfaceState, FailsafeMode, deviceName
Object Class, OpState, AdminState, Name, EventCode, ErrorCode, ConfigState, deviceName
HTTP XMLManagervalue, XMLManagerhref, reqTenSec, reqOneMin, reqTenMin, reqOneHr, reqOneDay, reuseTenSec, reuseOneMin, reuseTenMin, reuseOneHr, reuseOneDay, createTenSec, createOneMin, createTenMin, createOneHr, createOneDay, returnTenSec, returnOneMin, returnTenMin, returnOneHr, returnOneDay, offerTenSec, offerOneMin, offerTenMin, offerOneHr, offerOneDay, destroyTenSec, destroyOneMin, destroyTenMin, destroyOneHr, destroyOneDay, deviceName
DocumentCache XMLManagervalue, XMLManagerhref, CacheCount, DocCount, CacheSizeKiB, KiByteCount, deviceName
StylesheetCahce XMLManagervalue, XMLManagerhref, CacheSize, CacheCount, ReadyCount, PendingCount, BadCount, DupCount, CacheKBCount, deviceName

The deviceName is propagated through all data types returned against the same device. This allows us to disambiguate data to a specific device.

Source Type: iibrest

This source represents the IIB Management REST interface.

Attributes
Attribute Name Description Required
host Host name or IP address where the service is running Yes
port Port the service is listening on Yes
name Name used to reference the source within the XML document. The value must be unique within the document Yes
interval The interval between runs of the process to acquire data from the service. No, defaults to 30 seconds
username Authenticated user No
password Authenticated users password No
secure Connection make use of SSL/TLS No, defaults to false
Data Types

BROKER represents the status data from the IIB Broker. The fields presented below

Sub-type Fields
Node AdminAgentPID, AdminSecurity, cacheTimeout, FixpackCapability, hostName, hostPort, listenerHost, longDesc, name, operationMode, platformArchitecture, platformName, platformVersion, portRange, queueManager, runMode, shortDesc, sourceName, version
Server activeUserExitList, broker, catalogClusterEndPoints, cbfEnabled, clientsDefaultToSSL, connectionEndPoints, consoleMode, coordinationQMgr, domainName, EISProviders, enableCatalogService, enableContainerService, enabled, enableJMX, failedMessageWaitTime, haManagerPort, httpNodesUseEmbeddedListener, httpPort, httpsPort, inactiveUserExitList, injectionMode, isRunning, jmxServicePort, jvmDebugPort, jvmMaxHeapSize, jvmMinHeapSize, jvmNativeStackSize, jvmSystemProperty, jvmVerboseOption, listenerHost, listenerPort, longDesc, name, processId, runMode, server, shortDesc, soapNodesUseEmbeddedListener, sslAlias, sslProtocol, testRecordMode, threadLocalProxyNameManagers, traceLevel, traceNodeLevel, userTraceLevel, uuid, validationThreads
Application barFileName, broker, defaultDotNetAppDomain, deployTime, isRunning, javaIsolation, longDesc, modifyTime, name, runMode, server, shortDesc, startMode, testRecordMode, traceLevel, traceNodeLevel, userTraceLevel, uuid, version
RESTAPI barFileName, baseURL, broker, defaultDotNetAppDomain, definitionsURL, deployTime, isRunning, javaIsolation, localBaseURL, localDefinitionsURL, longDesc, modifyTime, name, runMode, server, shortDesc, startMode, testRecordMode, traceLevel, traceNodeLevel, userTraceLevel, uuid, version
MessageFlow activeUserExitList, additionalInstances, barFileName, broker, commitCount, commitInterval, coordinatedTransaction, deployTime, inactiveUserExitList, isRunning, longDesc, maximumRateMsgsPerSec, modifyTime, name, restapi, runMode, server, shortDesc, startMode, testRecordMode, traceLevel, userTraceLevel, uuid, version

Source Type: mq

Attributes
Attribute Name Description Required
host Host name or IP address where the service is running Yes
port Port the service is listening on Yes
name Name used to reference the source within the XML document. The value must be unique within the document Yes
channel The MQ channel used to connect to the MQ Queue Manager Yes
qmgr The name of the Queue Manager, this will override the name attribute when connecting No
interval The interval between runs of the process to acquire data from the service. This is only used for Data Types that are acquired using a synchronous interface (see Data Types below for more details) No, defaults to 30 seconds
username Authenticated user No
password Authenticated users password No
secure Connection make use of SSL/TLS No, defaults to false
cipher To be used in SSL/TLS connections. The values come from SSL/TLS CipherSpecs and CipherSuites in IBM MQ classes for Java, e.g. SSL_RSA_WITH_3DES_EDE_CBC_SHA (*) Yes if secure="true"
securityexit Name of the security exit to be used in MQ connections as described in the IBM MQ documentation Assigning a channel exit in IBM MQ classes for Java No
brokerVersion Version of WMB/IIB that is present for an MQ Source which is collecting FLOWSTATS or RESOURCES. It only needs to be set for an MQ instance (Version 7.5 and above) connected to WMB 7 or 8. Otherwise it will default to the correct version. No
uow Is this source used as part of a unit of work? This is only used for MQ events, accounting and statistics. Default is "false". No
failWait Should an output fail, with an associate unit of work, this attribute defines how long to wait before retrying in millis. It is only appropriate where uow="true" and defaults to 200ms. No
mqmonManaged Applicable to only MQ V9+, it defines whether the queues, defined later in child element mqmonQueue, are managed subscriptions, defaults to false (which means unmanaged). No
iibSubScope (Introduced in version 2.0.1) defines the subscription scope used to acquire FLOWSTATS or RESOURCES messages, defaults to QMGR (which means only receive messages that have been posted to the connected queue manager). The only other valid value is ALL, use this with caution if the Queue Manager is included in a distributed publish and subscribe deployment. No

(*) the SSLv3 protocol and ciphers are not supported in the latest Java 7 and above versions or by Square Bubble Version 2.

For further details as to the use of SSL/TLS, refer to the Operating Guide

Child Elements
Child Name Description
adminQueue Defines the queue to be queried to extract MQEVENT data, which includes events, statistics and accounting data. For example to retrieve QMGR events, you could add SYSTEM.ADMIN.QMGR.EVENT.
mqmonQueue Defines the queue that acts as a subscription destination for MQMON* data, which is provided in the system admin topic from $SYS/MQ. This queue must be created beforehand and can be used as a destination for administratively managed subscriptions or unmanaged subscriptions. This is determined by the use of the mqmonManaged attribute on the mq source.
Data Types

The following data types are available from an mq source, note there are no sub-types listed in this table however MQEVENT and MQMON do have implicit subtypes, discussed below, and, FLOWSTATS and RESOURCES (which are really IIB types) are covered in the mqtt source section.

Data Type Fields
CHANNELS BUFFERS_RCVD, BUFFERS_SENT, BYTES_RCVD, BYTES_SENT, CHANNEL_INSTANCE_TYPE, CHANNEL_NAME, CHANNEL_START_DATE, CHANNEL_START_TIME, CHANNEL_STATUS, CHANNEL_SUBSTATE, CHANNEL_TYPE, COMPRESSION_RATE, COMPRESSION_TIME, CONNECTION_NAME, CURRENT_SHARING_CONVS, EXIT_TIME_INDICATOR, HB_INTERVAL, HDR_COMPRESSION, LAST_MSG_DATE, LAST_MSG_TIME, LOCAL_ADDRESS, MAX_SHARING_CONVS, MCA_JOB_NAME, MCA_STATUS, MCA_USER_ID, MONITORING_CHANNEL, MSG_COMPRESSION, MSGS, Q_MGR_NAME, REMOTE_APPL_TAG, REMOTE_VERSION, SSL_CERT_ISSUER_NAME, SSL_KEY_RESET_DATE, SSL_KEY_RESETS, SSL_KEY_RESET_TIME, SSL_SHORT_PEER_NAME, STOP_REQUESTED
CONNECTIONS APPL_DESC, APPL_TAG, APPL_TYPE, ASYNC_STATE, CHANNEL_NAME, CONNECTION_ID, CONNECTION_NAME, CONNECT_OPTIONS, CONN_INFO_TYPE, EXTERNAL_UOW_ID, GENERIC_CONNECTION_ID, PROCESS_ID, Q_MGR_NAME, Q_MGR_UOW_ID, THREAD_ID, UOW_LOG_EXTENT_NAME, UOW_LOG_START_DATE, UOW_LOG_START_TIME, UOW_START_DATE, UOW_START_TIME, UOW_TYPE, USER_IDENTIFIER
FLOWSTATS See FLOWSTATS in mqtt source type
LISTENERS IP_ADDRESS, LISTENER_CONTROL, LISTENER_DESC, LISTENER_NAME, LISTENER_START_DATE, LISTENER_START_TIME, LISTENER_STATUS, PORT, Q_MGR_NAME, XMIT_PROTOCOL_TYPE
MQEVENT (*) accountingToken, applName, applType, applicationIdData, applicationOriginData, backoutCount, characterSet, command, compCode, control, correlationId, encoding, expiry, feedback, groupId, messageFlags, messageId, messageSequenceNumber, messageType, originalLength, persistence, priority, putApplicationName, putApplicationType, putDateTime, reason, replyToQueueManagerName, replyToQueueName, report, srcQMgr, userId, version
MQMON* (*) accountingToken, applName, applType, applicationIdData, applicationOriginData, backoutCount, characterSet, command, compCode, control, correlationId, encoding, expiry, feedback, groupId, messageFlags, messageId, messageSequenceNumber, messageType, monitorClass, monitorType, originalLength, persistence, priority, putApplicationName, putApplicationType, putDateTime, reason, replyToQueueManagerName, replyToQueueName, report, srcQMgr, userId, version
QMGR sourceName, hostName, hostPort, CHINIT_STATUS, CMD_SERVER_STATUS, CONNECTION_COUNT, CURRENT_LOG_EXTENT_NAME, INSTALLATION_DESC, INSTALLATION_NAME, INSTALLATION_PATH, LOG_PATH, MEDIA_LOG_EXTENT_NAME, PERMIT_STANDBY, Q_MGR_NAME, Q_MGR_START_DATE, Q_MGR_STATUS, RESTART_LOG_EXTENT_NAME
QSTATS (**) HIGH_Q_DEPTH, MSG_DEQ_COUNT, MSG_ENQ_COUNT, Q_MGR_NAME, Q_NAME, TIME_SINCE_RESET
QUEUES CURRENT_Q_DEPTH, LAST_GET_DATE, LAST_GET_TIME, LAST_PUT_DATE, LAST_PUT_TIME, MEDIA_LOG_EXTENT_NAME, MONITORING_Q, OLDEST_MSG_AGE, OPEN_INPUT_COUNT, OPEN_OUTPUT_COUNT, Q_MGR_NAME, Q_NAME, Q_STATUS_TYPE, Q_TIME_INDICATOR1, Q_TIME_INDICATOR2
RESOURCES See mqtt source type
SERVICES ALTERATION_DATE, Q_MGR_NAME, SERVICE_CONTROL, SERVICE_DESC, SERVICE_NAME, SERVICE_START_ARGS, SERVICE_START_COMMAND, SERVICE_STATUS, SERVICE_STOP_ARGS, SERVICE_STOP_COMMAND, SERVICE_TYPE, STDERR_DESTINATION, STDOUT_DESTINATION

(*) The MQEVENT and MQMON* data types have subtypes and additional fields depending upon their internal type and class where available. The sections below provide further details...

(**) Use of QSTATS is heavily discouraged by IBM as only a single program should make acceptable use of this service. Instead, the accounting and statistics events provide (from data types MQEVENT or MQMON*) much richer data and can be shared.

MQEVENT data type

The MQEVENT data type has subtypes identified in the command field (with further details provided by the reason field). Sub-types include (but are not limited to)

  1. Event types (See also Event Types and Controlling events):
    1. Q_MGR_EVENT, Queue manager event,
    2. PERFM_EVENT, Performance event,
    3. CHANNEL_EVENT, Channel event,
    4. CONFIG_EVENT, Configuration event (which may include a generated configuration change summary, see below for more details),
    5. COMMAND_EVENT, Command event, and
    6. LOGGER_EVENT, Logger event.
  2. Accounting types (See also Accounting messages):
    1. ACCOUNTING_MQI, MQI accounting messages contain information relating to the number of MQI calls made using a connection to a queue manager, and
    2. ACCOUNTING_Q, Queue accounting messages contain information relating to the number of MQI calls made using connections to a queue manager, by queue.
  3. Statistics types (See also Statistics messages):
    1. STATISTICS_CHANNEL Channel statistics messages contain information relating to the activity of a channel during a configured interval,
    2. STATISTICS_MQI, MQI statistics messages contain information relating to the number of MQI calls made during a configured interval, and
    3. STATISTICS_Q, Queue statistics messages contain information relating to the activity of a queue during a configured interval.

Unless you are familiar with the specific event types, you should consider collecting some sample data to give you some familiarity with the available fields.

For optimal performance, the configuration of MQ should be used to control which events are generated (as described in Controlling events), in preference to using includes/excludes or filtering in Square Bubble. You may wish to add field excludes however to cut down on the data provided.

For CONFIG_EVENT messages that pertain to a configuration change, we will, where detected, generate a change summary based on the before and after state of the object being changed. This message will have the sub type CONFIG_EVENT

MQMON* data type

The MQMON* data type has subtypes identified in the monitorType field.

The data message formats from MQ V9+ System Topics (as described in System topics for monitoring and activity trace) are not explicitly documented. Instead they are described in meta data which we extract from the queue manager and generate data according to this definition. This means we have to harvest this data for each mq source applicable. Including a MQMON* specification, or including a mqmonQueue child element, will trigger this behaviour.

Include MQMONALL if you want all data otherwise define the class with the prefix MQMON such as MQMONDISK if you want to restrict to only seeing data of monitor class DISK. Each monitor class has a set of monitor types which can be added as subtypes. Finally fields are defined (in monitor elements meta data) and can be included, excluded or filtered. The dynamic nature of this interface suggests that you should try extracting the data, perhaps to a file, and use that to help decide which monitor classes, types and fields you wish to store in a production environment.

Source Type: mqtt

This source represents the IIB embedded MQTT server to acquire flowstats and resources data.

Attributes
Attribute Name Description Required
host Host name or IP address where the service is running Yes
port Port the service is listening on Yes
name Name used to reference the source within the XML document. The value must be unique within the document Yes
username Authenticated user No
password Authenticated users password No
Data Types
FLOWSTATS

Represents the flowstats data from the IIB Broker. The fields presented below may vary depending upon the options used when running the IIB service and the version of the product being used.

Sub-type Fields
MessageFlow AccountingOrigin, ApplicationName, BrokerLabel, BrokerUUID, CPUTimeWaitingForInputMessage, ElapsedTimeWaitingForInputMessage, EndDate, EndTime, ExecutionGroupName, ExecutionGroupUUID, GMTEndTime, GMTStartTime, MaximumCPUTime, MaximumElapsedTime, MaximumSizeOfInputMessages, MessageFlowName, MinimumCPUTime, MinimumElapsedTime, MinimumSizeOfInputMessages, NumberOfThreadsInPool, RestApiName, StartDate, StartTime, TimesMaximumNumberOfThreadsReached, TotalCPUTime, TotalElapsedTime, TotalInputMessages, TotalNumberOfBackouts, TotalNumberOfCommits, TotalNumberOfErrorsProcessingMessages, TotalNumberOfMessagesWithErrors, TotalNumberOfMQErrors, TotalNumberOfTimeOutsWaitingForRepliesToAggregateMessages, TotalSizeOfInputMessages
Nodes ApplicationName, BrokerLabel, CountOfInvocations, ExecutionGroupName, Label, MaximumCPUTime, MaximumElapsedTime, MessageFlowName, MinimumCPUTime, MinimumElapsedTime, NumberOfInputTerminals, NumberOfOutputTerminals, RestApiName, TotalCPUTime, TotalElapsedTime, Type
ThreadStatistics * ApplicationName, BrokerLabel, CPUTimeWaitingForInputMessage, ElapsedTimeWaitingForInputMessage, ExecutionGroupName, MaximumSizeOfInputMessages, MessageFlowName, MinimumSizeOfInputMessages, Number, RestApiName, TotalCPUTime, TotalElapsedTime, TotalNumberOfInputMessages, TotalSizeOfInputMessages

* There are currently no dashboards that display ThreadStatistics, so you can exclude this subtype without any impact on the shipped dashboards.

RESOURCES

Represents the resources reported by the IIB Broker. The fields presented below may vary depending upon the options used when running the IIB service and the version of the product being used.

Sub-type Fields
common type (=sub-type), name, BrokerLabel, ExecutionGroupName, timestamp
CICS RequestSuccess, RequestFailures, RequestSecurityFailures, ConnectionAttemptFailures
ConnectDirect InboundTransfers, InboundBytes, NoHitTransfers, OutboundTransfers, OutboundBytes
CORBA OutboundInvocations, OutboundSuccessfulInvocations, OutboundCorbaExceptions
DecisionServices SuccessfulDecisions, FailedDecisions, RulesMatched
File FilesRead, RecordsRead, BytesRead, FilesCreated, RecordsWritten, BytesWritten
FTEAgent InboundTransfers, InboundBytes, OutboundTransfers, OutboundBytes
FTP FTPGets, BytesReceived, FTPPuts, BytesSent, Protocol
GlobalCache TotalMapActions, MapReads, MapWrites, MapRemoves, FailedActions, MapsUsed, ConnectionFailures, Connects
JDBCConnectionPools NameOfJDBCProvider, MaxSizeOfPool, ActualSizeOfPool, CumulativeRequests, CumulativeDelayedRequests, CumulativeTimedOutRequests, MaxDelayInMilliseconds
JMS NumberOfOpenJMSConnections, NumberOfClosedJMSConnections, NumberOfOpenJMSSessions, NumberOfClosedJMSSessions, NumberOfMessagesReceived, NumberOfMessagesSent, NumberOfMessagesBrowsed, NumberOfJMSConnectionFailures
JVM (For Memory related data) InitialMemoryInMB, UsedMemoryInMB, CommittedMemoryInMB, MaxMemoryInMB
(For GC related data) CumulativeGCTimeInSeconds, CumulativeNumberOfGCCollections
ODBC ExecuteSuccess, ExecuteFailure, ActiveConnections, ClosedConnections, ConnectionErrors
Parsers Threads, ApproxMemKB, MaxReadKB, MaxWrittenKB, Fields, Reads, FailedReads, Writes, FailedWrites
Security TotalCacheEntries, TotalOperations, TotalSuccessfulOperations, TotalOperationsServicedByCache
Sockets TotalSockets, TotalMessages, TotalDataSent_KB, TotalDataReceived_KB, SentMessageSize_0-1KB, SentMessageSize_1KB-10KB, SentMessageSize_10KB-100KB, SentMessageSize_100KB-1MB, SentMessageSize_1MB-10MB, SentMessageSize_Over10MB, ReceivedMessageSize_0-1KB, ReceivedMessageSize_1KB-10KB, ReceivedMessageSize_10KB-100KB, ReceivedMessageSize_100KB-1MB, ReceivedMessageSize_1MB-10MB, ReceivedMessageSize_Over10MB
SOAPInput InboundMessagesTotal, RepliesSentTotal, InboundMessagesMadeFlow, InboundMessagesFaultedBeforeFlow, SuccessfulRepliesSent, FaultRepliesSent, PolicySetApplied
TCPIPClientNodes, TCPIPServerNodes OpenConnections, ClosedConnections, MessagesReceived, MessagesSent, BytesReceived, BytesSent, FailedConnections (Client only), FailedSSLConnections (Server only)

Source Type: syslog

Attributes
Attribute Description Required
host IP address where the syslog messages will come from. Syslog messages from other IPs wil be discarded. Yes
port Port that the syslog server will bind to. Yes
name Name used to reference the service. The value must be unique within the document Yes
bindAddress Local address where the syslog server will bind to, defaults to 0.0.0.0 (all available network interfaces). No
Data Types

SYSLOG represents DataPower's syslog messages

Message Fields
All level, type, message, deviceName, messageTypeId, host, gw-host, gtid, trans, domain
Latency as for All, plus the endpoint, and:
  • clientRequest, time taken for the client to send the request
  • fshProcessing, time taken for the Front Side Handling/Processing
  • beConnect, time taken to connect to the back end
  • beRequest, time taken to send the request to the back end
  • beProcessing, time taken by the back end processing the request
  • beResponse, time taken to send the response from the back end
  • bshProcessing, time taken for the Back Side Handling/Processing
  • clientResponse, time take to send the client response
Certificate Monitor as for All, plus certName, expiryDate, certStatus

The deviceName is propagated through all data types returned against the same device. This allows us to disambiguate data to a specific device. The device name is taken from the local identifier field specified on the log target. We strongly recommend you set this to the name returned by the datapower source, which is the System (or Appliance) Name as displayed in System Settings.


Includes / Excludes

This applies to all sources except SYSLOG. includes and excludes are expressed as an optional set of data items and an optional set of filters. Data items can be defined as either:

Omitting includes means all available default data types, which is the same as specifying ALL, and is applicable only to sources. Specifying just a data type implies all sub-types and fields available. It important to be aware that excludes override includes.

Each data type will also have a mandatory set of fields which can not be excluded, this includes:

The reason for this is that some fields are required to define the context in which the data is provided. For example, providing details about a Message Flow only makes sense if we also deifne the IIB Node and Server to which the instance applies, otherwise it would have limited value.

Use of includes should be used for Major and sub-types, only, where known. There is a danger when using an include of a typographic error which may result in data not being generated. Whilst we attempt to validate the entries, fields (and some sub types), are only validated at run time, and whilst we can warn you that data has not been matched, it is typically too late. Excludes on the other hand only have the negative effect of including data you don't want.

Also when you use an include (within includes section), you must define all include values that you require, as this overrides the default behaviour. For example, if you only include RESOURCES.JVM in a mqtt source, this will prevent all other subtypes being generated and other major types (in this case FLOWSTATS). So if you wanted JVM information and FLOWSTATS you must include both, or exclude all other RESOURCES sub-types with a default includes. Filters, however do not share this behaviour, they have no effect on include or exclude definitions, further discussed below.

Filters

Filters are defined only against fields but apply to the whole entity they are taken from. They can be matched to a value or a wildcarded value. Wildcards are denoted by asterisk ( *) and can only be applied at the end of a value string. For example to match all Domain objects whose class name begins W (such as WSGateway, WebAppFW etc), the following filter would be used:

<filter>DOMAIN.Object.Class=W*</filter>

Filter matching can also be negated by using the exclamation mark (!) before the equal sign. As an example, to include domain object data whose class does not begin with Log, use the following:

<filter>DOMAIN.Object.Class!=Log*</filter>

Using filters in the includes section defines expression(s) that must be matched in order to allow the data to be passed to an output destination. If more than one filter expression is present, at least one expression must match (logical OR).

Using filters in the excludes section defines expression(s) that must not be matched in order to allow the data to be passed to an output destination. They are subject to a Negated logical AND (NAND) operation. Here again, exclude filters override include filters.

Filters which contain invalid field names will be ignored, and a warning may be logged.


Output Destinations

Tag: outputDestination

Data can be output to a number of destinations and can be taken from a number of sources. This is achieved by defining one or more OutputDestination sections in the XML. Each outputDestination has the following attributes/child elements:

Attribute Description Default Required
type Type of output which is SPLUNK, SPLUNKHEC, ELASTIC or FILE none Yes
format Required output format JSON for FILE, ELASTIC and SPLUNKHEC, KEYVALUE for SPLUNK No
fromSource Is a list of sources that data will be taken from All sources No
port The TCP port used by the receiver none Yes for ELASTIC, SPLUNK or SPLUNKHEC
host The host name or IP address of the receiver none Yes for ELASTIC, SPLUNK or SPLUNKHEC
username The Username used to authenticate to the receiver none Optional, only supported for types ELASTIC, SPLUNK and SPLUNKHEC
password The Password used to authenticate to the receiver. none Optional, only supported for typesELASTIC, SPLUNK and SPLUNKHEC
secure The connection is made using TLS/SSL False Optional, only supported for typesELASTIC, SPLUNK and SPLUNKHEC
threads The number of threads used to send data to the destination. 3 for SPLUNK or SPLUNKHEC Optional, only supported for types SPLUNK and SPLUNKHEC
token The token used to authenticate to the Splunk HTTP Event Collector. none Yes if type is SPLUNKHEC

Note that a number of attributes are only used by specific types however they can be specified for others. In this situation they are ignored. For example attributes username and password are permitted for type FILE but are not used.

SPLUNKHEC requires a Splunk HTTP Event Collector to be set up to add data to the sqrbbl index (available once the Square Bubble Splunk app has been installed). This can also be used with HTTPS however you will need to acquire the appropriate certificate and add this to a truststore as defined in the Operating Guide. Your Splunk administrator should be able to provide this for you.

The following child elements are also supported:

Child Description Default Required
filename The filename to be used to receive data. The value also supports the use of substitution variables (see below) none Only for FILE type
outputData A set of major data types to be output from the defined sources (in attribute fromSource above) All data No

At least one OutputDestination must be defined.

The optional OutputData section can be added, which is used to filter the available data by data type. This is only supported for major data type i.e. no further reduction of data is available here.. For Example, to send or write only DOMAIN data, you would include the following in your outputDestination:

...
    <outputDestination ...
        <outputData>
            <include>DOMAIN</include>
        </outputData>
    </outputDestination>
...

Note this will naturally ignore all sources that do not provide DOMAIN data.

Filename substitution variables

To enable reuse of the configuration files in different environments, filenames may include variables that will be substituted at runtime to suit the target environment. Variables are denoted using the ${variablename} syntax. The variable values are taken from the java system properties. There are a number of system properties set up by default but you can add variable using the -D switch on the java command. Some of the standard system properties include:

A more complete list be can found here (for Oracle JVM).

In addition we have added a special variable ${date}, which will resolve to the date in YYYY-MM-dd format

As an example a filename could be defined as:

...
        <filename>${java.io.tmpdir}${file.separator}${user.name}-datapower-${date}.json</filename>
...

which will resolve, for example on a linux platform using the sqrbbl user, to /tmp/sqrbbl-datapower-2016-06-01.json.


Examples

Single Source DataPower

The simplest example is to acquire all available data from a single source. For example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <datapower host="localhost" name="Datapower" port="5444" interval="60" username="a" password="b" />

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

This will acquire device and domain data and send it to splunk.

Datapower & Syslog

In the following configuration we combine DataPower info from the REST management interface with data from syslog.

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <datapower host="localhost" name="Datapower" port="5444" interval="60" username="a" password="b" />

    <syslog host="localhost" name="Syslog" port="1443" bindAddress="0.0.0.0" />

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

DataPower filtered

There are a lot of data items you may wish to exclude from datapower such as cache information and objects related to logging. The following example covers a number of excludes to reduce the possible noise:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <datapower host="localhost" name="Datapower" port="5444" interval="60" username="a" password="b" >
        <excludes>
            <exclude>DOMAIN.DocumentCache</exclude>
            <exclude>DOMAIN.StylesheetCache</exclude>
            <filter>DOMAIN.Object.Class=Log*</filter>
        </excludes>
    </datapower>

    <outputDestination type="SPLUNK" username="admin" password="changeme" host="localhost" port="8089"/>

</config>

This configuration ingests all DataPower information, but excludes stats for the document cache and the stylesheet cache for each domain. It also discards the object data for all object whose Class begins with "Log", such as "LogTarget" or "LogLabel".


Single Source using MQ

The simplest example is to acquire all available data from a single source. For example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mq name="DVNODE01" channel="SYSTEM.DEF.SVRCONN" host="localhost" port="1414" qmgr="DVQM01"/>

    <outputDestination type="SPLUNK" host="localhost" port="8089" username="admin" password="changeme"/>

</config>

This will acquire flowstats and resource data if enabled on the associated MQ queue manager

MQ filtered

There are a lot of data items you may wish to include such as queues whose names begin with MYAPP1 oy MYAPP2. The following example covers a number of filters to reduce the possible noise:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <iibrest host="localhost" name="DVNODE01" port="4414" interval="60"/>

    <!-- MQ -->
    <mq host="localhost" name="DVQM01" port="1414" channel="SYSTEM.DEF.SVRCONN" interval="60">
        <includes>
            <filter>QUEUES.Q_NAME=MYAPP1*</filter>
            <filter>QUEUES.Q_NAME=MYAPP2*</filter>
        </includes>
        <excludes>
            <exclude>CONNECTIONS.Q_MGR_UOW_ID</exclude>
            <exclude>CONNECTIONS.EXTERNAL_UOW_ID</exclude>
            <exclude>CONNECTIONS.UOW_LOG_EXTENT_NAME</exclude>
            <exclude>CONNECTIONS.UOW_LOG_START_DATE</exclude>
            <exclude>CONNECTIONS.MONITOR_KB</exclude>
            <exclude>QMGR.LOG_PATH</exclude>
            <exclude>QMGR.MEDIA_LOG_EXTENT_NAME</exclude>
            <exclude>QMGR.INSTALLATION_DESC</exclude>
            <exclude>QMGR.RESTART_LOG_EXTENT_NAME</exclude>
            <exclude>QMGR.CURRENT_LOG_EXTENT_NAME</exclude>
            <exclude>QUEUES.Q_STATUS_TYPE</exclude>
            <exclude>LISTENERS.MONITOR_KB</exclude>
        </excludes>
    </mq>

    <!-- First file gets all. -->
    <outputDestination type="FILE">
        <fileName>/tmp/iib-All.json</fileName>
    </outputDestination>

    <!-- Second file gets FLOWSTATS only -->
    <outputDestination fromSource="DVNODE01-MQ" type="FILE">
        <fileName>/tmp/iib-flowstats.json</fileName>
        <outputData>
            <include>FLOWSTATS</include>
        </outputData>
    </outputDestination>

</config>

Using the QUEUES.Q_NAME filter in the includes will result in more calls to MQ (1 per filter), however, the amount of data retrieved will be reduced. MQ has lots of pre-defined queues which may not be of interest.

This example also shows FLOWSTATS data being written to both files and all data being written to the first file.

IIB V10 MQTT

You can acquire flowstats and resource data from the MQTT server (assuming it is configured to do so), which is only available from Version 10. For example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mqtt name="DVNODE01" host="localhost" port="11883" />

    <outputDestination fromSource="DVNODE01" type="SPLUNK" host="localhost" port="8089" username="admin" password="changeme" />

</config>

Note that the inclusion of RESOURCES and FLOWSTATS data is implicit for an MQTT based node so they are omitted.

IIB V10 MQTT filtered

Not all FLOWSTATS and RESOURCES data from the MQTT (or MQ) server will be of interest to you so you can filter the unwanted data out. ThreadStatictics data is not used in our provided dashboards and Nodes data can be very large even when no invocations are made, so we can filter them out. Also RESOURCES data can be limited to only the types required. For example:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <mqtt name="DVNODE01" host="localhost" port="11883">
        <includes>
            <include>FLOWSTATS</include>
            <include>RESOURCES.JVM</include>
            <include>RESOURCES.SOAPInput</include>
            <include>RESOURCES.DecisionServices</include>
            <include>RESOURCES.Parsers</include>
            <include>RESOURCES.Security</include>
            <include>RESOURCES.JMS</include>
            <filter>FLOWSTATS.Nodes.CountOfInvocations!=0</filter>
        </includes>
        <excludes>
            <exclude>FLOWSTATS.ThreadStatictics</exclude>
        </excludes>
    </mqtt>

    <outputDestination fromSource="DVNODE01" type="SPLUNK" host="localhost" port="8089" username="admin" password="changeme" />

</config>

Note that the includes are overridden by excludes.

IIB V9 & V10 runMode

BROKER status data is available using the REST interface, the bulk of the dashboards are concerned only with the runMode field. This can be acquired as follows:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://innovations.syntegrity.com/sqrbbl/generated/config2"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <iibrest name="localhost" name="TESTNODE2" port="4415"
            interval="60">
        <includes>
            <include>BROKER.Node.runMode</include>
            <include>BROKER.Server.runMode</include>
            <include>BROKER.Application.runMode</include>
            <include>BROKER.RESTAPI.runMode</include>
            <include>BROKER.MessageFlow.runMode</include>
        </includes>
    </iibrest>

    <outputDestination type="SPLUNKHEC" host="localhost" port="8088" token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" />

</config>

Only the runMode attribute will be included from the available fields by BROKER sub-type.