Nagios, Log4J, Cloud Watch – examples within a project

15th December 2017

Introduction

A typical Client Application system utilizes a number of monitoring and notification services in order to discover and resolve infrastructure and application problems before they affect business processes.

 

The following utilities are commonly used and can be considered best of breed:

  • Nagios – monitoring of the health of the server environment,
  • Log4j – run-time application monitoring
  • AWS monitoring utilities

 

1. Nagios

Nagios is a powerful, open-source application that monitors systems and infrastructure. Nagios is configured to monitor the health of the server environment. Notification emails are sent in case of problems. Example – following tests:

  1. Custom Nagios plug-in to check availability of Client Application sync service.

Time interval:  Each minute

The plug-in “pings” Client Application sync service. The plug-in returns one of the following codes:

  • OK: exit code 0: indicates a service is working properly,
  • WARNING: exit code 1: indicates a service is in warning state,
  • CRITICAL: exit code 2: indicates a service is in critical state,
  • UNKNOWN: exit code 3: indicates a service is in unknown state.

Nagios can integrate with Cloudwatch and pull in CloudWatch logs to the Nagios dashboard (Nagios XI dashboard is extremely impressive and easy to set up and customize, with many APIs available to ingest logs from other sources).

2. Log4j

Apache Log4j is a powerful Java based logging utility.

Key benefits of Log4j:

  • Straightforward configuration via xml file and appenders,
  • Ability to log application output to various destinations such as console, database, text file,
  • Different logging levels can be set: TRACE, DEBUG, INFO, WARN, ERROR and FATAL,
  • Ability to send email notifications,
  • Ability to integrate with third party systems such as Splunk (www.splunk.com) or AWS CloudWatch (www.aws.amazon.com/cloudwatch),
  • Ability to write custom logging events filters

A typical Client Application can utilise Log4j version 1.2 for logging to monitor:

  • Sync services,
  • Administration and Managers portals,
  • Client Application – Java SWING based client for eg.

Log4j version 2 is used in REST services.

    2.1 Log4j configuration details
     2.1.1 Client Application Java SWING based client

Configuration file (log4j 1.2):  EXAMPLE

<?xml version=”1.0″ encoding=”UTF-8″ ?>

<!DOCTYPE log4j:configuration SYSTEM “log4j.dtd”>

<log4j:configuration>

 

<appender name=”stdout”  class=”org.apache.log4j.ConsoleAppender”>

<param name=”Threshold” value=”INFO” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d [%t] %-5p %c{1} – %m%n” />

</layout>

</appender>

<appender name=”INFO”  class=”org.apache.log4j.ConsoleAppender”>

<param name=”Target” value=”System.out” />

<param name=”Threshold” value=”INFO” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d %-5p %c – %m%n” />

</layout>

</appender>

 

<appender name=”R”  class=”org.apache.log4j.RollingFileAppender”>

<param name=”File” value=”bunzl-fc.log” />

<param name=”MaxFileSize” value=”5MB” />

<param name=”MaxBackupIndex” value=”2″ />

<param name=”BufferSize” value=”512″ />

<param name=”Threshold” value=”INFO” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d [%t] %-5p %c – %m%n” />

</layout>

<filter class=”com.xxxxx.client.database.logging.DbStatsLog4jFilter”>

<param name=”SuppressEqualsMessages” value=”false”/>

<param name=”WriteStatistics” value=”false”/>

</filter>

</appender>

 

<appender name=”S”  class=”org.apache.log4j.RollingFileAppender”>

<param name=”File” value=”db_stats.log” />

<param name=”MaxFileSize” value=”5MB” />

<param name=”MaxBackupIndex” value=”2″ />

<param name=”BufferSize” value=”512″ />

<param name=”Threshold” value=”INFO” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d [%t] %-5p %c – %m%n” />

</layout>

<filter class=”com.xxxxx.client.database.logging.DbStatsLog4jFilter”>

<param name=”SuppressEqualsMessages” value=”true”/>

<param name=”WriteStatistics” value=”true”/>

</filter>

</appender>

<!–                           –>

<!– setup log4j’s root logger –>

<!–                           –>

<root>

<level value=”all” />

<appender-ref ref=”stdout”/>

<appender-ref ref=”INFO”/>

<appender-ref ref=”R”/>

<appender-ref ref=”S”/>

 

</root>

</log4j:configuration>

 

 

Existing Log4j appenders:

  • Console appender,
  • Rolling file appender to log application output. Rolling file appender automatically archives log files when they reach certain size and start logging to a new file.

Configuration:

  • Max file size: 5MB
  • Max files to archive: 2
  • Log4j logging level threshold: INFO
  • Rolling file appender to log DB stats. This appender uses custom filter named DbStatsLog4jFilter which prevents from logging same information within short period of time.
            2.1.2 Sync service and portals

Configuration file (log4j 1.2): EXAMPLE

<?xml version=”1.0″ encoding=”UTF-8″ ?>

<!DOCTYPE log4j:configuration SYSTEM “log4j.dtd”>

<log4j:configuration>

<appender name=”stdout”  class=”org.apache.log4j.ConsoleAppender”>

<param name=”Threshold” value=”INFO” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d %-5p %c – %m%n” />

</layout>

</appender>

<appender name=”INFO”  class=”org.apache.log4j.ConsoleAppender”>

<param name=”Target” value=”System.out” />

<param name=”Threshold” value=”INFO” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d %-5p %c – %m%n” />

</layout>

</appender>

<appender name=”R”  class=”org.apache.log4j.RollingFileAppender”>

<param name=”Threshold” value=”INFO” />

<param name=”File” value=”application.log” />

<param name=”MaxFileSize” value=”5MB” />

<param name=”MaxBackupIndex” value=”10″ />

<param name=”BufferSize” value=”512″ />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d %p – %m%n” />

</layout>

</appender>

<appender name=”LoginAppender”  class=”org.apache.log4j.RollingFileAppender”>

<param name=”Threshold” value=”INFO” />

<param name=”File” value=” application_login.log” />

<param name=”MaxFileSize” value=”5MB” />

<param name=”MaxBackupIndex” value=”10″ />

<param name=”BufferSize” value=”512″ />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”%d %p – %m%n” />

</layout>

</appender>

<!–                          –>

<!– Declare the SMTPAppender –>

<!–                          –>

<appender name=”EMAIL”  class=”org.apache.log4j.net.SMTPAppender”>

<param name=”BufferSize” value=”512″ />

<param name=”SMTPHost” value=”mail.magma.ca” />

<param name=”From” value=”xxx@yyyy.com” />

<param name=”To” value=”xxxx@zzzz.com” />

<param name=”Subject” value=”xxxx SYNCHRONIZATION ERROR – SERVER SIDE” />

<param name=”Threshold” value=”ERROR” />

<layout class=”org.apache.log4j.PatternLayout”>

<param name=”ConversionPattern” value=”[%d{ISO8601}]%n%n%-5p%n%n%c%n%n%m%n%n” />

</layout>

<filter class=”com.xxxxx.common.log4j.filters.xxxxLog4jFilter”>

<param name=”SendTimeInterval” value=”100000″/>

<param name=”SuppressEqualsMessages” value=”true”/>

</filter>

</appender>

<logger name=”com.xxxxx.providers.authentication.zzzzLoginModule”>

<appender-ref ref=”LoginAppender”/>

</logger>

<logger name=”com.xxxxx.portal.login.PortalSignInSession”>

<appender-ref ref=”LoginAppender”/>

</logger>

<!– setup log4j’s root logger –>

<root>

<level value=”all” />

<!– Uncomment next line if you deploy in real system not development –>

<appender-ref ref=”EMAIL”/>

<appender-ref ref=”stdout”/>

<appender-ref ref=”INFO”/>

<appender-ref ref=”R”/>

</root>

</log4j:configuration>

Existing Log4j appenders:

  • Console appender
  • Rolling file appender to log application output. Rolling file appender automatically archives log files when they reach certain size and start logging to a new file.

Configuration:

  • Max file size: 5MB
  • Max files to archive: 10
  • Log4j logging level threshold: INFO
  • Rolling file appender to log login information. This appender works only in the classes responsible for application logging services. These dedicated, login information log files are later used by Fail2ban utility which blocks certain IPs if the IPs are the source for multiple login failure within short period of time
  • Email appender which sends email notification in case of ERROR or FATAL log events. The appender uses custom xxxxLog4jFilter filter to prevent from sending same email notification within 100000 milliseconds (100 seconds)
            2.1.3 REST service

Configuration file (log4j 2): EXAMPLE

<?xml version=”1.0″ encoding=”UTF-8″ ?>

<Configuration>

<Appenders>

<File name=”A1″ fileName=”xxxx-Client Application-2.log”>

<PatternLayout>

<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>

</PatternLayout>

</File>

<Console name=”STDOUT” target=”SYSTEM_OUT”>

<PatternLayout pattern=”%d %-5p [%t] %C{2} (%F:%L) – %m%n”/>

</Console>

</Appenders>

<Loggers>

<Root level=”debug”>

<AppenderRef ref=”STDOUT”/>

<AppenderRef ref=”A1″/>

</Root>

</Loggers>

</Configuration>

Existing Log4j appenders:

  • Console / System.out appender,
  • File appender to log application output.

Note: Additional appenders will be added once REST services go into production environment.

3. AWS monitoring utilities

AWS CloudWatch (hypervisor, server resource), is used to provide alerts over certain  thresholds, AWS CloudTrail (API access and logs), is configured to provide alerts over certain thresholds. There is a single admin dashboard with a detailed overview of the platform’s cloud state.  Will add more customized info and screen shots from an existing CW dashboard.