Transforming log and trace data in your application with Generic Log Adapter

October 3, 2005

Author: Dave Smith (smith@ca.ibm.com)

Table of Contents:
1.0 Introduction
2.0 Programming with the GLA API
3.0 Coding a simple example
4.0 Packaging GLA with your application
4.1 Packaging the run-time JAR files
4.2 Packaging the schema files
4.3 Packaging the adapter configuration files
5.0 Downloading the GLA files



1.0 Introduction

Generic Log Adapter (GLA) transforms proprietary log and trace data to the Common Base Event format. It is a feature of the Eclipse Test and Performance Tools Platform (TPTP) project.

GLA parses log files based on regular expression rules or java parsing classes specified in adapter configuration files. The configuration files define how to transform log file records into a standard form that can be used by other tools. Currently, GLA transforms log file records into standard Common Base Event objects. However, GLA is extendable to generate any kind of structured event representation.

GLA is written in the Java programming language and can easily be embedded in any Java application. The application may be a tool for examining or processing log files. For example, in TPTP, Log and Trace Analyzer uses GLA to import log files so that they can be displayed and correlated in the views of Log and Trace Analyzer.

Another example use of GLA is for an application that generates a log file in a proprietary format. The application can make its log file available for an administrator to monitor with TPTP Log and Trace Analyzer. Using the Adapter Configuration Editor in the Generic Log Adapter perspective of TPTP, an adapter configuration file can be created that continuously parses the proprietary log data into CommonBaseEvent objects and sends them to a logging agent. The application can create an instance of GLA to run the adapter configuration file in a separate thread. An administrator can then attach to the logging agent for the application using Log and Trace Analyzer and continuously monitor the log of the application in the Log View of Log and Trace Analyzer.

This document is intended for application developers who want to exploit the GLA technology in their programs to transform log and trace data to a standard format like Common Base Event. The programming interface and packaging details required to use GLA in another application are described. The discussion is based on TPTP 4.0 version of GLA.

This document assumes the reader is familiar with the concepts of GLA and how to create an adapter configuration file using Adapter Configuration Editor. For more information on GLA and how to use the Adapter Configuration Editor, see the video An Introduction to the Generic Log Adapter




2.0 Programming with the GLA API

GLA is implemented in the org.eclipse.hyades.logging.adapter plugin. To use GLA in an application, the main class org.eclipse.hyades.logging.adapter.Adapter must be instantiated. You can use the following Adapter instance methods to configure and run GLA.

setContextConfigPath


public void setContextConfigPath(java.lang.String configFilePath)
Sets the file path of the context configuration portion of the adapter configuration. This method must be called before the validate and start methods in order for the configuration to be used. If this method is not called, the default file name SimpleContextConfiguration.xml will be used. Use the Adapter Configuration Editor to generate the configuration file.

Typically, the context configuration and component configuration are created in the same file using Adapter Configuration Editor. In this case, call setContextConfigPath and setComponentConfigPath with the same file path.

Parameters:
configFilePath - The file path of the context configuration

setComponentConfigPath


public void setComponentConfigPath(java.lang.String configFilePath)
Sets the file path of the component configuration portion of the adapter configuration. This method must be called before the validate and start methods in order for the configuration to be used. If this method is not called, the default file name SimpleComponentConfigurations.xml will be used. Use the Adapter Configuration Editor to generate the configuration file.

Typically, the context configuration and component configuration are created in the same file using Adapter Configuration Editor. In this case, call setContextConfigPath and setComponentConfigPath with the same file path.

Parameters:
configFilePath - The file path of the component configuration

validate


public void validate()
              throws AdapterException
Validates the Generic Log Adapter configuration files specified using the setContextConfigPath and setComponentConfigPath methods. If you do not specify the context or component configuration file paths, the default configuration files SimpleContextConfiguration.xml and SimpleComponentConfigurations.xml are validated. It initializes the Adapter instance with the configuration specified in the configuration files but does not run the Adapter instance. To run the Adapter use the start method.

Throws:
AdapterException - if the configuration is invalid

start


public void start(boolean separateThread,
                  boolean daemon)
           throws AdapterException
Starts Generic Log Adapter with the configuration files specified using the setContextConfigPath and setComponentConfigPath methods. If you do not specify the context or component configuration file paths, the Adapter instance uses SimpleContextConfiguration.xml and SimpleComponentConfigurations.xml as the default configuration files, respectively. Note, the configuration files are validated before the Adapter instance uses them.

Generic Log Adapter can be started in a separate thread by specifying true as the first parameter and in addition started as a daemon thread by specifying true for both parameters. If true is specified as the first parameter, start will return immediately after starting the Generic Log Adapter thread. Otherwise, if the first parameter is false, Generic Log Adapter will be started in the current thread, regardless of the value of the second parameter, and start will return when Generic Log Adapter has finished executing all of the contexts configured in the Adapter instance. Generic Log Adapter runs each context configured in the Adapter instance in a separate thread. Therefore, if the Adapter instance is configured with two contexts and start(true, true) is called, three threads will be started to execute Generic Log Adapter with two contexts.

Parameters:
separateThread - Run the adapter in a separate thread
daemon - Run the adapter as a daemon thread.
Throws:
AdapterException - If an error occurs during execution.

stop


public void stop()
Stops all of the adapter contexts. Data still being processed by the contexts will be flushed to the last component of each context. The stop method waits until all contexts are stopped before stopping the internal Generic Log Adapter logging context in order to capture all messages from all contexts.


hardStop


public void hardStop()
Stops all of the adapter contexts. Data still being processed by the contexts will be flushed to the last component of each context. The hardStop method also disables any context listeners during flushing. Context listeners implement the IContextListener interface and are only supported when Generic Log Adapter is running in an Eclipse environment. The hardStop method does not wait for all contexts to stop before stopping the internal Generic Log Adapter logging context so some messages from the other contexts may not be captured.


main


public static void main(java.lang.String[] args)
Adapter main method to run Generic Log Adapter as a standalone application.

Parameters:
args - -cc contextConfigurationPath -ac componentConfigurationPath

getStatus


public IStatus getStatus()
Gets the status of the adapter instance.

Returns:
the org.eclipse.hyades.logging.adapter.IStatus object representing the status.

setLogOutputter


public void setLogOutputter(IOutputter outputter)
Set the Adapter log outputter for Generic Log Adapter messages. Generic Log Adapter uses an internal context to log trace and error messages during the execution of the Adapter instance. The outputter specified with setLogOutputter is used in the internal logging context. This method should be called before calling validate() and start(), which may log validation and run-time execution messages. If this method is not called, GLA will log its messages to the file hgla.log in the current directory.

Parameters:
outputter - component where GLA will log its messages.

The methods are typically called in the following order:

  1. setContextConfigPath(adapterConfigFileName)
  2. setComponentConfigPath(adapterConfigFileName)
  3. setLogOutputter(glaLogOutputter) - optional
  4. validate() - optional
  5. start(separateThread, daemonThread)

The following methods are typically called from a different thread than the one that called start if start is called with separateThread=false. If start is called with separateThread=true then these methods can be called from the same thread that called start.

Note: Missing from this programming interface is the ability to create or modify adapter configuration files programmatically. This capability is under consideration for a future release. Please see the TPTP feature enhancement request 96433




3.0 Coding a simple example

The following example shows a simple main program to run the adapter configuration file c:\test.adapter.

import org.eclipse.hyades.logging.adapter.Adapter;
import org.eclipse.hyades.logging.adapter.AdapterException;
import org.eclipse.hyades.logging.adapter.IStatus;
import org.eclipse.hyades.logging.adapter.impl.Status;
import org.eclipse.hyades.logging.adapter.outputters.CBEFileOutputter;

public class RunAdapter {

    /**
     * This is a simple program to run the Generic Log Adapter
     * with a hardcoded adapter configuration file.
     */
    public static void main(String[] args) {

        String adapterConfigFile = "c:\\test.adapter";

        /* Create an Adapter instance */
        Adapter adapter = new Adapter();

        /* Set the adapter configuration file */
        adapter.setContextConfigPath(adapterConfigFile);
        adapter.setComponentConfigPath(adapterConfigFile);

        /* Create a file outputter to capture GLA
         * log messages.
         * WARNING:  The following code does not work
         * in TPTP 4.0 due to bugzilla 108187 in
         * CBEFileOutputter.  Please comment the
         * following three lines of code if using
         * TPTP 4.0.
         */
        CBEFileOutputter fo = new CBEFileOutputter();
        fo.setDirectory("c:\\logs");
        fo.setFileName("testgla.log");

        /* Set the outputter to capture the GLA log messages */
        adapter.setLogOutputter(fo);

        /* Validate the adapter configuration file */
        try {
            adapter.validate();
        }
        catch (AdapterException e) {
            System.err.println("Adapter configuration file is invalid.");
            System.err.println(e.getMessage());
            return;
        }

        System.out.println("Adapter configuration was validated.");

        /* Start the adapter as a separate thread */
        try {
            adapter.start(true, true);
        }
        catch (AdapterException e) {
            /* If we get an exception at this point we likely do
             * not have a valid configuration and hence the log file
             * is likely not flushed.  We need to write this to stderr.
             */
            System.err.println("GLA Exception occurred");
            System.err.println(e.getMessage());
            adapter.stop();
            return;
        }

        System.out.println("Adapter was started.");

        /* Wait 30 seconds */
        try {
            Thread.sleep(30000);
        }
        catch (InterruptedException e) {
        	
        }

        /* Get the adapter status */
        IStatus status = adapter.getStatus();

        if (status != null) {
            /* print the current adapter status out */
            printAdapterStatus(status);
        }

        /* If the adapter is still running then stop it */
        if (status != null && status.isActive()) {
            adapter.stop();
            System.out.println("Adapter was stopped.");
        }
    }

    /**
     * Print the adapter status.
     *
     * @param status Adapter IStatus object
     */
    private static void printAdapterStatus(IStatus status) {
        /* print the current adapter status out */
        if (status.isActive()) {
            System.out.println("Adapter is active!  It has processed "
                + status.getItemsProcessedCount() + " items in "
                + ((Status)status).getElpasedTimeInSeconds()
                + " seconds.");
        }
        else {
            System.out.println("Adapter is not active!  It processed "
                + status.getItemsProcessedCount() + " items in "
                + ((Status)status).getElpasedTimeInSeconds()
                + " seconds.");
        }
    	
        /* Get the status of the contexts */
        IStatus [] contexts = status.getChildrenStatus();

        if (contexts != null && contexts.length > 0) {
            for (int i=0; i < contexts.length; i++) {
                /* For each context, print its status */
                if (contexts[i] != null) {
                    IStatus cstatus = contexts[i];
                    if (cstatus.isActive()) {
                        System.out.println("Context " + cstatus.getName()
                            + " is active!  It has processed "
                            + cstatus.getItemsProcessedCount()
                            + " items in "
                            + ((Status)cstatus).getElpasedTimeInSeconds()
                            + " seconds.");
                    }
                    else {
                        System.out.println("Context " + cstatus.getName()
                            + " is not active!  It processed "
                            + cstatus.getItemsProcessedCount()
                            + " items in "
                            + ((Status)cstatus).getElpasedTimeInSeconds()
                            + " seconds.");
                    }

                    /* Get the status of the components */
                    IStatus [] components = cstatus.getChildrenStatus();
                    if (components != null && components.length > 0) {
                        for (int j=0; j < components.length; j++) {
                            /* For each component, print its status */
                            IStatus cpstatus = components[j];
                            if (cpstatus.isActive()) {
                                System.out.println("  Component "
                                    + cpstatus.getName()
                                    + " is active and has processed "
                                    + cpstatus.getItemsProcessedCount()
                                    + " items in "
                                    + ((Status)cpstatus).getElpasedTimeInSeconds()
                                    + " seconds.");
                            }
                            else {
                                System.out.println("  Component "
                                    + cpstatus.getName()
                                    + " is not active and processed "
                                    + cpstatus.getItemsProcessedCount()
                                    + " items in "
                                    + ((Status)cpstatus).getElpasedTimeInSeconds()
                                    + " seconds.");
                            }
                        }
                    }
                }
            }
        }
    }
}

This sample program performs the following operations:

  1. Creates an Adapter instance.
  2. Gives the Adapter instance the adapter configuration file path to execute.
  3. Gives the Adapter instance a CBEFileOutputter instance to use to capture the GLA log messages. Note: There is bug in CBEFileOutputter in TPTP 4.0 that causes this program to fail. Comment this portion of the program if you are using TPTP 4.0. This problem has been fixed in TPTP 4.1. See bugzilla 108187 for more details.
  4. Validates the adapter configuration against the schema and validates the content of the configuration.
  5. Starts the execution of the Adapter instance with the specified adapter configuration on a separate daemon thread.
  6. Waits 30 seconds and then retrieves the status of the Adapter instance.
  7. Prints the status of the Adapter instance to System.out PrintStream.
  8. Stops the execution of the Adapter instance if it is still active.



4.0 Packaging GLA with your application

To use GLA in your application you must package the required GLA run-time files with your application and make them available at the time of execution. The following GLA files are required:

The following jar files are required to be on the java classpath:

hgla.jar GLA run-time classes
hparse.jar GLA static parser support and example classes
hlcore.jar TPTP logging core classes
hlcbe101.jar TPTP CommonBaseEvent implementation classes
hexr.jar TPTP remote agent execution classes
emf.common_2.1.0.jar Eclipse standalone EMF common classes
emf.ecore_2.1.0.jar Eclipse standalone EMF core classes

In addition, any custom GLA component or static parser classes that you have created must be on the java classpath in order for them to be found by GLA run-time.

To perform schema validation on the adapter configuration file, the following GLA schema files must be accessible to the GLA run-time. To make them accessible the GLA_HOME environment variable must be set to the location of the schema files.

   Adapter.xsd
   ComponentConfiguration.xsd
   Context.xsd
   Extractor.xsd
   Formatter.xsd
   Outputter.xsd
   Parser.xsd
   ProcessUnit.xsd
   Sensor.xsd

The adapter configuration files to be executed by GLA must be accessible. These can be packaged with your application as well.




The GLA run-time jar files and schema files can be acquired by downloading the Generic Log Adapter package from the TPTP Download page