XpoLog help - Patterns
 

General

A pattern is the heart of XpoLog's engine. It describes the structure of the data XpoLog parses to records, and could have significant influence on the efficiency of XpoLog's engine. Records in the log can probably be presented using different patterns, therefore it is recommended to describe the logged information in most detailed and descriptive way. Doing so will give greater possibilities in manipulating and analyzing the data.

XpoLog defines a language that contains several expressions that are the building blocks of patterns. Each pattern is built out of some of these expressions, and could, in addition, contain constant string that do not belong to the language.
For example, the pattern {string} is a pattern that matches every string of characters.

Note: every pattern expression should be surrounded by curly brackets.


XpoLog's Pattern Language

Following is a list of all expressions that exist in XpoLog's pattern language and the way to use each of them:

 

string - any string of characters, including multi line strings

  • Syntax: {string}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    Hello world. {string} Hello world.
    Hello world. H{string} wor{string} first column = ello,
    second column =  ld.
    Hello
    world.
    {string} Hello world.

text - any single-line string of characters

  • Syntax: {text}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    Hello world. {text} Hello world.

date - a date string

  • Syntax: {date, MM-dd-yyyy}

  • Special attributes:

    • timeDiff: specify time offset in milliseconds

    • locale: specify the locale in which the log was written

    • localeView: specify the locale in which the log should be displayed

  • Examples of special attributes:

    Text in log Pattern Result in log view
    18:05:23 {date,timeDiff=1000,HH:mm:ss} 18:05:24
    18:05:23 {date,timeDiff=-5000,HH:mm:ss} 18:05:18
      {date,locale=it,dd MM yyyy HH:mm:ss}  
      {date,locale=it; localeView=en,dd MM yyyy}  
  • Examples of optional identifiers:

    Identifier Text in log Pattern

    MM - numeric month

    01-25-1986 {date,MM-dd-yyyy}
    MMMMM - full textual month 25/July/1986 {date,dd/MMMMM/yyyy}
    MMM - textual month 25/Jul/1986 {date,dd/MMM/yyyy}
    dd - numeric day 01:25:1986 {date,MM:dd:yyyy}
    EEEEE - full textual day Friday 01-25-00 {date,EEEEE MM-dd-yy}
    EEE - textual day Fri 01-25-00 {date,EEE MM-dd-yy}
    yy - 2 digit year 25/Jul/86 {date,dd/MMM/yy}
    yyyy - 4 digit year 25/Jul/1986 {date,dd/MMM/yyyy}
    HH - 24 hour 18:05:23 {date,HH:mm:ss}
    hh - 12 hour 10:00:00 AM {date,hh:mm:ss a}
    a - AM/PM marker
    mm - minute 18-05-23 {date,HH-mm-ss}
    ss - second 18:05:23 {date,HH:mm:ss}
    SSS - millisecond 18:05:23 253 {date,HH:mm:ss SSS}
    z - general time zone 18:05:23 EST {date,HH:mm:ss z}
    Z - RFC 822 time zone 18:05:23 -0400 {date,HH:mm:ss Z}
    'TEXT' - a constant text that appears in the date string 07-1986D25 {date,MM-yyyy'D'dd}

timestamp - a timestamp representing a date string

  • Syntax: {timestamp}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    56895633232 {timestamp,yyyy/MM/dd} 2007/11/13

number - a numeric string

  • Syntax: {number}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    Thread-1 {string}-{number} first column = Thread,
    second column =  1
    Test 5 done {string} {number} {string} first column = Test,
    second column =  5,
    third column = done
    calling to id 5667 calling to id {number} 5667

choice - a set of strings that can appear in a record

  • Syntax: {choice,value1;value2...,valueN} - all optional choices should appear as a semi-colon separated list inside the tag

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    first record = red,
    second record = green,
    third record = blue
    {choice,red;green;blue} the matching choice of each record will be displayed

priority - a set of priorities that can appear in a record

  • Syntax: {priority,priority1;priority2...,priorityN} - all optional priorities should appear as a semi-colon separated list inside the tag

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    first record = DEBUG,
    second record = INFO,
    third record = ERROR
    {priority,DEBUG;INFO;ERROR} the matching priority of each record will be displayed

ip - an IP address

  • Syntax: {ip}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    127.0.0.1 {ip} 127.0.0.1

regexp - a regular expression, used to extract part of the data from another column
read more about regular expressions in the regular expressions help page

  • Syntax: {regexp,[refIndex=index | refName=column name],(expression)}

  • Special attributes:

    • refIndex (optional): the zero-based index of the source column

    • refName (optional): the name of the source column

    • multiLine (optional): indicates whether the record spreads over more than one line

    • expression: the regular expression that will be extracted out of the value in the source column

  • Example:


  • Assuming you have the next record in your log, and you need to extract the error code (highlighted in the example) that is embedded in the message:

    Mon Jul 10 04:33:51 2006 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
    GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
    ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
    GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...

    The pattern you should use is:

    {date:Date,EEE MMM dd HH:mm:ss yyyy}{regexp:Error Code,refName=Message;multiLine=true,(ORA-\d+)}{string:Message}

    This will result in the following view in the log view:

    Mon Jul 10 04:33:51 2006 ORA-336 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
    GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
    ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
    GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...

term - a constant string that appears in a record and needs to be displayed in the log view

  • Syntax: {term,TERM} TERM being the constant string

  • No special attributes

  • Example:
    Assuming you have the next two records in your log:

    03/07/2005 03:44:56 app1 IP=192.168.11.44 success (where IP is a source IP)
    03/07/2005 03:44:57 app2 IP=192.168.12.33 failure (where IP is a destination IP)

    The multi-pattern you should use is:

    {date,dd/MM/yyyy HH:mm:ss} {term,app1} IP={text:Source IP} {string}
    {date,dd/MM/yyyy HH:mm:ss} {term,app2} IP={text:Destination IP} {string}

    This will result in the following view in the log view:

    03/07/2005 03:44:56 app1 192.168.11.44                       success
    03/07/2005 03:44:57 app2                       192.168.12.33 failure

freetext - any text you wish to display in the log view, usually used in multi-pattern logs to distinguish records

  • Syntax: {freetext,FREETEXT} FREETEXT being the text you wish to display in the log view

  • No special attributes

  • Example:
    Assuming you have the next records in your log:

    03/07/2005 03:44:56 $ success
    03/07/2005 03:44:57 % failure

    The multi-pattern you should use is:

    {date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string}
    {date,dd/MM/yyyy HH:mm:ss} {freetext,Percentage}% {string}

    Note that the freetext tag does not replace any text in the pattern, therefore it could be placed anywhere in the pattern and should not be separated with a space.

    This will result in the following view in the log view:

    03/07/2005 03:44:56 Dollar success
    03/07/2005 03:44:57 Percentage failure

block - an optional string that does not appear in all records

  • Syntax: {block,start,emptiness=true}XXX{block,end,emptiness=true}

  • Special attributes:

    • start: indicates a block start

    • end: indicates a block end

    • emptiness: indicates whether the block's data could be missing

  • Examples:

    Text in log Pattern
    first record = 64.236.16.52:8080,
    second record = 64.236.16.52,
    third record = 64.236.16.52:7001
    {string}{block,start,emptiness=true}:{string}{block,end,emptiness=true}

tab - a tab delimiter

  • Syntax: {tab}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    one    123 {string}{tab}{number} first column = one,
    second column =  123

eol - end of line, used in records that spread over more than one line

  • Syntax: {eol}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    name : logger
    level : db0
    name : {string}{eol}level : {priority,db0;db1} first column = logger,
    second column =  db0

eoe - end of entity, used to mark the end of a record, improves the efficiency of the parsing process

  • Syntax: {eoe}

  • No special attributes

  • Examples:

    Text in log Pattern Result in log view
    name : logger
    level : db0
    name : logger
    level : db1
    name : {string}{eol}level : {priority,db0;db1}{eoe} first row = logger,db0
    second row = logger,db1

Note:
All types support the following attributes:
Name: this attribute should always follow the tag name with a leading colon.
           Examples: {string:Title}, {date:Start Date,dd/mm/yyyy}, {priority:Severity,DEBUG;INFO;ERROR}, {number:Status Code}
uiMessageLength: this attribute allows you to limit the length of data displayed in a specific column. If the data is longer than specified, it will be divided to several lines.
                             Example:
{string:Title,uiMessageLength=20}
charsLength: this attribute allows you to force the existence of a fixed number of characters in a string, even if there are less characters in the record.
                      Example: {string:Title,charsLength=10}

Special cases
  1. { -> '{' - the left curly bracket character ( { ) can be used as literal only if surrounded by a single quote.
  2.  ' -> '' - the quote character ( ' ) can be used as literal only if preceded by another quote.

What happens if my pattern is wrong?
You may see wrong data or no data at all.

IMPORTANT!!!
After defining the pattern it is highly recommended to click on the 'verify pattern' link - you will then see the results of your definition in the table at the bottom of the page.
If you don't see data at all or you see wrong data - check your pattern definition again.

Examples
Here you can see some complete pattern examples for a given log:

Log 1
2003-02-12 12:37:26 ContextConfig[/examples]: Missing application web.xml, using defaults only
2003-02-12 12:37:26 StandardManager[/examples]: Seeding random number generator class java.security.SecureRandom
2003-02-12 12:37:30 StandardManager[/examples]: Seeding of random number generator has been completed
2003-02-12 12:37:30 StandardWrapper[/examples:default]: Loading container servlet default
2003-02-12 12:37:30 StandardWrapper[/examples:invoker]: Loading container servlet invoker
Data pattern: {date,yyyy-MM-dd HH:mm:ss} {string}[/{string}]: {string}

Log 2

127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET / HTTP/1.1" 200 1494
127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET /apache_pb.gif HTTP/1.1" 200 2326
127.0.0.1 - - [26/Dec/2001:19:52:48 +0200] "GET /examples/ HTTP/1.1" 404 277
127.0.0.1 - - [26/Dec/2001:19:54:37 +0200] "GET /examples/jsp/snp/snoop.jsp HTTP/1.1" 404 294
127.0.0.1 - - [28/Dec/2001:09:54:37 +0200] "GET /puga/main.html HTTP/1.1" 404 282

Data pattern: {string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number}

Log 3
[Wed Dec 26 19:52:48 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/_vti_bin/owssvr.dll
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/msoffice/cltreq.asp
Data pattern: [{string} {date,EEE MMM dd HH:mm:ss yyyy] [{priority,debug;info;warn;error;fatal}] [{string} {string}] {string}

Log 4

28/02/03 20:23:16 ERR Critical error on section 34 on module 5 [Channel 9] 4.4.4.4 
28/02/03 20:25:35 DBG information arrived to fusion zone, restoring states [Channel 39] 4.8.4.9
28/02/03 20:33:22 WRN port collision seeking another [Channel 19] 4.4.4.4
28/02/03 20:33:22 FLW DB connection open structure initiated [Channel 9] 4.23.12.5

Data pattern: {date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR} {string} [{string}] {string}

Log 5
5      d      MBGN     Talk to port
9      f       MLPT1    Port open
0      x      MCOM     Com port open
Data pattern: {number}{tab}{string}{tab}{string}{tab}{string}

Log 6
5      d      MBGN     procId=123   Talk to port
9      f       MLPT1    Port open
0      x      MCOM     procId=456   Com port open
Data pattern: {number}{tab}{string}{tab}{block,start,emptiness=true}procId={string}{block,end,emptiness=true}{string}{tab}{string}
 

Records in the log can probably be presented by a combination of types. It is recommended to present the logged information in most detailed way. Doing so will give greater possibilities in manipulating and analyzing the data. such as filtering by a specific ip, priority, date or specific text column. You should try to be as descriptive as possible.
Note that you can configure  several patterns for one log using XpoLog multi pattern -
Learn more about Multi Patterns

Each command will be treated as a column of data in the log view table.
 

For any definition problem please contact XpoLog support with log example by email support@xplg.com and we will help you define a pattern.

 

Additional links

         Multi Pattern help
         Header Pattern help