|
|
A pattern is the heart of XpoLog's engine. It
describes the structure of the data XpoLog parses to records, and
could have significant
influence on the efficiency of XpoLog's engine.
Records in the log can probably be presented using different patterns, therefore it is recommended to describe the logged
information in most detailed and descriptive way. Doing so will give
greater possibilities in manipulating and analyzing the data.
XpoLog defines a language that contains several expressions that are
the building blocks of patterns. Each pattern is built out of some
of these expressions,
and could, in addition, contain constant string that do not belong
to the language.
For example, the pattern {string} is a pattern that matches
every string of characters.
Note: every pattern expression should
be surrounded by curly brackets. |
|
Following is a list of all expressions that
exist in XpoLog's pattern language
and the way to use each of them: |
|
string - any string of
characters, including multi line strings
-
Syntax: {string}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
Hello world. |
{string} |
Hello
world. |
Hello world. |
H{string} wor{string} |
first
column = ello,
second column = ld. |
Hello
world. |
{string} |
Hello
world. |
|
text - any
single-line string of characters
-
Syntax: {text}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
Hello world. |
{text} |
Hello
world. |
|
date - a date
string
-
Syntax: {date, MM-dd-yyyy}
-
Special attributes:
-
timeDiff: specify time offset in milliseconds
-
locale: specify the locale in
which the log was written
-
localeView: specify the locale in
which the log should be displayed
-
Examples of special attributes:
Text
in log |
Pattern |
Result in log view |
18:05:23 |
{date,timeDiff=1000,HH:mm:ss} |
18:05:24 |
18:05:23 |
{date,timeDiff=-5000,HH:mm:ss} |
18:05:18 |
|
{date,locale=it,dd MM yyyy
HH:mm:ss} |
|
|
{date,locale=it; localeView=en,dd
MM yyyy} |
|
-
Examples of optional identifiers:
Identifier |
Text
in log |
Pattern |
MM - numeric month |
01-25-1986 |
{date,MM-dd-yyyy} |
MMMMM - full textual month |
25/July/1986 |
{date,dd/MMMMM/yyyy} |
MMM - textual month |
25/Jul/1986 |
{date,dd/MMM/yyyy} |
dd - numeric day |
01:25:1986 |
{date,MM:dd:yyyy} |
EEEEE - full textual day |
Friday 01-25-00 |
{date,EEEEE MM-dd-yy} |
EEE - textual day |
Fri 01-25-00 |
{date,EEE MM-dd-yy} |
yy - 2 digit year |
25/Jul/86 |
{date,dd/MMM/yy} |
yyyy - 4 digit year |
25/Jul/1986 |
{date,dd/MMM/yyyy} |
HH - 24 hour |
18:05:23 |
{date,HH:mm:ss} |
hh - 12 hour |
10:00:00 AM |
{date,hh:mm:ss
a} |
a - AM/PM marker |
mm - minute |
18-05-23 |
{date,HH-mm-ss} |
ss - second |
18:05:23 |
{date,HH:mm:ss} |
SSS - millisecond |
18:05:23 253 |
{date,HH:mm:ss SSS} |
z - general time zone |
18:05:23 EST |
{date,HH:mm:ss z} |
Z - RFC 822 time zone |
18:05:23 -0400 |
{date,HH:mm:ss Z} |
'TEXT' - a constant text that appears in the date
string |
07-1986D25 |
{date,MM-yyyy'D'dd} |
|
timestamp - a timestamp
representing a date string
-
Syntax: {timestamp}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
56895633232 |
{timestamp,yyyy/MM/dd} |
2007/11/13 |
|
number - a numeric string
-
Syntax: {number}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
Thread-1 |
{string}-{number} |
first
column = Thread,
second column = 1 |
Test 5 done |
{string} {number} {string} |
first
column = Test,
second column = 5,
third column = done |
calling to id 5667 |
calling to id {number} |
5667 |
|
choice - a set of
strings that can appear in a record
|
priority - a set of
priorities that can appear in a record
|
ip - an IP
address
-
Syntax: {ip}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
127.0.0.1 |
{ip} |
127.0.0.1 |
|
regexp - a regular
expression, used to extract part of the data from another
column
read more about regular expressions in the
regular expressions help page
|
term - a constant string that appears
in a record and needs to be displayed in the log view
-
Syntax: {term,TERM} TERM being the
constant string
-
No special attributes
-
Example:
Assuming you have the next two records in your log:
03/07/2005 03:44:56 app1 IP=192.168.11.44 success (where IP is a
source IP)
03/07/2005 03:44:57 app2 IP=192.168.12.33 failure (where IP is a
destination IP)
The multi-pattern you should use is:
{date,dd/MM/yyyy HH:mm:ss} {term,app1} IP={text:Source
IP} {string}
{date,dd/MM/yyyy HH:mm:ss} {term,app2} IP={text:Destination
IP} {string}
This will result in the following view in the log view:
03/07/2005 03:44:56 app1 192.168.11.44
success
03/07/2005 03:44:57 app2
192.168.12.33 failure
|
freetext - any text you wish to
display in the log view, usually used in multi-pattern logs to
distinguish records
-
Syntax: {freetext,FREETEXT} FREETEXT
being the text you wish to display in the log view
-
No special attributes
-
Example:
Assuming you have the next records in your log:
03/07/2005 03:44:56 $ success
03/07/2005 03:44:57 % failure
The multi-pattern you should use is:
{date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string}
{date,dd/MM/yyyy HH:mm:ss} {freetext,Percentage}%
{string}
Note that the freetext tag does not replace any text in the
pattern, therefore it could be placed anywhere in the pattern
and should not be separated with a space.
This will result in the following view in the log view:
03/07/2005 03:44:56 Dollar success
03/07/2005 03:44:57 Percentage failure
|
block - an optional
string that does not appear in all records
-
Syntax: {block,start,emptiness=true}XXX{block,end,emptiness=true}
-
Special attributes:
-
start: indicates a block start
-
end: indicates a block end
-
emptiness: indicates whether the block's data could
be missing
-
Examples:
Text
in log |
Pattern |
first record =
64.236.16.52:8080,
second record = 64.236.16.52,
third record = 64.236.16.52:7001 |
{string}{block,start,emptiness=true}:{string}{block,end,emptiness=true} |
|
tab - a tab
delimiter
-
Syntax: {tab}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
one 123 |
{string}{tab}{number} |
first
column = one,
second column = 123 |
|
eol - end of line, used in records that spread over more than one line
-
Syntax: {eol}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
name : logger
level : db0 |
name : {string}{eol}level :
{priority,db0;db1} |
first
column = logger,
second
column = db0 |
|
eoe - end of entity, used to mark the end of a
record, improves the efficiency of the parsing process
-
Syntax: {eoe}
-
No special attributes
-
Examples:
Text
in log |
Pattern |
Result in log view |
name : logger
level : db0
name : logger
level : db1 |
name : {string}{eol}level :
{priority,db0;db1}{eoe} |
first row =
logger,db0
second row = logger,db1 |
|
Note:
All types support the following attributes:
Name: this attribute should always follow the tag name with a
leading colon.
Examples:
{string:Title}, {date:Start Date,dd/mm/yyyy},
{priority:Severity,DEBUG;INFO;ERROR}, {number:Status Code}
uiMessageLength: this attribute allows you to limit the
length of data displayed in a specific column. If the data is longer
than specified, it will be divided to several lines.
Example: {string:Title,uiMessageLength=20}
charsLength: this attribute allows
you to force the existence of a fixed number of characters in a
string, even if there are less characters in the record.
Example: {string:Title,charsLength=10} |
- { -> '{' - the left curly bracket
character ( { ) can be used as literal only if surrounded by a
single quote.
- ' -> '' - the quote character ( '
) can be used as literal only if preceded by another quote.
|
|
You may see wrong data or
no data at all.
IMPORTANT!!!
After defining the pattern it is highly recommended to click on
the 'verify pattern'
link - you will then see the results of your definition in the table
at the bottom of the page.
If you don't see data at all or you see wrong data - check your
pattern definition again. |
|
Here you can see some
complete pattern examples for a given log: |
|
2003-02-12 12:37:26 ContextConfig[/examples]:
Missing application web.xml, using defaults only
2003-02-12 12:37:26 StandardManager[/examples]: Seeding random
number generator class java.security.SecureRandom
2003-02-12 12:37:30 StandardManager[/examples]: Seeding of random
number generator has been completed
2003-02-12 12:37:30 StandardWrapper[/examples:default]: Loading
container servlet default
2003-02-12 12:37:30 StandardWrapper[/examples:invoker]: Loading
container servlet invoker |
Data pattern: {date,yyyy-MM-dd HH:mm:ss}
{string}[/{string}]: {string} |
|
127.0.0.1 - - [26/Dec/2001:19:49:23
+0200] "GET / HTTP/1.1" 200 1494
127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET /apache_pb.gif
HTTP/1.1" 200 2326
127.0.0.1 - - [26/Dec/2001:19:52:48 +0200] "GET /examples/ HTTP/1.1"
404 277
127.0.0.1 - - [26/Dec/2001:19:54:37 +0200] "GET /examples/jsp/snp/snoop.jsp
HTTP/1.1" 404 294
127.0.0.1 - - [28/Dec/2001:09:54:37 +0200] "GET /puga/main.html
HTTP/1.1" 404 282 |
Data pattern: {string} - - [{date,dd/MMM/yyyy:HH:mm:ss
Z}
+0200] "{string}" {number} {number} |
|
[Wed Dec 26 19:52:48 2001] [error]
[client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not
exist: c:/devapp/apache/apache/htdocs/_vti_bin/owssvr.dll
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not
exist: c:/devapp/apache/apache/htdocs/msoffice/cltreq.asp |
Data pattern: [{string} {date,EEE MMM dd HH:mm:ss yyyy]
[{priority,debug;info;warn;error;fatal}] [{string} {string}]
{string} |
|
28/02/03 20:23:16 ERR Critical error on section 34 on
module 5 [Channel 9] 4.4.4.4 28/02/03 20:25:35 DBG
information arrived to fusion zone, restoring states [Channel 39] 4.8.4.9
28/02/03 20:33:22 WRN port collision seeking another
[Channel 19] 4.4.4.4 28/02/03 20:33:22 FLW DB connection open
structure initiated [Channel 9] 4.23.12.5 |
Data pattern: {date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR}
{string} [{string}] {string} |
|
5
d MBGN Talk to
port
9 f
MLPT1 Port open
0 x
MCOM Com port open |
Data pattern: {number}{tab}{string}{tab}{string}{tab}{string} |
|
5
d MBGN procId=123
Talk to port
9 f
MLPT1 Port open
0 x
MCOM procId=456 Com port open |
Data pattern: {number}{tab}{string}{tab}{block,start,emptiness=true}procId={string}{block,end,emptiness=true}{string}{tab}{string} |
Records in the log can probably be
presented by a combination of types. It is recommended to
present the logged information in most detailed way. Doing so will
give greater possibilities in manipulating and analyzing the data.
such as filtering by a specific ip, priority, date or specific
text column. You should try to be as descriptive as possible.
Note that you can configure several
patterns for one log using XpoLog multi pattern -
Learn more about Multi
Patterns |
Each command will be treated as a
column of data in the log view table. |
For any definition problem please contact
XpoLog support with log example by email
support@xplg.com
and we will help you define a pattern. |
|
Multi Pattern help
Header Pattern help
|