XpoLog help - Definitions in Logs
 
As described earlier, the system health is measured both in terms of risk and in terms of anomalies. Both risk and anomaly definitions can be set for each and every log in your system.
 
  • Defining log risks
    Risks for a log are defined as the occurrence of certain events in that log. For instance, the occurrence of an access log event with a 404 status may denote that an illegal request has been submitted. Or an event of an error log with a priority "fatal" means something pretty bad happened in your application.
    To isolate these events, XpoLog uses a set of filters, each corresponding to an event or a multitude of events. From the log view select from the log navigation bar the filter’s menu by pressing the small arrow next to the filter’s "go" link. Select "edit" to edit an already defined filter or "new" to create a new one. For more information on defining filters, see Filter Definition. In the filter definition page, the last section is denoted "System Health". This is where you set the risk level that corresponds to the occurrence of (one or more) events defined by that filter. The lowest risk is 1 and the highest is 10. If the event denoted by that filter does not mean any risk to your system, leave the risk weight at "None". For instance, in an error log containing priorities, you might set a filter called "fatal" by selecting the "fatal" entry of the "Priorities" section in the filter definition page, and set the risk weight to be 9.
    To complete the System Health definition enter the condition for the given risk weight by selecting the number of events and the operation (more then, less then, equals and not equals). The condition defines the number of occurrences of that log within the minimal time frame of 5 minutes. For instance if the occurrence of at least 5 error events in an error log in a time frame of five minutes means a risk weight of 8, select 8 at the risk weight, "More Then" in the next combo box and enter "4" in the text box to complete your definition.
    Log’s configurations in XpoLog generated either by the detection wizard or by running an application wizard have already their risk levels set for their predefined filters. You can always edit these filters to change their risk level or enter new filters with new risk levels that correspond to events you consider risky.

  • Defining log statistics
    Anomalies are defined as deviations in terms of the number of occurrences with relation to previously computed averages. Statistics for logs is computed for every hour of the week and takes into account both the number of total events of the log and the number of unique occurrences of values for selected log columns ("Column statistics"). For instance, if a log consists of a "priority" column, statistics may be computed for each occurrence of a priority value, such as "warn", "error" or "fatal". The statistics then holds information of the number of each "warn", "error" and "fatal" occurrences for each hour of the week. If for instance there are, in a certain hour, many "error" events, this will be displayed in the dashboard as an anomaly. Column statistics can be defined in two ways: the first one consists of computing the statistics for each of the values of that column. When computing that columns anomaly, the occurrences of column values of the generated data are each compared to their statistics; the second method for computing statistics consists of computing the average number of occurrences of these values. In this case, when computing that column anomaly the number of occurrences of each of the values is compared to that average. The first method should be used when a limited and known number of values is expected for a column, such as in the case of priority, while the second method should be used when a large or unknown number of values is expected, such as in a "URL" column of a proxy log or in an "IP" column of an access log. If a certain IP occurs a certain number of times in a certain hour within the time frame used for the generation of the dashboard data, then this number is compared to the average number of unique occurrences of IP’s in that hour of the week and not to that specific value’s statistics.
     
    In order to set the statistics parameters of a certain log, go to this log’s edit wizard. See 'Log Configuration' for more details on how to define logs. Click 'next' in the Log’s general definition’s page to get to the 'Log pattern administration' page and click 'next' again here to get to the 'Log field admin' page. In the 'statistics settings' section, select the 'Compute statistics' option to enable statistical evaluation of the log. When this option is selected, the 'Compute statistics' combo box is displayed next to each of the log’s fields (columns). Select one of the following options:

    1. 'do not compute' – no statistics (and no anomaly) will be computed for this column.

    2. 'compute unique values' – statistics will be computed for each of this column's values.
    3. 'compute average' – only the average of the occurrences of unique values will be computed. Anomaly for each unique value is then computed based on this average statistics.

    Selecting 'compute unique values' for a column where many unique values are expected will result in large statistics data and less accurate anomalies, so always consider in these cases using 'compute average'.