LogQL

LogQL is the query language that is used by Grafana’s Loki.

There are two types of LogQL queries:

  • Log queries which return the contents of log lines.
  • Metric queries that calculate values based on the counts of logs from a log query.

A basic log query consists of two parts:

  • A log stream selector.
  • A filter expression.

Due to Loki’s design, all LogQL queries must contain a log stream selector.

Stream selectors

The log stream selector limits which records a query will be evaluated on. As stream selectors match on the indexed key-value labels, this filtering is very cheap and fast to do.

The log stream selector is written by wrapping the key-value pairs in a pair of curly braces:

{app="mysql",name="mysql-backup"}
{host="td1-plg-vmweb01",log_type="nginx_access_log"}
{host=~"^td1-plg-vmweb.*",log_type="nginx_access_log"}

The following comparison operators are supported:

  • =: exactly equal.
  • !=: not equal.
  • =~: regex matches.
  • !~: regex does not match.

Regexes use Go RE2 syntax. The matching is case-sensitive by default and can be switched to case-insensitive prefixing the regex with (?i).

Filter expressions

After writing the log stream selector, the resulting set of logs can be further filtered with a search expression:

{host=~"^td1-plg-vmweb.*",log_type="nginx_access_log"} |= `"status_code":500`
{host=~"^td1-plg-vmweb.*",log_type="nginx_access_log"} |~ `"status_code":[45][0-9]{2}`

The following filter operators are supported:

  • |=: Log line contains string.
  • !=: Log line does not contain string.
  • |~: Log line matches regular expression.
  • !~: Log line does not match regular expression.

Regexes use Go RE2 syntax. The matching is case-sensitive by default and can be switched to case-insensitive prefixing the regex with (?i).

Metric queries

LogQL allows a query to be wrapped with functions to count and aggregate entries per stream. These are called metric queries.

Metric queries can be used to calculate things such as the rate of error messages, or the top N log sources with the most amount of logs over the last 3 hours.

Range Vector aggregation

LogQL shares the same range vector concept from Prometheus, except the selected range of samples include a value of 1 for each log entry.

An aggregation can be applied over the selected range to transform it into an instance vector.

The currently supported functions for operating over are:

  • rate: calculates the number of entries per second
  • count_over_time: counts the entries for each log stream within the given range.
  • bytes_rate: calculates the number of bytes per second for each stream.
  • bytes_over_time: counts the amount of bytes used by each log stream for a given range.

Examples

Count all the log lines within the last five minutes for the MySQL job:

count_over_time({job="mysql"}[5m])

Get the per-second rate of all non-timeout errors within the last ten seconds for the MySQL job.

rate({job="mysql"} |= "error" != "timeout" [10s])

It should be noted that the range notation [5m] can be placed at end of the log stream filter or right after the log stream matcher. The two syntaxes below are equivalent:

rate({job="mysql"} |= "error" != "timeout" [5m])
rate({job="mysql"}[5m] |= "error" != "timeout")

Aggregation operators

Aggregation operators can be used to aggregate the element of a single vector, resulting in a new vector of fewer elements but with aggregated values:

  • sum: Calculate sum over labels
  • min: Select minimum over labels
  • max: Select maximum over labels
  • avg: Calculate the average over labels
  • stddev: Calculate the population standard deviation over labels
  • stdvar: Calculate the population standard variance over labels
  • count: Count number of elements in the vector
  • bottomk: Select smallest k elements by sample value
  • topk: Select largest k elements by sample value

The aggregation operators can either be used to aggregate over all label values or a set of distinct label values by including a without or a by clause:

<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]

Examples

Get the top 10 applications by the highest log throughput:

topk(10,sum(rate({region="us-east1"}[5m]) by (name)

Get the count of logs for the last five minutes, grouping by level:

sum(count_over_time({job="mysql"}[5m]) by (level)

Get the rate of HTTP GET requests from NGINX logs:

avg(rate({job="nginx"} |= "GET") [10s]) by (region)

Binary operators

Arithmetic Binary Operators

  • + (addition)
  • - (subtraction)
  • * (multiplication)
  • / (division)
  • % (modulo)
  • ^ (power/exponentiation)

Binary arithmetic operators are defined between two literals (scalars), a literal and a vector, and two vectors.

Between two literals, the behavior is obvious: They evaluate to another literal that is the result of the operator applied to both scalar operands (1 + 1 = 2).

Between a vector and a literal, the operator is applied to the value of every data sample in the vector, e.g. if a time series vector is multiplied by 2, the result is another vector in which every sample value of the original vector is multiplied by 2.

Between two vectors, a binary arithmetic operator is applied to each entry in the left-hand side vector and its matching element in the right-hand vector. The result is propagated into the result vector with the grouping labels becoming the output label set. Entries for which no matching entry in the right-hand vector can be found are not part of the result.

Double the rate of a a log stream’s entries:

sum(rate({app="foo"})) * 2

Get proportion of warning logs to error logs for the foo app:

sum(rate({app="foo", level="warn"}[1m])) / sum(rate({app="foo", level="error"}[1m]))

Logical/set binary operators

  • and (intersection)
  • or (union)
  • unless (complement)

vector1 and vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped.

vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.

vector1 unless vector2 results in a vector consisting of the elements of vector1 for which there are no elements in vector2 with exactly matching label sets. All matching elements in both vectors are dropped.

Comparison operators

  • == (equality)!= (inequality)
  • > (greater than)
  • >= (greater than or equal to)
  • < (less than)
  • <= (less than or equal to)

Comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs.

By default they filter.

Their behavior can be modified by providing bool after the operator, which will return 0 or 1 for the value rather than filtering.

Between two scalars, these operators result in another scalar that is either 0 (false) or 1 (true), depending on the comparison result.

The bool modifier must not be provided.

1 >= 1 is equivalent to 1

Between a vector and a scalar, these operators are applied to the value of every data sample in the vector, and vector elements between which the comparison result is false get dropped from the result vector.

If the bool modifier is provided, vector elements that would be dropped instead have the value 0 and vector elements that would be kept have the value 1.

Filters the streams which logged at least 10 lines in the last minute:

count_over_time({foo="bar"}[1m]) > 10

Attach the value(s) 0 / 1 to streams that logged less/more than 10 lines:

count_over_time({foo="bar"}[1m]) > bool 10

Between two vectors, these operators behave as a filter by default, applied to matching entries.

Vector elements for which the expression is not true or which do not find a match on the other side of the expression get dropped from the result, while the others are propagated into a result vector.

If the bool modifier is provided, vector elements that would have been dropped instead have the value 0 and vector elements that would be kept have the value 1, with the grouping labels again becoming the output label set.

Return the streams matching app=foo without app labels that have higher counts within the last minute than their counterparts matching app=bar without app labels:

sum without(app) (count_over_time({app="foo"}[1m])) > sum without(app) (count_over_time({app="bar"}[1m]))

Same as above, but vectors have their values set to 1 if they pass the comparison or 0 if they fail/would otherwise have been filtered out:

sum without(app) (count_over_time({app="foo"}[1m])) > bool sum without(app) (count_over_time({app="bar"}[1m]))

Operator order

When chaining or combining operators, you have to consider operator precedence:

Generally, you can assume regular mathematical convention with operators on the same precedence level being left-associative.

More details can be found in the Golang language documentation.

1 + 2 / 3 is equal to 1 + ( 2 / 3 ).

2 * 3 % 2 is evaluated as (2 * 3) % 2.