Pitfalls of graphing topk queries from Prometheus or Loki within Grafana
The topk()
function in Prometheus and Loki returns the topk per interval.
This means that at time T1, the topk series returned is a different set as that on T2.
As a result of this, the graph in Grafana may contain more series in total than what you would expect.
As of Grafana 5.3.0 there’s a feature that allows correct graphing of the top N series over a duration using the $__range_s
variable, as explained in Graph top N time series in Grafana.
At the moment of writing, this is not supported for queries targeting Loki however (issue #25561).