Metrics

This interface will serve Prometheus metrics from /metrics. The following metrics are included by default, others are transform specific.

NameLabelsData typeDescription
shotover_transform_total_counttransformcounterCounts the amount of times the transform is used
shotover_transform_failures_counttransformcounterCounts the amount of times the transform fails
shotover_transform_latency_secondstransformhistogramThe latency for a message batch to go through the transform
shotover_transform_pushed_latency_secondstransformhistogramThe latency for a pushed message from the DB to go through the transform
shotover_chain_total_countchaincounterCounts the amount of times chain is used
shotover_chain_failures_countchaincounterCounts the amount of times chain fails
shotover_chain_latency_secondschainhistogramThe latency for running chain
shotover_chain_messages_per_batch_countchainhistogramThe number of messages in each batch passing through chain.
shotover_available_connections_countsourcegaugeThe number of connections currently connected to source
shotover_source_to_sink_latency_secondssinkhistogramThe milliseconds between reading a request from a source TCP connection and writing it to a sink TCP connection
shotover_sink_to_source_latency_secondssourcehistogramThe milliseconds between reading a response from a sink TCP connection and writing it to a source TCP connection

Metric data types

Counter

A single value, which can only be incremented, not decremented. Starts out with an initial value of zero.

Histogram

Measures the distribution of values for a set of measurements and starts with no initial values.

Every 20 seconds one of the 3 chunks of historical values are cleared. This means that values are held onto for around 60 seconds.

Gauge

A single value that can increment or decrement over time. Starts out with an initial value of zero.

Log levels and filters

You can configure log levels and filters at /filter. This can be done by a POST HTTP request to the /filter endpoint with the env_filter string set as the POST data. For example:

curl -X PUT -d 'info, shotover_proxy=info, shotover::connection_span::info` http://127.0.0.1:9001/filter

Some examples of how you can tweak this filter:

  • configure the first info to set the log level for dependencies
  • configure shotover=info to set the log level for shotover itself
  • set shotover::connection_span=info to shotover::connection_span=debug to attach connection info to most log events, this is disabled by default due to a minor performance hit.

For more control over filtering you should understand The tracing filter format.