PCA Questions and Answers

Asingle-target exporterin Prometheus is designed to expose metrics for aspecific service instancerather than multiple dynamic endpoints. TheRedis Exporteris a prime example — it connects to one Redis server instance and exports its metrics (like memory usage, keyspace hits, or command statistics) to Prometheus.

By contrast, exporters like theSNMP ExporterandBlackbox Exportercan probe multiple targets dynamically, making themmulti-target exporters. TheNode Exporter, while often deployed per host, is considered ahost-level exporter, not a true single-target one in configuration behavior.

The Redis Exporter is instrumented specifically for a single Redis endpoint per configuration, aligning it with Prometheus’s single-target exporter definition. This design simplifies monitoring and avoids dynamic reconfiguration.

TheSilencefeature in Alertmanager allows operators tomute specific alertsfor a defined period. Each silence includes a matcher (labels), a creator, a comment, and an expiration time.

Silencing is useful during maintenance windows or known outages to prevent alert noise. Unlike inhibition, silences are manual and explicit.

Thedelta()function in PromQL calculates thedifference between the first and last samplesin a range vector over a specified time window. This function is primarily used withgauge metrics, as they can move both up and down, and delta() captures that net change directly.

For example, if a gauge metric like node_memory_Active_bytes changes from 1000 to 1200 within a 5-minute window, delta(node_memory_Active_bytes[5m]) returns 200.

Unlike rate() or increase(), which are designed for monotonically increasing counters, delta() is ideal for metrics representing resource levels, capacities, or instantaneous measurements that fluctuate over time.

Theevaluation_intervalparameter defines how frequently Prometheusevaluates its recording and alerting rules. It determines the schedule at which the rule engine runs, checking whether alert conditions are met and generating new time series for recording rules.

For example, setting:

global:

evaluation_interval: 30s

means Prometheus evaluates all configured rules every 30 seconds. This setting differs from scrape_interval, which controls how often Prometheus collects data from targets.

Having a proper evaluation interval ensures alerting latency is balanced with system performance.

Metamonitoringrefers tomonitoring the monitoring system itself—ensuring that Prometheus, Alertmanager, exporters, and dashboards are functioning properly. In other words, it’s the observability of your observability stack.

This practice helps detect issues such as:

Prometheus not scraping targets,

Alertmanager being unreachable,

Exporters not exposing data, or

Storage being full or corrupted.

Without metamonitoring, an outage in the monitoring system could go unnoticed, leaving operators blind to actual infrastructure problems. A common approach is to usea secondary Prometheus instance(or external monitoring service) to monitor the health metrics of the primary Prometheus and related components.

The metric prometheus_tsdb_lowest_timestamp_seconds provides theoldest stored sample timestampin Prometheus’s local TSDB (in Unix epoch seconds). To determine theage or approximate dateof this timestamp, you compare it with thecurrent time(using time() in PromQL).

The expression:

(time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400

converts the difference between the current time and the oldest timestamp fromseconds into days(1 day = 86,400 seconds). This gives the number of days since the earliest sample was stored, allowing you to infer the time range and approximate month manually.

The other options are invalid because PromQL does not support direct date formatting (format_date) or month() extraction functions.

ThePrometheus alerting philosophyemphasizessignal over noise— meaning alerts should focus only onactionable and user-impacting issues. The best practice is toalert on symptoms that indicate potential or actual user-visible problems, not on every internal metric anomaly.

This approach reduces alert fatigue, avoids desensitizing operators, and ensures high-priority alerts get the attention they deserve. For example, alerting on “service unavailable” or “latency exceeding SLO” is more effective than alerting on “CPU above 80%” or “disk usage increasing,” which may not directly affect users.

Option B correctly reflects this principle: keep alerts meaningful, few, and symptom-based. The other options contradict core best practices by promoting excessive or equal-weight alerting, which can overwhelm operations teams.

To calculate a percentile (e.g., 95th percentile) from histogram data in Prometheus, the correct function ishistogram_quantile(). It estimates quantiles based on cumulative bucket counts.

Example:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

This computes the 95th percentile request duration across all observed instances over the last 5 minutes.

Theavg_over_time()function calculates theaverage value of a time series over a specified range vector. It is used to measure how a gauge metric (like available filesystem bytes) behaves over time rather than at a single instant.

For example:

avg_over_time(node_filesystem_avail_bytes[1h])

This query returns theaverage amount of available filesystem spaceobserved across all samples within the last hour for each time series.

By contrast:

avg() performs aggregation acrossdifferent series at a single point, not over time.

sum() and sum_over_time() compute totals rather than averages.

Thus, only avg_over_time() provides the correct temporal average.

In Prometheus alerting rules, theforfield specifies how long a condition must remaintrue continuouslybefore the alert transitions from thependingto thefiringstate. This feature prevents transient spikes or brief metric fluctuations from triggering false alerts.

Example:

alert: HighRequestLatency

expr: http_request_duration_seconds_avg > 1

for: 5m

labels:

severity: warning

annotations:

description: "Request latency is above 1s for more than 5 minutes."

In this configuration, Prometheus evaluates the expression every rule evaluation cycle. The alert only fires if the condition (http_request_duration_seconds_avg > 1) remains true for5 consecutive minutes. If it returns to normal before that duration, the alert resets and never fires.

This mechanism addsstability and noise reductionto alerting systems by ensuring only sustained issues generate notifications.

Therate()function computes theper-second increaseof a counter metric over a specified range, whilesum by (label)aggregates those rates across dimensions — in this case, the cloud label.

The correct query is:

sum by (cloud)(rate(apiserver_request_total{job="kube-apiserver"}[5m]))

This expression:

Calculates the rate of increase in API requests per second for each instance.

Groups and sums those rates by cloud, giving the total request rate per cloud provider.

Option A incorrectly places by (cloud) after rate(), which is not valid syntax.

Option B returns raw counter totals (not rates).

Option D incorrectly applies rate() after aggregation, which distorts the calculation since rate() must operate on individual time series before aggregation.

Following Prometheus’smetric naming conventions, every metric should indicate:

Whatit measures (the quantity or event).

The unitof measurement inbase SI unitsas a suffix.

Since the metric measuresresponse size, the base unit isbytes. Therefore, the correct and compliant metric name is:

grpc_response_size_bytes

This clearly communicates that it measures gRPC response payload sizes expressed in bytes.

The _bytes suffix is the Prometheus-recommended unit indicator for data sizes. The other options violate naming rules:

_total is reserved for counters.

_sum is used internally by histograms or summaries.

Omitting the unit (grpc_response_size) is discouraged, as it reduces clarity.

Deduplicationin Alertmanager ensures thatidentical alerts from multiple Prometheus serversor rule evaluations do not trigger duplicate notifications.

Alertmanager compares alerts based on their labels and fingerprints; if an alert with identical labels already exists, it merges or refreshes the existing one instead of creating a new notification.

This mechanism is essential in high-availability setups where multiple Prometheus instances monitor the same targets.