Prometheus monitoring has become the default way to collect metrics in cloud-native environments, but its architecture can feel opaque if you are coming from a traditional monitoring background. The pull-based model, the four metric types, the separation between server and Alertmanager, each piece makes sense in isolation but the overall picture takes time to assemble. Teams often adopt Prometheus because Kubernetes integration expects it, then discover they need to understand cardinality, retention, and scaling trade-offs that are not obvious from a quick start guide. This guide breaks down how Prometheus monitoring works, what its core components do, and where it runs into limits at scale.
What is Prometheus?
Prometheus is an open-source monitoring and alerting toolkit originally built at SoundCloud in 2012. It was the second project to graduate from the Cloud Native Computing Foundation (CNCF) after Kubernetes, and it has since become the de facto standard for metrics collection in cloud-native infrastructure.
At its core, Prometheus is a time-series database with a query language called PromQL. It stores numerical samples as timestamped data points, each identified by a metric name and a set of key-value labels. You query those samples with PromQL to build dashboards, trigger alerts, or investigate incidents.
Prometheus only handles metrics. It does not collect logs or distributed traces. Teams that need all three signals either run separate tools for each (Prometheus for metrics, Loki or ELK for logs, Jaeger or Zipkin for traces) or consolidate onto a full-stack observability platform.
How Prometheus monitoring works
Prometheus uses a pull-based model. Instead of waiting for applications to push metrics to it, the Prometheus server actively reaches out to each target on a regular schedule and scrapes its current metrics over HTTP. Every target exposes a /metrics endpoint that returns data in a text-based format, and Prometheus parses that response into its time-series database.
The scrape interval is configurable and defaults to 15 seconds in most production setups. A failed scrape also doubles as a health check: if Prometheus cannot reach a target, the up metric for that target flips to 0, which makes it easy to alert on missing targets without additional instrumentation.
This pull model was a deliberate design choice. It gives Prometheus control over collection timing, makes debugging straightforward (you can curl any /metrics endpoint yourself), and avoids the need for agents to know where to send data. The trade-off is that ephemeral jobs, like cron tasks or batch scripts that exit before Prometheus can scrape them, need a workaround. The Prometheus Pushgateway exists for this case: short-lived jobs push their metrics to the Pushgateway, and Prometheus scrapes the Pushgateway as a long-lived target.
In dynamic environments like Kubernetes, service discovery replaces static target lists. Prometheus connects to the Kubernetes API and automatically discovers new pods and services as they appear, then starts scraping them without any manual configuration changes.
Once metrics are stored, you query them with PromQL. It supports instant queries (what is the value right now), range queries (what did this look like over the last hour), and label-based filtering with aggregation operators like sum, avg, and topk. The two functions you will use constantly are rate(), which turns a raw counter into a per-second rate while handling counter resets, and histogram_quantile(), which calculates percentiles like p99 latency from histogram buckets at query time. A typical request rate query looks like sum(rate(http_requests_total[5m])) by (status_code).
Prometheus architecture components
The Prometheus monitoring stack is built from several independent components. Each one has a specific job, and understanding the boundaries between them is key to operating the stack effectively.
The Prometheus server
The server is the central component. It is a single Go binary that handles three things: scraping targets on a schedule, storing the collected samples in its local time-series database, and evaluating alerting and recording rules against the stored data. The server exposes a web UI on port 9090 where you can run ad-hoc PromQL queries, check scrape target status, and view alert states.
Local storage has no built-in clustering or replication. Data lands in a write-ahead log first, then gets compacted into two-hour blocks on disk. The default retention period is 15 days, and extending it increases disk and memory requirements proportionally.
Exporters
Most systems do not expose a /metrics endpoint natively. Exporters are helper applications that run alongside the system you want to monitor, collect its raw metrics, translate them into the Prometheus text format, and expose them on a /metrics endpoint for the server to scrape.
Common exporters include node_exporter for Linux host metrics (CPU, memory, disk, network), blackbox_exporter for probing HTTP and TCP endpoints, and database-specific exporters like postgres_exporter or mysqld_exporter. For your own applications, official client libraries in Go, Java, Python, Ruby, and Rust let you define custom metrics directly in code.
Service discovery
Static target lists break down in dynamic environments where pods and containers are constantly created and destroyed. Service discovery solves this by letting Prometheus query your platform’s API for an up-to-date list of targets. Prometheus ships with native support for Kubernetes, Consul, AWS EC2, and file-based discovery, among others.
When you deploy a new service in Kubernetes, the platform automatically tells Prometheus about it, and scraping begins immediately. When a pod is terminated, Prometheus stops trying to scrape it. No manual configuration changes needed.
Alertmanager
Prometheus evaluates alerting rules and decides when an alert should fire, but it does not handle notifications itself. That job belongs to Alertmanager, a separate component that receives alerts from the Prometheus server and manages their delivery.
Alertmanager handles deduplication (so you do not get the same alert from every replica of a service), grouping (so related alerts collapse into one notification), routing (so alerts go to the right team or channel), and silencing (so you can suppress alerts during maintenance windows). It integrates with Slack, email, PagerDuty, and other notification systems.
Prometheus metric types
Prometheus defines four metric types. Choosing the right one matters because using the wrong type produces data that behaves incorrectly under aggregation, and the problem often does not surface until you try to query across multiple instances.
Every time series in Prometheus is identified by a metric name plus a set of key-value labels. The number of unique label combinations is what people mean by cardinality, and it is the single most important thing to watch in production. A label like user_id or request_uuid can turn a metric with a few hundred series into millions overnight, which overwhelms the server’s memory and storage. Always use labels for values with a low and finite number of possibilities.
Counter
A counter is a cumulative metric that only increases or resets to zero on application restart. Use it for things like total requests served, errors occurred, or bytes transmitted. The raw value of a counter is rarely useful by itself. You typically apply the rate() function to convert it into a per-second rate, which handles counter resets automatically and gives you meaningful throughput numbers.
Gauge
A gauge represents a single numerical value that can go up and down at any time. It is the right choice for measuring point-in-time states like current memory usage, active connections, or queue depth. Never apply rate() to a gauge. Instead, use functions like max_over_time(), min_over_time(), or avg_over_time() depending on what you need.
Histogram
A histogram samples observations into configurable cumulative buckets. It exposes a _count of observations, a _sum of their values, and a series of _bucket counters for each configured threshold. Histograms are aggregatable, which means you can reliably combine them from thousands of application instances and use histogram_quantile() in PromQL to calculate percentiles like p99 latency across your entire fleet. This makes histograms the right choice for any latency measurement in a distributed system.
Summary
A summary also samples observations, but it calculates quantiles on the client side and exposes them as pre-computed values. The critical limitation is that summary quantiles cannot be aggregated across instances. You cannot average the p99 from server A with the p99 from server B to get a meaningful system-wide p99. Most teams avoid summaries in production for this reason and use histograms instead, unless they need exact quantiles from a single process.
Common use cases for Prometheus monitoring
Kubernetes monitoring
Kubernetes is the dominant use case for Prometheus. The standard stack combines several components that cover different layers of the cluster. node_exporter collects host-level metrics from each node. kube-state-metrics exposes orchestration data like pod status and deployment replica counts. cAdvisor reports container-level resource usage. The Prometheus Operator automates deployment and makes scrape configurations declarative through Custom Resource Definitions like ServiceMonitor.
During a memory leak investigation, for example, cAdvisor shows rising memory on a specific pod, kube-state-metrics confirms the pod has restarted multiple times, and node_exporter verifies the underlying node is healthy. That triangulation points the investigation directly at the application code.
Infrastructure and application monitoring
Outside Kubernetes, node_exporter is usually the first thing teams deploy. It pairs well with the predict_linear() PromQL function for capacity planning, which projects when a disk will fill based on current growth so you can alert weeks ahead instead of getting paged the night it runs out.
For application monitoring, client libraries let you instrument your code directly. A practical baseline is to track query count, errors, and latency for every external dependency your service calls. That three-signal pattern pays off across every future incident.
Prometheus limitations
Prometheus was built as a near-real-time metrics system, not a cross-stack observability backend. That focus creates three limitations that widen as your deployment grows.
Long-term storage and retention. Prometheus stores data locally with a default retention of 15 days. Local storage has no clustering or replication, so teams that need months or years of retention typically send data to a remote storage backend using the remote_write protocol. Open-source projects like Thanos, Grafana Mimir, and VictoriaMetrics dominate this space, each offering long-term storage and a global query interface across multiple Prometheus instances.
Metrics only, no logs or traces. Prometheus handles metrics exclusively. Correlating a latency spike with the specific error log and distributed trace that caused it requires separate systems with their own query languages. On-call engineers end up pivoting between tools during an incident, mentally stitching together context that lives in different places.
Single-node architecture. A single Prometheus server has no built-in horizontal scaling or cross-instance querying. Federation and agent mode with remote_write forward metrics to centralized backends, but both approaches add architectural complexity. As scrape targets multiply and cardinality grows, a single server can run out of memory, and the mitigation (sharding across multiple instances) introduces operational overhead.
For teams that want metrics and logs in one place without self-hosting a Prometheus stack, Simple Observability provides a managed alternative. A single-command agent auto-detects services and centralizes both metrics and logs into one dashboard, with built-in alerting and no YAML configuration.
Conclusion
Prometheus is the right starting point for cloud-native metrics monitoring. Its pull model, multi-dimensional data model, and PromQL give you a solid foundation for understanding system behavior over time. The limitations around retention, metrics-only scope, and single-node scaling are real but manageable if you understand them before committing to self-hosting at scale. If your team does not have the engineering bandwidth to operate Thanos or Mimir alongside Prometheus, a managed platform that handles storage, scaling, and cross-signal correlation is a practical alternative.