Apache Flink monitoring, done right
The Flink metrics that actually predict incidents, the tools to collect them, and how to turn them into alerts your team trusts.
How to monitor Apache Flink — the short answer
Apache Flink exposes hundreds of metrics through its metric system. You can read them live in the built-in Flink Web UI (port 8081), query them through the REST API, or export them with a metric reporter such as Prometheus. Production monitoring means keeping their history and alerting on a small set of signals: job restarts, checkpoint failures, backpressure, consumer lag, and TaskManager / JVM health. Teams either build this themselves with Prometheus and Grafana, or use a Flink monitoring console such as Flink Manager by Vinteros Labs, which ships metric history, thresholds, and notifications out of the box.
The Flink metrics that matter
| Area | Flink metrics | Why it matters | Alert when |
|---|---|---|---|
| Job availability |
| A job that keeps restarting is failing silently — the Web UI only shows it as RUNNING again. | numRestarts increases, or downtime > 0 for more than a few minutes. |
| Checkpointing |
| Checkpoints are your recovery point. Slow or failing checkpoints mean long recovery and growing state. | Any failed checkpoint, or duration approaching the checkpoint timeout. |
| Backpressure |
| Shows which operator is the bottleneck. A busy operator downstream of back-pressured ones is the culprit. | Sustained high backPressuredTimeMsPerSecond on a source or operator. |
| Throughput |
| Drops to zero reveal stalled sources; spikes reveal replays after recovery. | Throughput at zero while the source still has data. |
| Lag & freshness |
| Consumer lag and watermark lag tell you how far behind real time your results are. | pendingRecords keeps growing, or watermark stops advancing. |
| Cluster & JVM |
| Lost TaskManagers, exhausted slots, and GC pressure precede most job failures. | TaskManager count drops, no free slots, or GC time climbs steadily. |
Apache Flink monitoring tools
1. Built-in Flink Web UI
Live per-job view on port 8081: checkpoints, backpressure, subtask metrics, and exceptions. No history beyond a short window and no alerting, for one cluster.
2. Flink REST API
The same data as JSON — e.g. GET /jobs/overview, GET /jobs/<jobid>/checkpoints, GET /taskmanagers — for scripts and custom health checks.
3. Metric reporters (Prometheus, JMX, …)
Flink's metric system exports every metric to an external backend. The Prometheus reporter exposes an endpoint (port 9249 by default) for scraping.
4. Grafana + Alertmanager
Dashboards and alert rules on top of Prometheus. Powerful, but every panel, threshold, and alert is yours to build and maintain.
5. Flink Manager
A Flink UI with monitoring built in: metric history collected from JMX/Prometheus reporters, a metric dictionary with warning/critical thresholds, cluster health widgets, and persistent notifications — next to SQL, jobs, and RBAC.
Enable the Prometheus reporter
Add these lines to your Flink configuration and restart the JobManager and TaskManagers. Prometheus can then scrape each process on port 9249:
metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
metrics.reporter.prom.port: 9249Flink Manager reads the same reporters, so enabling Prometheus or JMX once gives you both: Grafana for infrastructure teams and a Flink-aware console for the people who operate the jobs.
Flink monitoring best practices
- Alert on symptoms users feel — lag, restarts, failed checkpoints — before resource metrics.
- Keep metric history: most Flink incidents are trends (growing state, slowing checkpoints) that a live dashboard cannot show.
- Use per-metric warning and critical thresholds instead of a single alert level, to avoid alert fatigue.
- Find bottlenecks with backpressure and busy time together, not CPU alone.
- Correlate metrics with change events — deployments, savepoints, and configuration changes — using an audit trail.
- Monitor every cluster from one place: per-cluster dashboards hide cross-cluster capacity problems.
Comparing consoles? See Flink UI alternatives compared or read what a Flink UI is.
Apache Flink monitoring — FAQ
- Use Flink's metric system. For a live view, open the built-in Flink Web UI on the JobManager (port 8081). For history and alerting, enable a metric reporter such as the Prometheus reporter and visualize it in Grafana, or use a Flink monitoring console such as Flink Manager by Vinteros Labs, which stores metric history, applies warning/critical thresholds, and raises notifications.
- Job restarts (numRestarts), checkpoint health (lastCheckpointDuration, numberOfFailedCheckpoints), backpressure (backPressuredTimeMsPerSecond, busyTimeMsPerSecond), throughput (numRecordsInPerSecond, numRecordsOutPerSecond), lag (pendingRecords, watermarks), and cluster/JVM health (registered TaskManagers, free task slots, heap usage, GC time).
- Add metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory and metrics.reporter.prom.port: 9249 to your Flink configuration, let Prometheus scrape the JobManager and TaskManagers on that port, then build Grafana dashboards and Alertmanager rules on the collected metrics.
- Open the job in the Flink Web UI and check the BackPressure tab, or watch the backPressuredTimeMsPerSecond and busyTimeMsPerSecond task metrics. The bottleneck is usually the first operator that is busy (close to 1000 ms/s) while the operators upstream of it are back-pressured.
- Yes. Flink Manager by Vinteros Labs is a Flink UI with monitoring built in: it collects metric history from Flink's JMX or Prometheus reporters, lets you set warning and critical thresholds per metric, and raises persistent job and system notifications — without building Grafana dashboards by hand.
Monitor Apache Flink without building dashboards
Flink Manager gives you metric history, thresholds, and notifications next to SQL, jobs, and RBAC. Free trial license — no credit card required.
