Introduction

Ceph is the backbone of most hyper-converged Proxmox VE deployments, giving administrators shared, redundant block storage without a dedicated SAN. It works well when it works, but when Object Storage Daemons (OSDs) start flapping — cycling between up and down, or dropping out of the cluster map entirely — the effects ripple across every VM and container that depends on RBD-backed storage. Guests stall, migrations fail, and the Proxmox VE web UI starts throwing HEALTH_WARN or HEALTH_ERR banners that many administrators have learned to dread.

This article walks through a complete, field-tested process for diagnosing and resolving flapping or down OSDs in a Proxmox VE cluster running Ceph, from Proxmox VE 9.2 with Ceph Squid 19.2.3 or Ceph Tentacle 20.2.1 back through Proxmox VE 8.x clusters still on Ceph Reef or Quincy. The underlying troubleshooting method is version-independent: identify the failing daemons, understand why the monitors are marking them down, stabilize the cluster, then fix the root cause instead of just restarting services and hoping it does not happen again.

The audience for this guide is anyone running Ceph inside Proxmox VE in production or in a serious homelab: system administrators keeping a cluster healthy day to day, DevOps and infrastructure engineers who inherited a hyper-converged environment, MSPs managing storage for multiple clients, and homelab users who want to understand what is actually happening under the hood instead of just clicking "restart" in the GUI.

Problem Overview

In a Proxmox VE Ceph cluster, each physical disk assigned to Ceph runs as an independent OSD daemon (ceph-osd@<id>.service) on its host node. The Ceph monitors (MONs) track the state of every OSD in the cluster map and use peer reports — OSDs pinging each other over the cluster network — combined with periodic heartbeats to decide whether an OSD is up or down, and in or out of the CRUSH map.

An OSD is considered "flapping" when it repeatedly transitions between up and down within a short window. A single OSD going down once is often benign — a disk hiccup, a service restart during a package upgrade — but repeated flapping is a symptom of a deeper problem: an overloaded cluster network, a failing disk, insufficient memory for the OSD process, or misconfigured MTU on the storage VLAN. Left unaddressed, flapping triggers constant peering and rebalancing, which consumes I/O bandwidth that guest VMs and containers need, degrading overall cluster performance well beyond the single failing disk.

A persistently down OSD (rather than flapping) usually points to a harder failure: a dead or dying drive, a corrupted OSD data directory, or a service that will not start due to a configuration or permission problem. Both scenarios require different diagnostic paths, which this article separates clearly.

Symptoms

The following symptoms typically show up together or in sequence when OSDs are flapping or down in a Proxmox VE Ceph cluster:

  • The Datacenter → Ceph panel in the Proxmox VE web UI shows a HEALTH_WARN or HEALTH_ERR status instead of HEALTH_OK.
  • One or more OSDs show a red or grey status indicator under Datacenter → Ceph → OSD, cycling between "up" and "down" over minutes or hours.
  • VM and container disk I/O latency spikes, sometimes severely enough that guests appear to hang for several seconds.
  • ceph -s reports placement groups (PGs) in peering, degraded, undersized, or backfilling states that never fully resolve.
  • The cluster log (ceph -w or journalctl -u ceph-mon@$(hostname)) shows repeated osd.X marked down and osd.X boot entries for the same OSD ID.
  • Slow ops warnings appear: N slow requests are blocked or N osds have slow ops.
  • Live migrations or backup jobs targeting Ceph-backed storage time out or fail with I/O errors.
  • SMART or drive health alerts correlate in time with the OSD state changes, when the cause is a failing disk rather than networking.

Root Cause

Flapping and down OSDs in Proxmox VE Ceph clusters trace back to a handful of recurring root causes:

Cluster network saturation or misconfiguration

Ceph OSDs exchange heartbeats over the cluster network (if a separate one is configured) and the public network. If that network is saturated by backfill/recovery traffic, VM traffic sharing the same physical NIC, or an MTU mismatch (for example jumbo frames enabled on the switch but not on the Proxmox VE bridge, or the reverse), heartbeat packets get delayed or dropped. The monitors then mark the OSD down even though the daemon itself is healthy, and it flaps back to up as soon as a heartbeat gets through.

Failing or overloaded physical disks

A drive nearing end of life produces slow or failed I/O, which causes the OSD daemon to miss internal timeouts and either crash or become unresponsive to heartbeats. Consumer-grade SSDs without power-loss protection are a particularly common cause in homelab and budget deployments because they perform poorly under Ceph's synchronous write pattern.

Insufficient memory or CPU contention

Each OSD process has a memory footprint governed by osd_memory_target (4 GiB by default). On nodes running many OSDs alongside memory-hungry VMs, the kernel OOM killer can terminate ceph-osd processes under memory pressure, which shows up as a sudden, unexplained OSD crash.

Clock skew between nodes

Ceph monitors are sensitive to time drift. If NTP/chrony is not properly synchronized across cluster nodes, Ceph reports clock skew warnings and can misjudge heartbeat timing, contributing to spurious down marks.

Scheduled SMART scans or filesystem trims

Periodic SMART self-tests or aggressive TRIM/discard operations on the underlying block devices can briefly stall an OSD's I/O path long enough to miss a heartbeat window, producing flapping that correlates with a fixed time of day.

Software or firmware bugs after an upgrade

Mixed Ceph versions during a rolling upgrade, or a drive firmware/HBA driver regression introduced by a kernel update, can produce slow ops and flapping that only appears after a Proxmox VE or Ceph version change.

Diagnosis

Work through the following checks in order — each one narrows down the root cause before you touch any configuration.

1. Confirm overall cluster health

ceph -s
ceph health detail

ceph health detail lists exactly which OSDs, PGs, or daemons are implicated, instead of just the summary line.

2. Identify which OSDs are affected

ceph osd tree
ceph osd df tree

ceph osd tree shows the CRUSH hierarchy with the current up/down and in/out state per OSD, grouped by host. ceph osd df tree adds utilization percentages, which is useful for spotting a near-full OSD that Ceph may be deprioritizing.

3. Check the OSD's own logs

journalctl -u ceph-osd@<id>.service --since "1 hour ago"

Look for I/O errors referencing the underlying block device, out-of-memory kills, or repeated heartbeat_check: no reply from messages that name a specific peer OSD — the latter points to a network problem rather than a local disk problem.

4. Check monitor logs for heartbeat failures

journalctl -u ceph-mon@$(hostname).service --since "1 hour ago" | grep -i "marked down"

This tells you which OSD was marked down, by which peers it was reported, and how often — a single OSD repeatedly reported down by peers on a different host strongly suggests a network path issue between those two hosts.

5. Test the cluster network directly

ping -M do -s 8972 <peer-cluster-ip>
iperf3 -c <peer-cluster-ip>

The ping -M do -s 8972 test validates a 9000-byte jumbo frame path end to end (8972 bytes of payload plus 28 bytes of ICMP/IP header equals 9000). A failure here with jumbo frames enabled is a very common cause of OSD flapping that has nothing to do with the disks themselves.

6. Check for clock skew

ceph time-sync-status
chronyc tracking

7. Check underlying disk health

smartctl -a /dev/sdX
nvme smart-log /dev/nvme0n1

Look at reallocated sector counts, media errors, and, on NVMe, the critical_warning and percentage_used fields.

8. Check for slow ops and their source

ceph daemon osd.<id> dump_historic_slow_ops

This shows the actual operations that exceeded the slow-op threshold, including which stage of processing they were stuck in — useful for distinguishing a slow disk from a slow network from monitor contention.

Step-by-Step Solution

Step 1 — Stabilize the cluster before investigating further

If OSDs are actively flapping, stop the churn first so recovery traffic does not make diagnosis harder and does not consume bandwidth other guests need:

ceph osd set noout
ceph osd set nodown

noout prevents Ceph from permanently marking flapping OSDs as out of the CRUSH map (which would trigger a full data rebalance), and nodown temporarily freezes OSD up/down state so you can work without the map changing under you. Both are temporary diagnostic flags — do not leave them set indefinitely, since nodown in particular hides genuine failures from the cluster.

Step 2 — Resolve a confirmed network cause

If the jumbo frame test in the Diagnosis section failed, align the MTU across every hop: the physical switch ports, the Linux bond, the VLAN interface, and the Proxmox VE bridge used for the Ceph cluster network. Edit /etc/network/interfaces on each node consistently and reload networking, then re-run the ping -M do -s 8972 test before moving on. If the cluster network is sharing a physical NIC with VM traffic or the Corosync network, separate them onto dedicated links or VLANs — Ceph heartbeat traffic is latency-sensitive and should not compete with bulk VM traffic.

Step 3 — Resolve a confirmed disk cause

If smartctl or NVMe health data shows a failing drive, do not attempt to nurse it back — plan an OSD replacement. First mark it out and let the cluster rebalance safely:

ceph osd out <id>
ceph osd ok-to-stop <id>

Wait for ceph osd ok-to-stop to confirm the OSD can be stopped without dropping below your pool's minimum replica count, then stop and destroy it cleanly:

pveceph stop --service osd.<id>
pveceph osd destroy <id> --cleanup

Physically replace the drive, wipe any residual LVM/Ceph signatures, and create the new OSD through the Proxmox VE web UI or CLI as described in the Commands section below.

Step 4 — Resolve a confirmed memory cause

If OOM kills are visible in dmesg or journalctl -k, either add memory to the node, reduce the number of OSDs per node, or lower the per-OSD memory target so the aggregate footprint fits available RAM with headroom for VMs:

ceph config set osd osd_memory_target 3221225472

This example sets a 3 GiB target (in bytes) instead of the 4 GiB default — only do this on nodes that are genuinely memory-constrained, since a lower target reduces the OSD's read cache and can hurt performance.

Step 5 — Resolve clock skew

Ensure chrony is installed and enabled on every node, pointed at the same time sources (ideally an internal NTP server for consistency):

systemctl enable --now chrony
chronyc sources

Step 6 — Restart a genuinely stuck OSD

Once the underlying cause is addressed, if an OSD is still down but the disk and network are healthy, restart the daemon:

systemctl restart ceph-osd@<id>.service
systemctl status ceph-osd@<id>.service

Step 7 — Unset the diagnostic flags

Once OSDs report stable and PGs are recovering normally, remove the temporary flags:

ceph osd unset noout
ceph osd unset nodown

Step 8 — Confirm full recovery

ceph -s
ceph osd tree
ceph -w

Watch ceph -w for a period after unsetting the flags to make sure PGs return to active+clean and no OSD resumes flapping.

Commands

The table below summarizes the commands used throughout this workflow, grouped by purpose.

PurposeCommand
Cluster health summaryceph -s
Detailed health issuesceph health detail
CRUSH tree with OSD stateceph osd tree
OSD utilizationceph osd df tree
OSD service logsjournalctl -u ceph-osd@<id>.service
Monitor logsjournalctl -u ceph-mon@$(hostname).service
Freeze out-markingceph osd set noout
Freeze down-markingceph osd set nodown
Remove freeze flagsceph osd unset noout / ceph osd unset nodown
Mark OSD outceph osd out <id>
Check safe to stopceph osd ok-to-stop <id>
Stop OSD via Proxmox VEpveceph stop --service osd.<id>
Destroy OSD and wipepveceph osd destroy <id> --cleanup
Create new OSDpveceph osd create /dev/sdX
Restart OSD daemonsystemctl restart ceph-osd@<id>.service
Disk health (SATA/SAS)smartctl -a /dev/sdX
Disk health (NVMe)nvme smart-log /dev/nvme0n1
Slow-op detailceph daemon osd.<id> dump_historic_slow_ops
Time sync statusceph time-sync-status
Set OSD memory targetceph config set osd osd_memory_target <bytes>

Configuration Examples

The following excerpt from /etc/pve/ceph.conf shows a healthy separation between the public and cluster networks, which is the single most effective preventive measure against heartbeat-related flapping:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.10.20.0/24
     public_network = 10.10.10.0/24
     fsid = 3c undefined-uuid
     mon_allow_pool_delete = false
     mon_host = 10.10.10.11 10.10.10.12 10.10.10.13
     ms_bind_ipv4 = true
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3

Corresponding network interface configuration in /etc/network/interfaces for the dedicated Ceph cluster link, with jumbo frames enabled consistently:

auto ens1f0
iface ens1f0 inet manual
     mtu 9000

auto vmbr1
iface vmbr1 inet static
     address 10.10.20.11/24
     bridge-ports ens1f0
     bridge-stp off
     bridge-fd 0
     mtu 9000

A per-node override for a memory-constrained host, added under Datacenter → Ceph → Configuration or directly via ceph config set, targeting only that host's OSDs:

ceph config set osd/host:pve-node03 osd_memory_target 3221225472

An example crontab-free approach to avoiding SMART-scan-induced flapping — disabling automatic offline SMART tests on Ceph OSD disks where periodic scans overlap with backup windows:

smartctl -s off -o off /dev/sdX

Run this only after confirming your monitoring stack polls SMART data another way (for example through node-exporter's smartctl_exporter), so you do not lose visibility into drive health entirely.

Common Mistakes

  • Leaving noout or nodown set permanently. These flags are diagnostic tools, not a fix. A genuinely failed OSD left frozen with nodown will silently serve stale data reads until it is properly addressed.
  • Restarting the OSD service repeatedly without checking logs. A restart can mask a symptom for a few minutes while the underlying disk or network problem continues to degrade.
  • Enabling jumbo frames on only some hops. Partial MTU 9000 configuration (switch configured, Proxmox VE bridge left at 1500, or vice versa) causes silent packet fragmentation or drops that are much harder to diagnose than a fully consistent MTU everywhere.
  • Running the Ceph public and cluster network over the same physical link as VM traffic without QoS. Backup jobs or a noisy VM can starve Ceph heartbeats during peak load.
  • Ignoring near-full OSD warnings until a pool hits the full ratio. A cluster running above roughly 85% utilization on any OSD is far more prone to slow ops and rebalance-triggered flapping.
  • Mixing consumer SSDs without power-loss protection into a production Ceph pool. These drives perform Ceph's synchronous writes an order of magnitude slower than datacenter-grade SSDs, producing chronic slow ops that look like flapping.
  • Replacing a disk without first confirming ceph osd ok-to-stop. Pulling an OSD that is not safe to stop can drop a PG below its minimum replica size and cause temporary I/O unavailability for guests using that data.
  • Not correlating OSD flap timestamps with cron jobs. SMART self-tests, ZFS scrubs on unrelated pools, and backup schedules often explain flapping that otherwise looks random.

Best Practices

  • Use a dedicated, physically separate network (or at minimum a dedicated VLAN with guaranteed bandwidth) for the Ceph cluster network, isolated from VM and Corosync traffic.
  • Keep MTU consistent end to end if using jumbo frames, and validate it with ping -M do -s 8972 after any switch or NIC change.
  • Size osd_memory_target and node RAM together: budget roughly 4–5 GiB of RAM per OSD at default settings, plus whatever guests on that node require.
  • Run chrony on every node against a consistent, low-latency time source, and alert on clock skew warnings rather than discovering them during an incident.
  • Monitor SMART/NVMe health continuously (via Proxmox VE's built-in disk health view or an external tool) instead of waiting for a drive to trigger flapping before investigating.
  • Keep OSD utilization below the nearfull threshold (85% by default) through capacity planning, and add capacity proactively rather than reactively.
  • Stagger firmware, kernel, and Ceph version upgrades across a maintenance window per node, verifying ceph -s returns to HEALTH_OK between nodes during a rolling upgrade.
  • Document your CRUSH map's failure domains (host, rack, or datacenter) so that OSD replacement and rebalancing behavior is predictable and reviewed, not improvised during an incident.
  • Use ceph osd ok-to-stop as a standard pre-check before any planned maintenance that stops an OSD, not just during emergencies.
  • Keep pool size/min_size settings (commonly 3/2) consistent with your actual failure-domain tolerance, and review them after any change to cluster size.

FAQ

Why does the Proxmox VE web UI show an OSD as down even though the service is running?

The web UI reflects the Ceph monitor's view of OSD state, not the local systemd unit state directly. A running ceph-osd process can still be marked down by the monitors if its heartbeats are not reaching peers or monitors in time — check ceph osd tree and the monitor logs rather than only systemctl status.

Is it safe to set noout during routine node maintenance?

Yes — setting noout before rebooting a single node for maintenance is standard practice; it prevents Ceph from starting an unnecessary rebalance while the node is briefly offline. Unset it once the node and its OSDs are back online.

How many OSDs can I lose before I lose data with the default pool settings?

With the common size = 3, min_size = 2 configuration, the pool tolerates the loss of one replica (one OSD, or one full host if your CRUSH failure domain is host-based) while remaining fully readable and writable, and can tolerate a second loss without immediate data loss, though it will stop serving writes for affected PGs until min_size is met again.

Should I use ceph osd down to force an OSD offline for maintenance?

No — ceph osd down <id> only flags the OSD as down in the map; the daemon keeps running and will typically report itself back up within seconds. To take an OSD out of service for maintenance, mark it out and stop the service with pveceph stop --service osd.<id>.

Can flapping OSDs cause VM corruption?

Ceph's replication and consistency guarantees mean that flapping alone should not corrupt VM disk data — writes are acknowledged only once they meet the pool's min_size. However, if flapping progresses to a point where PGs drop below min_size, guests will see I/O stalls or errors rather than corruption, since Ceph blocks writes it cannot safely replicate.

Why did flapping start right after I added a new node to the cluster?

Adding a node triggers CRUSH map changes and data rebalancing, which significantly increases traffic on the cluster network. If that network was already close to saturated, the added rebalance traffic is often enough to push heartbeat latency past the failure threshold. Check bandwidth utilization on the cluster network during the rebalance window.

Conclusion

Flapping or down OSDs are rarely a Ceph problem in isolation — they are almost always a symptom of a network, disk, memory, or timing issue that Ceph is simply the first system to notice and report loudly. The diagnostic sequence in this article — confirm cluster health, identify affected OSDs, check logs at both the OSD and monitor level, validate the network path, check disk health, and check for clock skew — will identify the root cause in the large majority of real-world cases without guesswork.

Once you have identified the cause, resist the temptation to just restart the affected service and move on. Fix the underlying network configuration, replace the failing drive, right-size memory, or correct time synchronization, and only then remove the temporary noout/nodown flags. A Ceph cluster that returns to HEALTH_OK because the root cause was fixed will stay healthy; one that returns to HEALTH_OK because the flags were removed will flap again the next time load increases.