On this Page:

MDS Notifications

CephFS MDS notifications stream real-time file and directory change events from the metadata server (MDS) to external consumers over UDP or Kafka. Use them to build event-driven pipelines, monitor file-system activity, or integrate CephFS with external systems.

NOTE

Preview feature. MDS notifications are a preview and are off by default. Until the feature is enabled for your cluster, the Notifications tab does not appear in the croit UI. Please contact croit support to have it enabled before you follow this guide.

MDS notifications are also a croit-exclusive feature, available only on croit Enhanced OS images. The notification interface is not part of upstream Ceph, so on other images the feature is unavailable.

How it works

The MDS watches the file system and emits an event whenever a path is created, deleted, modified, or moved. Each notification endpoint you register is a consumer the MDS pushes those events to:

  • UDP — the MDS sends datagrams to an ip:port. Lightweight and low-overhead, but delivery is best-effort (fire-and-forget).
  • Kafka — the MDS publishes JSON events to a topic on a broker, optionally over SSL/TLS with SASL authentication. Durable and ordered.

Events flow directly from the MDS to your consumer — they do not pass through the croit management node.

Requirements

  • A CephFS file system with a running MDS.
  • Nodes running a croit Enhanced OS image.
  • For Kafka over SSL/SASL: the broker's CA certificate available on the MDS nodes.

Configuring in the croit UI

Open File Storage → Notifications and select your CephFS volume. The tab lists the registered endpoints and lets you add or remove them.

Add a UDP endpoint

Click Create, choose UDP, and fill in:

Field Description
Name An identifier for this consumer.
IP Address The address the MDS sends UDP datagrams to.
Port The destination port (default 9876).

Add a Kafka endpoint

Click Create, choose Kafka, and fill in:

Field Description
Name An identifier for this endpoint.
Topic The Kafka topic events are published to.
Broker The Kafka broker address, e.g. kafka1:9092.
Use SSL/TLS Enable to connect to the broker over SSL/TLS.
CA Certificate Path Path to the broker's CA certificate on the MDS node (SSL only).
SASL Mechanism PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. Leave as (none) for no SASL.
Username / Password SASL credentials (SASL only).

IMPORTANT

SASL authentication is only offered once SSL/TLS is enabled — the MDS refuses to send credentials over a cleartext connection. The password cannot be changed after the endpoint is created; to change it, remove the endpoint and add it again.

Remove an endpoint

Select one or more rows and click Delete.

Notification format

Kafka messages are structured JSON payloads describing a file or directory event. The fields vary by event type. For a UDP endpoint the payload is the same JSON, but the stream is prefixed with the message length.

Example — CREATE

{
  "timestamp": "2024-09-29T17:50:03.723730267Z",
  "mds_id": 0,
  "session_id": "877edd36-55e0-47a8-8c3a-fe81457df010",
  "seq_id": 2,
  "mask": 16,
  "path": "/dir1/dir2/dir3"
}

mask: 16 is CEPH_MDS_NOTIFY_CREATE — a file or directory was created at /dir1/dir2/dir3.

Example — move (MOVED_FROM / MOVED_TO)

{
  "timestamp": "2024-09-29T17:51:02.716590304Z",
  "mds_id": 0,
  "session_id": "877edd36-55e0-47a8-8c3a-fe81457df010",
  "seq_id": 6,
  "src_mask": 512,
  "dest_mask": 1024,
  "src_path": "/dir1/dir2/dir3/x.txt",
  "dest_path": "/dir1/dir2/dir4/y.txt"
}

src_mask: 512 (MOVED_FROM) and dest_mask: 1024 (MOVED_TO) describe a file renamed/moved from /dir1/dir2/dir3/x.txt to /dir1/dir2/dir4/y.txt.

Event types (mask values)

The mask, src_mask, and dest_mask fields are bit flags. Decode them with:

Flag Hex Description
CEPH_MDS_NOTIFY_ACCESS 0x00000001 File was accessed
CEPH_MDS_NOTIFY_SET_ATTRIB 0x00000002 Metadata changed (e.g. chmod, chown)
CEPH_MDS_NOTIFY_CLOSE_WRITE 0x00000004 File closed after write
CEPH_MDS_NOTIFY_CLOSE_NOWRITE 0x00000008 File closed without write
CEPH_MDS_NOTIFY_CREATE 0x00000010 File or directory created
CEPH_MDS_NOTIFY_DELETE 0x00000020 File or directory deleted
CEPH_MDS_NOTIFY_DELETE_SELF 0x00000040 Watched file/dir was itself deleted
CEPH_MDS_NOTIFY_MODIFY 0x00000080 File modified
CEPH_MDS_NOTIFY_MOVE_SELF 0x00000100 Watched object moved
CEPH_MDS_NOTIFY_MOVED_FROM 0x00000200 Object moved out of watched directory
CEPH_MDS_NOTIFY_MOVED_TO 0x00000400 Object moved into watched directory
CEPH_MDS_NOTIFY_OPEN 0x00000800 File was opened
CEPH_MDS_NOTIFY_CLOSE 0x00001000 File was closed
CEPH_MDS_NOTIFY_MOVE 0x00002000 Generic move operation
CEPH_MDS_NOTIFY_ONESHOT 0x00004000 Notify only once per watch
CEPH_MDS_NOTIFY_IGNORED 0x00008000 Watch was removed
CEPH_MDS_NOTIFY_SET_LAYOUT 0x00010000 Layout (striping, etc.) changed
CEPH_MDS_NOTIFY_SET_XATTRIB 0x00020000 Extended attribute set
CEPH_MDS_NOTIFY_REM_XATTRIB 0x00040000 Extended attribute removed
CEPH_MDS_NOTIFY_ONLYDIR 0x00080000 Marked when the object is a directory

Advanced: admin-socket commands

The croit UI drives the MDS admin socket for you. If you prefer to script endpoint management, the same operations are available via ceph tell. croit addresses the active MDS of a file system as mds.<fs_name>:0.

Add a Kafka topic

ceph tell mds.<fs_name>:<rank> add_topic \
  --topic_name=<topic> \
  --endpoint_name=<endpoint id> \
  --broker=<broker_address> \
  [--use_ssl=true|false] \
  [--username=<sasl_username>] \
  [--password=<sasl_password>] \
  [--ca_location=<ca_path>] \
  [--mechanism=<sasl_mechanism>]
Parameter Description Required
topic_name Kafka topic events are published to. Yes
endpoint_name Identifier for this endpoint. Yes
broker Kafka broker URL (e.g. kafka1:9092). Yes
use_ssl Use SSL/TLS (default false). No
username, password SASL credentials. No
ca_location Path to the CA certificate for SSL. No
mechanism SASL mechanism (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512). No

Remove a Kafka topic

ceph tell mds.<fs_name>:<rank> remove_topic \
  --topic_name=<topic> \
  --endpoint_name=<endpoint id>

Add a UDP endpoint

ceph tell mds.<fs_name>:<rank> add_udp_endpoint \
  --entity=<endpoint id> \
  --ip=<endpoint address> \
  --port=<port number>
Parameter Description Required
entity Identifier for this endpoint. Yes
ip IP address of the UDP consumer. Yes
port Port of the UDP consumer. Yes

Remove a UDP endpoint

ceph tell mds.<fs_name>:<rank> remove_udp_endpoint --entity=<endpoint id>

Verifying

After adding an endpoint, make a change in the file system (for example, create a directory or write a file) and confirm the event arrives at your consumer:

  • Kafka — consume the topic and look for the event JSON referencing the changed path.
  • UDP — listen on the configured ip:port and read the length-prefixed JSON.