- Health report in JSON for the monitoring
- When will the disk be full?
- Server inventory and security checks
- Sessions and errors of the day
- Maintenance window: lock, change the config, reopen
- Morning check line from the system statistics
- Do the logs cover the metric window?
- CMDB record with a hashed host id
- Trace a check in the log and find it back
- Peak or sustained CPU load?
- Publish the counters of an application as a Prometheus /metrics page
- Read the disk space of a node exporter and draw the free space of each disk
- Keep the daily export of a configuration in a Git repository and see what changed
- Warn the team on Slack and Teams when disks are almost full
- Read an nginx access log: server errors, robots and slow requests
- Send the orders of the day to Grafana or Datadog as OpenTelemetry metrics
- Send the depth of the work queues to a StatsD or Datadog agent
Health report in JSON for the monitoring
Every minute the supervision tool pulls one JSON document: CPU of the system and of the JVM, heap usage, RAM really available (cache excluded), swap, and an OK/WARN status above 85 % CPU or 90 % RAM.
#1. Identity and CPU load; json load "rx_health" "{}"; json iobject "rx_health" / "ts" (date systimestamp) STR; json iobject "rx_health" / "mentdb" (concat (name) " v" (version)) STR; json iobject "rx_health" / "cpu_sys" (metric current cpu system) NUM; json iobject "rx_health" / "cpu_jvm" (metric current cpu jvm) NUM; #2. JVM heap: used/committed/max in MB; -> "[jvm]" (metric current mem jvm); json iobject "rx_health" / "heap_used_pct" (math round (* 100 (/ (atom get [jvm] 1 "/") (atom get [jvm] 3 "/"))) 1) NUM; #3. RAM really available, free RAM and swap; -> "[total]" (metric total mem); json iobject "rx_health" / "ram_used_pct" (math round (* 100 (/ (- [total] (metric current available mem)) [total])) 1) NUM; json iobject "rx_health" / "ram_free_mb" (metric current free mem) NUM; json iobject "rx_health" / "ram_used_with_cache_mb" (metric current used mem) NUM; json iobject "rx_health" / "swap_used_mb" (metric current used swap mem) NUM; json iobject "rx_health" / "swap_free_mb" (metric current free swap mem) NUM; json iobject "rx_health" / "swap_total_mb" (metric total swap mem) NUM; #4. Verdict: WARN above 85 percent CPU or 90 percent RAM; json iobject "rx_health" / "status" (if (or (> (json select "rx_health" /cpu_sys) 85) (> (json select "rx_health" /ram_used_pct) 90)) {"WARN"} {"OK"}) STR; json doc "rx_health"
{
"ram_used_with_cache_mb": 18304.0,
"cpu_sys": 11.0,
"ram_free_mb": 127.0,
"swap_total_mb": 2048.0,
"ram_used_pct": 73.2,
"heap_used_pct": 15.2,
"swap_used_mb": 1149.0,
"ts": "2026-11-01 16:27:23",
"mentdb": "mona payet v4.2.3",
"cpu_jvm": 0.5,
"swap_free_mb": 898.0,
"status": "OK"
}- Store each report with
json iarrayin a daily array and plot the CPU withplot line - Send it to the supervision every minute: a
job addon a script that returnsjson doc
When will the disk be full?
The data disk grows by about 1.8 GB per day. The script reads the root file system from the metrics, keeps a 10 % safety margin and gives the number of days and the date when the margin is reached, with an alert below 30 days.
#1. Root file system from the metrics, sizes in MB; json load "rx_disk" (metric file system roots); -> "[mount]" (json select "rx_disk" "/[0]/absolutePath"); -> "[free]" (json select "rx_disk" "/[0]/freeSpace"); -> "[total]" (json select "rx_disk" "/[0]/totalSpace"); -> "[used_pct]" (math round (* 100 (/ (- [total] [free]) [total])) 1); #2. Growth of 1.8 GB per day, keep a 10 percent safety margin; -> "[usable]" (- [free] (* 0.10 [total])); -> "[days]" (math floor (/ [usable] 1800)); #3. Date when the margin is reached; -> "[limit]" (date add (date sysdate) "DAY" [days]); json unload "rx_disk"; concat "{\"mount\": \"" [mount] "\", \"used_pct\": " [used_pct] ", \"free_gb\": " (math round (/ [free] 1024) 1) ", \"days_left\": " [days] ", \"margin_reached_on\": \"" [limit] "\", \"alert\": " (if (< [days] 30) {1} {0}) "}"
{"mount": "/", "used_pct": 68.5, "free_gb": 145.2, "days_left": 56, "margin_reached_on": "2026-12-27", "alert": 0}- Measure the real growth: save
freeSpaceevery night and fit it withstat linear_regression - Several mounts: loop on
json countof the array
Server inventory and security checks
Before an audit, the administrator needs the OS, CPUs, Java runtime, time zone, MentDB build and mode, and the days left on the TLS certificate. Two checks are added without printing the account: MentDB must not run as root and must be started from its folder.
#1. Operating system and Java runtime; json load "rx_inv" "{}"; json iobject "rx_inv" / "os" (concat (os name) " " (os version) " " (os arch)) STR; json iobject "rx_inv" / "os_type" (os type) STR; json iobject "rx_inv" / "kernel" (concat (metric system name) " " (metric system version) " " (metric system architecture)) STR; json iobject "rx_inv" / "cpus" (metric system nb processor) INT; json iobject "rx_inv" / "java" (concat (metric java version) " " (metric java vendor)) STR; json iobject "rx_inv" / "tz_lang" (concat (os user_timezone) " " (os user_lang)) STR; #2. Security checks without printing the account: not root, started from the MentDB folder; json iobject "rx_inv" / "runs_as_root" (if (or (string equals_ignore_case (os user_name) "root") (string contains (os user_home) "/root")) {1} {0}) INT; json iobject "rx_inv" / "workdir_is_mentdb" (string contains (os user_dir) "MentDB") INT; #3. MentDB build, mode and size; json iobject "rx_inv" / "mentdb" (concat (version) " - " (ai firstname) " " (ai lastname)) STR; json iobject "rx_inv" / "mode" (if (parameter exist "SERVER_MODE") {parameter get value "SERVER_MODE"} {"?"}) STR; json iobject "rx_inv" / "mode_doc" (parameter get description "SERVER_MODE") STR; json iobject "rx_inv" / "mode_private" (parameter is_private "SERVER_MODE") INT; json iobject "rx_inv" / "mode_restore" (parameter generate update "SERVER_MODE") STR; json iobject "rx_inv" / "mql_functions" (function count) INT; json iobject "rx_inv" / "java_lines" (src count) INT; #4. Days before the TLS certificate expires; json load "rx_cert" (config cert); json iobject "rx_inv" / "cert_days_left" (json select "rx_cert" "/[1]/days_left") INT; json unload "rx_cert"; json doc "rx_inv"
{
"workdir_is_mentdb": 1,
"os": "Mac OS X 14.4 aarch64",
"cpus": 11,
"mode_restore": "parameter update \"SERVER_MODE\" \"DEV\";",
"kernel": "Mac OS X 14.4 aarch64",
"tz_lang": "Indian/Reunion fr",
"mode_doc": "Mode of the server (DEV, TEST, PROD...), shown in the administration.",
"mode_private": 0,
"cert_days_left": 1819,
"mql_functions": 1606,
"mode": "DEV",
"java": "21.0.13 Eclipse Adoptium",
"os_type": "Mac OS X",
"runs_as_root": 0,
"mentdb": "4.2.3 - mona payet",
"java_lines": 798979
}- Read the whole configuration with
config show: secret values are masked - Alert when
cert_days_leftis below 30 and renew the certificate
Sessions and errors of the day
Morning check: how many sessions are open and against which capacity, how many come from the MQL editor, the oldest one, the web socket sessions, the exceeded sessions, and the scripts that ended in error in the log today.
#1. Open sessions: how many come from the MQL editor, the oldest one; json load "rx_sess" (sessions); -> "[n]" (json count "rx_sess" /); -> "[mql]" 0; -> "[oldest]" 0; for (-> "[i]" 0) (< [i] [n]) (++ "[i]") { if (equal (json select "rx_sess" (concat "/[" [i] "]/origin")) "MQL") {++ "[mql]"}; -> "[oldest]" (math max [oldest] (json select "rx_sess" (concat "/[" [i] "]/used"))); }; #2. Session capacity from the metrics, without the editor marker; json load "rx_ms" (string substring (metric sessions) 39); #3. Scripts in error in the log today; json load "rx_ko" (string substring (log search "ko" null null null null (concat (date sysdate) " 00:00:00") (concat (date sysdate) " 23:59:59") DESC 1 50) 39); json load "rx_ws" (@sessions); concat "{\"me\": \"" (who) "\", \"my_commands\": " (cmdid) ", \"java_pid\": " (pid) ", \"open\": " (count sessions) ", \"capacity\": \"" (json select "rx_ms" "/data/[0]/open_session") "\", \"from_editor\": " [mql] ", \"websocket\": \"" (@count sessions) "/" (json count "rx_ws" /) "\"" ", \"oldest_s\": " [oldest] ", \"exceeded\": " (exceeded sessions) ", \"ws_exceeded\": " (@exceeded sessions) ", \"log_id\": " (log current id) ", \"errors_today\": " (json count "rx_ko" /data) ", \"log_retention_days\": " (log retention day) ", \"archive\": \"" (log archive path) " every " (log archive size) "\"}"
{"me": "admin", "my_commands": 5, "java_pid": 80950, "open": 2, "capacity": "2/100", "from_editor": 1, "websocket": "0/0", "oldest_s": 3760, "exceeded": 0, "ws_exceeded": 0, "log_id": 0, "errors_today": 0, "log_retention_days": 50, "archive": "archives/logs every 30"}- Open the detail of one process with
log show_timeorrestricted show_pid - The editor output of
metric sessionsandlog searchstarts with a 39-character marker:string substringremoves it beforejson load
Maintenance window: lock, change the config, reopen
A change request asks to keep 90 days of logs. The administrator closes the web services, web apps, jobs and mails, changes and reloads the config, purges the logs, closes the editor sessions left open by other users, then reopens and traces the window.
#1. Close web services, web apps, jobs and mails, keep the MQL editor open; maintenance set 0 1 1 1 0; maintenance set_lock "MAIL" 1; log write "Maintenance window opened" "OK" "change" "CHG-2026-118"; #2. Keep 90 days of logs instead of 50, apply it and purge; config set "LOG" "RETENTION_DAYS" "90"; config reload; log remove; #3. Close the editor sessions left open by other users; json load "rx_sess" (sessions); for (-> "[i]" 0) (< [i] (json count "rx_sess" /)) (++ "[i]") { -> "[id]" (json select "rx_sess" (concat "/[" [i] "]/id")); if (and (equal (json select "rx_sess" (concat "/[" [i] "]/origin")) "MQL") (not (equal [id] (sid)))) {kill [id]}; }; reset exceeded sessions; #4. Restart the scheduler, release the stack locks, reopen and trace; job scheduler restart; stack reset; maintenance set 0 0 0 0 0; maintenance set_lock "MAIL" 0; log trace "Maintenance window closed"; json unload "rx_sess"; maintenance get
- Check the doors at any time with
maintenance get config showsays for each key ifconfig reloadis enough or if a restart is needed
Morning check line from the system statistics
`metric system` returns a full text report. The script keeps only the runtime figures (Java, CPUs, JVM heap, RAM, root file system), leaves the identity fields out and builds one line for the morning check with a disk verdict.
#1. Full system statistics in one call; -> "[sys]" (metric system); #2. Keep only the runtime lines, no identity fields; -> "[java]" (string regexp_extract [sys] "Java: v_([^\n]+)" 1); -> "[os]" (string regexp_extract [sys] "nb_proc: ([0-9]+)" 1); -> "[heap]" (string regexp_extract [sys] "Used jvm mem: ([0-9]+)/([0-9]+)/([0-9]+)M" 0); -> "[ram]" (string regexp_extract [sys] "Used mem: ([0-9]+)/([0-9]+)M" 0); -> "[root]" (string regexp_extract [sys] "\[/\] ([0-9]+)%" 1); #3. One line for the morning check; concat "MentDB " (version) " on Java " [java] " | " [os] " CPUs | " [heap] " | " [ram] " | root fs " [root] "% used" (if (> [root] 85) {" -> CLEAN THE DISK"} {" -> ok"})
MentDB 4.2.3 on Java 21.0.13, Eclipse Adoptium | 11 CPUs | Used jvm mem: 167/512/2048M | Used mem: 13109/18432M | root fs 68% used -> ok
- Send the line to the team chat from a job every morning.
- Store the figures in a table to follow the heap and the disk over months.
- Get each value directly with
metric current mem jvmormetric file system rootswhen you need numbers.
Do the logs cover the metric window?
When an incident is investigated, the metrics (31 days) and the logs must cover the same period. The script reads the 31 days of the metrics, counts the business days and checks the log retention against the window.
#1. The 31 days kept by the metrics; -> "[days]" (metric date); -> "[first]" (string regexp_extract [days] "[0-9]{4}-[0-9]{2}-[0-9]{2}" 0); -> "[last]" (string regexp_extract [days] "([0-9]{4}-[0-9]{2}-[0-9]{2})\]" 1); #2. Working days and weekend days in the window; -> "[business]" (date nb_business_days [first] [last]); -> "[span]" (+ (date nb_day [first] [last]) 1); #3. Do the logs cover the same window?; -> "[ret]" (log retention day); concat "metric window " [first] " -> " [last] " (" [span] " days, " [business] " business days, starts on a " (date dayname [first] "en") ") | log retention " [ret] " days: " (if (>= [ret] [span]) {"the logs cover the whole metric window"} {"WARNING the logs are purged before the metrics"})
metric window 2026-10-02 -> 2026-11-01 (31 days, 21 business days, starts on a Friday) | log retention 50 days: the logs cover the whole metric window
- Raise the retention in the LOG section of the configuration when the logs are shorter than the metrics.
- Archive the logs before the purge (
log archive path). - Cross the busiest day of
metric sessionswithlog searchon that day.
CMDB record with a hashed host id
Each server must be registered in the CMDB without exposing its hostname in reports. The script builds a stable host id (SHA-256 of hostname, arch and CPU count, 16 characters) and an inventory record in JSON.
#1. Stable id of the host for the CMDB, the hostname itself is not printed; -> "[host_id]" (string left (string sha_256 (concat (os hostname) "|" (os arch) "|" (metric system nb processor))) 16); #2. Inventory record; json load "rx_p4h_ci" "{}"; json iobject "rx_p4h_ci" / "host_id" [host_id] STR; json iobject "rx_p4h_ci" / "hostname_length" (string length (os hostname)) INT; json iobject "rx_p4h_ci" / "platform" (concat (os type) " " (os arch)) STR; json iobject "rx_p4h_ci" / "ram_gb" (math round (/ (metric total mem) 1024) 1) NUM; json iobject "rx_p4h_ci" / "swap_gb" (math round (/ (metric total swap mem) 1024) 1) NUM; json iobject "rx_p4h_ci" / "java" (metric java version) STR; json iobject "rx_p4h_ci" / "mentdb" (version) STR; json iobject "rx_p4h_ci" / "seen_at" (date to_iso (date systimestamp)) STR; json doc "rx_p4h_ci"
{
"java": "21.0.13",
"hostname_length": 26,
"host_id": "1581bc9d9e69bbd7",
"platform": "Mac OS X aarch64",
"ram_gb": 18.0,
"swap_gb": 2.0,
"mentdb": "4.2.3",
"seen_at": "2026-11-01T18:30:36+04:00"
}- Post the record to the CMDB API with
rest https. - Compare the host id with yesterday's one to detect a cloned or moved server.
- Add the certificate expiry from
config cert.
Trace a check in the log and find it back
An audit needs to prove that the memory check ran. A temporary check script with the activity log on writes its result with a client key and a unique reference; the script then finds the entry back with `log search` and deletes the check script.
#1. A check script with the activity log on: log write only writes from such a script; script create exe "tuto.p4h.memcheck" true 1 (param (var "[ref]" {true} "Reference of the run" is_null:false is_empty:false "run-1") ;) "Memory check traced in the log" { -> "[free_pct]" (math round (* 100 (/ (metric current available mem) (metric total mem))) 1); -> "[status]" (if (< [free_pct] 10) {"KO"} {"OK"}); log write (concat "Memory check: " [free_pct] "% available") [status] "tuto_p4h_check" [ref]; [status]; } "OK or KO"; #2. Run it with a unique reference; -> "[ref]" (concat "run-" (date current_ms)); -> "[status]" (execute "tuto.p4h.memcheck.exe" "[ref]" [ref]); #3. Find the entry back by its client key and value, like an auditor; json load "rx_p4h_log" (string substring (log search (string lower [status]) "tuto.p4h.memcheck.exe" "tuto_p4h_check" [ref] null (concat (date sysdate) " 00:00:00") (concat (date sysdate) " 23:59:59") DESC 1 10) 39); script delete "tuto.p4h.memcheck.exe"; concat "status " [status] " | entries found: " (json count "rx_p4h_log" /data) " | message: " (json select "rx_p4h_log" "/data/[0]/msg") " | script: " (json select "rx_p4h_log" "/data/[0]/script")
status OK | entries found: 1 | message: Memory check: 28.9% available | script: tuto.p4h.memcheck.exe
log writewrites only when the script was created with activateLog true: in the editor it returns 1 but writes nothing.- Schedule the check with
job addand search the KO entries every morning. - Use
log show_timewith the pid to measure the duration of a run.
Peak or sustained CPU load?
One CPU reading says little. The script samples the system CPU 12 times every 500 ms, then compares the mean, the 95th percentile and the max to tell a sustained load from short peaks, with the share of the MentDB JVM.
#1. Sample the system CPU 12 times, every 500 ms; json load "rx_p4h_cpu" "[]"; for (-> "[i]" 0) (< [i] 12) (++ "[i]") { json iarray "rx_p4h_cpu" / (metric current cpu system) NUM; wait 500; }; -> "[s]" (json doc "rx_p4h_cpu"); #2. Peak or sustained load? mean, 95th percentile, max; -> "[mean]" (math round (stat mean [s]) 1); -> "[p95]" (math round (stat percentile [s] 95) 1); -> "[max]" (stat max [s]); -> "[jvm]" (metric current cpu jvm); concat "cpu system over 6 s: mean " [mean] "%, p95 " [p95] "%, max " [max] "% | MentDB JVM share now " [jvm] "% | " (case (> [mean] 80) {"SUSTAINED load: add CPUs or move jobs"} (> [p95] 80) {"short peaks only"} {"normal"})
cpu system over 6 s: mean 8.1%, p95 11.9%, max 12% | MentDB JVM share now 0.1% | normal
- Run it from a job during the batch window to size the server.
- Plot the samples with
plot signal. - Add the JVM heap from
metric current mem jvmto the same loop.
Publish the counters of an application as a Prometheus /metrics page
Prometheus, Grafana Agent and most monitoring tools read the text format of a /metrics page. The counters are kept as JSON (from SQL or a script), prometheus format writes the page, and a web service returns it with the content type text/plain; version=0.0.4. The page is read back here to check it, as Prometheus would.
-> "[metrics]" "[{\"name\": \"orders_received_total\", \"type\": \"counter\", \"help\": \"Orders received since the start.\", \"samples\": [{\"labels\": {\"channel\": \"web\"}, \"value\": 1027}, {\"labels\": {\"channel\": \"shop\"}, \"value\": 412}]}, {\"name\": \"orders_failed_total\", \"type\": \"counter\", \"help\": \"Orders refused by the payment.\", \"samples\": [{\"labels\": {\"channel\": \"web\"}, \"value\": 9}, {\"labels\": {\"channel\": \"shop\"}, \"value\": 2}]}, {\"name\": \"queue_waiting\", \"type\": \"gauge\", \"help\": \"Messages waiting in the queue.\", \"value\": 14}, {\"name\": \"order_seconds\", \"type\": \"histogram\", \"help\": \"Time to process an order.\", \"samples\": [{\"buckets\": {\"0.1\": 820, \"0.5\": 1300, \"2\": 1431, \"+Inf\": 1439}, \"sum\": 402.7}]}]"; -> "[page]" (prometheus format [metrics]); json load "rec_pm" (prometheus parse [page] "{\"flat\": true}"); -> "[lines]" (json count "rec_pm" "/"); -> "[web]" (json select "rec_pm" "/[0]/value"); -> "[failed]" (json select "rec_pm" "/[2]/value"); json unload "rec_pm"; -> "[om]" (prometheus format [metrics] "{\"format\": \"openmetrics\"}"); -> "[res]" (concat [lines] " samples | web failure rate " (math round (* 100 (/ [failed] [web])) 2) " % | OpenMetrics ends with " (string right [om] 6) (string char 10) [page]); [res]
11 samples | web failure rate 0.88 % | OpenMetrics ends with # EOF
# HELP orders_received_total Orders received since the start.
# TYPE orders_received_total counter
orders_received_total{channel="web"} 1027
orders_received_total{channel="shop"} 412
# HELP orders_failed_total Orders refused by the payment.
# TYPE orders_failed_total counter
orders_failed_total{channel="web"} 9
orders_failed_total{channel="shop"} 2
# HELP queue_waiting Messages waiting in the queue.
# TYPE queue_waiting gauge
queue_waiting 14
# HELP order_seconds Time to process an order.
# TYPE order_seconds histogram
order_seconds_bucket{le="0.1"} 820
order_seconds_bucket{le="0.5"} 1300
order_seconds_bucket{le="2.0"} 1431
order_seconds_bucket{le="+Inf"} 1439
order_seconds_count 1439
order_seconds_sum 402.7- The histogram gives the 0.1, 0.5 and 2 second buckets: Prometheus computes the quantiles from them (histogram_quantile).
- Use the openmetrics format for a scraper that asks for application/openmetrics-text; it ends with # EOF.
Read the disk space of a node exporter and draw the free space of each disk
A node exporter (or any Prometheus exporter) gives the state of a machine as text. Here the text was saved from a scrape, so it runs without the exporter; on site, rest request on http://host:9100/metrics gives the same text. prometheus parse turns it into JSON, the free share of each disk is computed and drawn as horizontal bars.
-> "[scrape]" "# HELP node_filesystem_avail_bytes Free space.\n# TYPE node_filesystem_avail_bytes gauge\nnode_filesystem_avail_bytes{mountpoint=\"/\"} 2100000000 1762585200000\nnode_filesystem_avail_bytes{mountpoint=\"/data\"} 45000000000 1762585200000\n# HELP node_filesystem_size_bytes Size.\n# TYPE node_filesystem_size_bytes gauge\nnode_filesystem_size_bytes{mountpoint=\"/\"} 20000000000 1762585200000\nnode_filesystem_size_bytes{mountpoint=\"/data\"} 50000000000 1762585200000"; json load "rec_ne" (prometheus parse [scrape]); -> "[root]" (math round (* 100 (/ (json select "rec_ne" "/[0]/samples/[0]/value") (json select "rec_ne" "/[1]/samples/[0]/value"))) 1); -> "[data]" (math round (* 100 (/ (json select "rec_ne" "/[0]/samples/[1]/value") (json select "rec_ne" "/[1]/samples/[1]/value"))) 1); json unload "rec_ne"; -> "[chart]" (concat "{\"type\": \"bar\", \"horizontal\": true, \"title\": \"Free disk space\", \"categories\": [\"/\", \"/data\"], \"values\": [" [root] ", " [data] "], \"number_format\": {\"suffix\": \" %\"}, \"data_labels\": true, \"y_min\": 0, \"y_max\": 100}"); -> "[svg]" (chart svg [chart] "{\"width\": 400, \"height\": 160}"); -> "[res]" (concat "free on /: " [root] " % | free on /data: " [data] " % | SVG of " (string length [svg]) " characters, starts with " (string left [svg] 40)); [res]
free on /: 10.5 % | free on /data: 90 % | SVG of 1472 characters, starts with <svg xmlns="http://www.w3.org/2000/svg"
- Only 10.5 % is free on /: a scheduled script can send this chart by mail when a disk goes under a threshold.
- prometheus parse with flat gives one row per line, ready for sql insert or a table of all the disks.
Keep the daily export of a configuration in a Git repository and see what changed
Each night the parameters of an application are exported to a CSV file. Committed in a Git repository, every change gets a date, an author and a message, and git diff tells exactly which line changed between two nights; a tag marks the version that went to production. JGit is inside MentDB: no git program is needed, no hook of the repository is ever run, and the same commit (same content, author and date) gives the same id as the git command. With a remote (git clone, git push with token), the history can be kept on a Git server.
if (equal (file exist "tmp/rec_ops_git") "1") {file delete "tmp/rec_ops_git";}; if (equal (file exist "tmp/rec_ops_git") "0") {file mkdir "tmp/rec_ops_git";}; -> "[repo]" "tmp/rec_ops_git/config"; git init [repo]; csv write "tmp/rec_ops_git/config/settings.csv" "[{\"key\": \"smtp.host\", \"value\": \"mail.example.com\"}, {\"key\": \"backup.hour\", \"value\": \"02:00\"}]"; git add [repo] "settings.csv"; git commit [repo] "Export of 2026-11-09" "{\"author_name\": \"Ops robot\", \"author_email\": \"ops@example.com\", \"date\": \"2026-11-09T02:00:00Z\"}"; csv write "tmp/rec_ops_git/config/settings.csv" "[{\"key\": \"smtp.host\", \"value\": \"mail2.example.com\"}, {\"key\": \"backup.hour\", \"value\": \"02:00\"}, {\"key\": \"backup.keep_days\", \"value\": \"30\"}]"; json load "rg_st" (git status [repo]); git add [repo] "."; git commit [repo] "Export of 2026-11-10" "{\"author_name\": \"Ops robot\", \"author_email\": \"ops@example.com\", \"date\": \"2026-11-10T02:00:00Z\"}"; -> "[diff]" (git diff [repo] "{\"from\": \"HEAD~1\", \"to\": \"HEAD\", \"context\": 0}"); git tag [repo] "prod-2026-11-10"; json load "rg_log" (git log [repo] "{\"path\": \"settings.csv\"}"); -> "[res]" (concat "modified before the commit: " (json select "rg_st" "/modified/[0]") " | commits: " (json select "rg_log" "/[1]/message") ", then " (json select "rg_log" "/[0]/message") " (" (json select "rg_log" "/[0]/short_id") ") | diff: " (string char 10) [diff]); json unload "rg_st"; json unload "rg_log"; if (equal (file exist "tmp/rec_ops_git") "1") {file delete "tmp/rec_ops_git";}; [res]
modified before the commit: settings.csv | commits: Export of 2026-11-09, then Export of 2026-11-10 (f06d0a4) | diff: diff --git a/settings.csv b/settings.csv index bf5ba97..b18810d 100644 --- a/settings.csv +++ b/settings.csv @@ -2 +2 @@ -smtp.host,mail.example.com +smtp.host,mail2.example.com @@ -3,0 +4 @@ +backup.keep_days,30
- git diff without parameters shows what is not added yet; {"cached": true} what is added and not committed.
- A token is never written in a URL: git push "repo" "{\"token\": \"git_token\"}" reads it from a secret (secret set).
Warn the team on Slack and Teams when disks are almost full
A scheduled script reads the disk usage of the servers (here JSON rows; in real life sql to json or a metrics API) and keeps the servers over 90 %. One message lists them as fields, with a button to the dashboard. The same parameters build a Slack Block Kit message and a Teams Adaptive Card. With dry_run the functions send nothing and need no secret: they give the exact JSON that would be sent and its size, so the message can be checked here. In production, replace dry_run by webhook_url, the name of a secret (secret set) of the server that holds the webhook URL.
-> "[disks]" "[{\"host\": \"db-01\", \"used_pct\": 93, \"free_gb\": 14}, {\"host\": \"web-01\", \"used_pct\": 61, \"free_gb\": 80}, {\"host\": \"files-02\", \"used_pct\": 97, \"free_gb\": 3}]"; json load "rx_d" [disks]; -> "[f]" ""; -> "[k]" 0; for (-> "[i]" 0) (< [i] (json count "rx_d" "/")) (++ "[i]") { -> "[u]" (json select "rx_d" (concat "/[" [i] "]/used_pct")); if (> [u] 90) { if (> [k] 0) {-> "[f]" (concat [f] ", ");}; -> "[f]" (concat [f] "\"" (json select "rx_d" (concat "/[" [i] "]/host")) "\": \"" [u] " % (" (json select "rx_d" (concat "/[" [i] "]/free_gb")) " GB free)\""); -> "[k]" (+ [k] 1); }; }; json unload "rx_d"; -> "[params]" (concat "{\"title\": \"Disks almost full\", \"fields\": {" [f] "}, \"link_url\": \"https://grafana.example.org/d/disks\", \"link_text\": \"Open the dashboard\", \"dry_run\": true}"); -> "[text]" (concat [k] " servers are over 90 %"); json load "rx_s" (notify slack [text] [params]); json load "rx_t" (notify teams [text] [params]); -> "[res]" (concat "alerts: " [k] " | slack blocks: " (json count "rx_s" "/payload/blocks") ", first " (json select "rx_s" "/payload/blocks/[0]/type") ", fields " (json count "rx_s" "/payload/blocks/[2]/fields") ", " (json select "rx_s" "/bytes") " bytes | teams facts: " (json count "rx_t" "/payload/attachments/[0]/content/body/[2]/facts") ", " (json select "rx_t" "/bytes") " bytes of 28672"); json unload "rx_s"; json unload "rx_t"; [res]
alerts: 2 | slack blocks: 4, first header, fields 2, 575 bytes | teams facts: 2, 674 bytes of 28672
- Put the webhook URL in a secret (secret set) (for example SLACK_WEBHOOK_URL) and give {"webhook_url": "slack_webhook_url"}: the URL holds a secret, so it is refused in clear and never appears in a result or an error.
- Teams: create the webhook with the Workflows app of Teams. The old Office 365 connector URLs (webhook.office.com) stopped working in May 2026 and are refused with an explanation.
- A rate limit (HTTP 429) is an error that says how long to wait: in a loop over many alerts, catch it and stop, rather than sending again at once.
Read an nginx access log: server errors, robots and slow requests
The access log of a web server says who called what, the answer and the time it took. log parse detects the nginx format, gives typed fields (status and request_time are numbers, the date is ISO 8601 with its offset) and reports the lines it cannot read instead of failing. With useragent true, each row also says whether the visitor is a robot and which browser and device it uses. Here the log is a text; in production, log parse_file reads the log file itself as a stream, also a rotated .gz file.
-> "[nl]" (string char 10); -> "[log]" (concat "203.0.113.9 - - [03/Nov/2026:10:00:00 +0100] \"GET / HTTP/2.0\" 200 5120 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\" \"-\" rt=0.012" [nl] "66.249.66.1 - - [03/Nov/2026:10:00:02 +0100] \"GET /robots.txt HTTP/1.1\" 200 68 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" \"-\" rt=0.001" [nl] "198.51.100.4 - - [03/Nov/2026:10:00:05 +0100] \"POST /api/orders HTTP/2.0\" 502 157 \"https://shop.example.com/cart\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/15E148 Safari/604.1\" \"-\" rt=2.481" [nl] "203.0.113.9 - - [03/Nov/2026:10:00:09 +0100] \"GET /api/orders/42 HTTP/2.0\" 404 153 \"-\" \"curl/8.4.0\" \"-\" rt=0.004" [nl] "\x16\x03\x01 garbage from a scanner" [nl] "198.51.100.4 - - [03/Nov/2026:10:01:00 +0100] \"POST /api/orders HTTP/2.0\" 201 310 \"https://shop.example.com/cart\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/15E148 Safari/604.1\" \"-\" rt=1.210"); json load "rw_l" (log parse [log] "{\"useragent\": true, \"fields\": [\"timestamp\", \"client\", \"method\", \"path\", \"status\", \"request_time\", \"ua\"]}"); -> "[err]" 0; -> "[bots]" 0; -> "[slow]" ""; for (-> "[i]" 0) (< [i] (json count "rw_l" "/rows")) (++ "[i]") { if (> (json select "rw_l" (concat "/rows/[" [i] "]/status")) 499) {-> "[err]" (+ [err] 1);}; if (equal (json select "rw_l" (concat "/rows/[" [i] "]/ua/is_bot")) "true") {-> "[bots]" (+ [bots] 1);}; if (> (json select "rw_l" (concat "/rows/[" [i] "]/request_time")) 1) {-> "[slow]" (concat [slow] " " (json select "rw_l" (concat "/rows/[" [i] "]/path")) " (" (json select "rw_l" (concat "/rows/[" [i] "]/request_time")) " s, " (json select "rw_l" (concat "/rows/[" [i] "]/ua/os")) ")");}; }; -> "[res]" (concat "format " (json select "rw_l" "/format") " | rows " (json select "rw_l" "/parsed") ", unreadable " (json select "rw_l" "/unparsed") " (line " (json select "rw_l" "/unparsed_lines/[0]/line") ") | 5xx: " [err] " | robots: " [bots] " | slow:" [slow] " | first at " (json select "rw_l" "/rows/[0]/timestamp")); json unload "rw_l"; [res]
format nginx | rows 5, unreadable 1 (line 5) | 5xx: 1 | robots: 2 | slow: /api/orders (2.481 s, iOS) /api/orders (1.21 s, iOS) | first at 2026-11-03T10:00:00+01:00
- robots and HTTP libraries (curl, python-requests) are both bots: filter on ua/bot_name to keep only the search engines.
- For an Apache log or an IIS log the same script works: the format is detected. A format of your own is read with a grok pattern, see the next recipe.
- rdap ip on the address of an unknown client gives the owner and the country of its network (it asks the registries over the Internet).
Send the orders of the day to Grafana or Datadog as OpenTelemetry metrics
A scheduled script reads the orders (here JSON rows; in real life sql to json) and publishes two business metrics next to the technical ones: a counter of orders and a histogram of the amounts, built from the raw amounts (the buckets, the sum, the minimum and the maximum are computed). otel metrics turns them into an OTLP request in JSON, the format that every OpenTelemetry collector, Grafana Alloy, Datadog and the other backends read. With dry_run it sends nothing and gives the exact request, so the result can be checked here; in production give the endpoint of the collector, usually http://127.0.0.1:4318.
-> "[orders]" "[{\"shop\": \"Paris\", \"amount\": 120.5}, {\"shop\": \"Lyon\", \"amount\": 42}, {\"shop\": \"Paris\", \"amount\": 310}, {\"shop\": \"Lyon\", \"amount\": 75.25}, {\"shop\": \"Paris\", \"amount\": 18}]"; json load "rx_o" [orders]; -> "[n]" (json count "rx_o" "/"); -> "[vals]" ""; for (-> "[i]" 0) (< [i] [n]) (++ "[i]") { if (> [i] 0) {-> "[vals]" (concat [vals] ", ");}; -> "[vals]" (concat [vals] (json select "rx_o" (concat "/[" [i] "]/amount"))); }; json unload "rx_o"; -> "[metrics]" (concat "[{\"name\": \"shop.orders\", \"type\": \"counter\", \"unit\": \"{order}\", \"value\": " [n] ", \"start_time\": \"2026-11-11T00:00:00Z\", \"time\": \"2026-11-11T18:00:00Z\"}, {\"name\": \"shop.order.amount\", \"type\": \"histogram\", \"unit\": \"EUR\", \"bounds\": [50, 100, 250], \"values\": [" [vals] "], \"time\": \"2026-11-11T18:00:00Z\"}]"); json load "rx_m" (otel metrics "" [metrics] "{\"dry_run\": true, \"resource\": {\"service.name\": \"shop-backend\"}}"); -> "[h]" "/payload/resourceMetrics/[0]/scopeMetrics/[0]/metrics/[1]/histogram/dataPoints/[0]"; -> "[res]" (concat "data points: " (json select "rx_m" "/items") " | orders: " (json select "rx_m" "/payload/resourceMetrics/[0]/scopeMetrics/[0]/metrics/[0]/sum/dataPoints/[0]/asInt") " | amounts up to 50, 100, 250 and above: " (json select "rx_m" (concat [h] "/bucketCounts/[0]")) " " (json select "rx_m" (concat [h] "/bucketCounts/[1]")) " " (json select "rx_m" (concat [h] "/bucketCounts/[2]")) " " (json select "rx_m" (concat [h] "/bucketCounts/[3]")) " | sum " (json select "rx_m" (concat [h] "/sum")) ", max " (json select "rx_m" (concat [h] "/max")) " EUR"); json unload "rx_m"; [res]
data points: 2 | orders: 5 | amounts up to 50, 100, 250 and above: 2 1 1 1 | sum 565.75, max 310 EUR
- Production: replace "" by the endpoint (http://127.0.0.1:4318 for a local collector) and remove dry_run. For a SaaS backend, put its headers in a secret (secret set) (for example GRAFANA_OTLP_HEADERS=Authorization=Basic%20...) and give {"secret_headers": "grafana_otlp_headers"}: the key never appears in a script or a result.
- A counter is cumulative: send the total since start_time (here midnight) at each run, and the backend computes the rate. For the change since the last run, use "temporality": "delta".
- StatsD agents (Datadog, Telegraf) take the same measures with statsd send.
Send the depth of the work queues to a StatsD or Datadog agent
Each minute a script reads the depth of the work queues and the age of their oldest message, and sends them as gauges tagged with the name of the queue to the local agent (the Datadog agent, Telegraf or the Prometheus statsd_exporter listen on UDP port 8125). statsd send packs the lines in datagrams that never exceed max_packet bytes. With dry_run nothing is sent and the datagrams are given, so the lines can be checked here.
-> "[queues]" "[{\"queue\": \"orders\", \"depth\": 17, \"oldest_s\": 42.5}, {\"queue\": \"invoices\", \"depth\": 0, \"oldest_s\": 0}, {\"queue\": \"emails\", \"depth\": 250, \"oldest_s\": 610}]"; json load "rx_q" [queues]; -> "[m]" ""; for (-> "[i]" 0) (< [i] (json count "rx_q" "/")) (++ "[i]") { -> "[name]" (json select "rx_q" (concat "/[" [i] "]/queue")); if (> [i] 0) {-> "[m]" (concat [m] ", ");}; -> "[m]" (concat [m] "{\"name\": \"queue.depth\", \"type\": \"gauge\", \"value\": " (json select "rx_q" (concat "/[" [i] "]/depth")) ", \"tags\": {\"queue\": \"" [name] "\"}}, {\"name\": \"queue.oldest_seconds\", \"type\": \"gauge\", \"value\": " (json select "rx_q" (concat "/[" [i] "]/oldest_s")) ", \"tags\": {\"queue\": \"" [name] "\"}}"); }; json unload "rx_q"; json load "rx_s" (statsd send "127.0.0.1" "8125" (concat "[" [m] "]") "{\"flavor\": \"dogstatsd\", \"prefix\": \"mentdb.\", \"tags\": {\"env\": \"prod\"}, \"max_packet\": 128, \"dry_run\": true}"); -> "[res]" (concat "lines: " (json select "rx_s" "/lines") " | datagrams of 128 bytes at most: " (json select "rx_s" "/packets") " | first line: " (string substr (json select "rx_s" "/payloads/[0]") 0 46)); json unload "rx_s"; [res]
lines: 6 | datagrams of 128 bytes at most: 3 | first line: mentdb.queue.depth:17|g|#queue:orders,env:prod
- Production: remove dry_run. UDP has no answer, so a stopped agent is not an error: watch the agent itself.
- Without Datadog, use the default flavor etsy (no tags): put the queue in the name instead, for example queue.orders.depth.
- For a collector that speaks OpenTelemetry, send the same gauges with otel metrics and the attribute queue.
