Problem / 问题
SEL noise hides the critical events. Presence, boot, and cleared lines bury the one row that should page someone. Triage is tribal knowledge: the same phrase is cooling on a temperature sensor and a rail on a voltage sensor, and the person who knows the difference is not on every shift.
系统事件日志的噪声会把严重事件盖住。在位、启动和已经解除的行，把真正该叫醒人的那一行埋掉。分诊是口头经验：同一句短语在温度上是散热，在电压上是电源轨，而知道差别的人并不是每个班次都在。
Approach / 做法
A code table maps the sensor type and the event phrase to a severity, a category, and a runbook. A deasserted line drops one severity step. The script groups the capture by host and category, prints the runbook path, and exits 2 while any critical row remains. The flowchart below is the same path on paper. Samples are synthetic hosts only.
码表把传感器类型和事件短语对应到严重级别、类别和手册。已解除的行把级别降一档。脚本按主机和类别汇总，打印手册路径，只要还剩严重项就退出码 2。下面的流程图是同一条路径。示例主机都是合成的。
Triage flowchart / 分诊流程
Collect the SEL and SDR, match the phrase, lower a deasserted line, then open the runbook, schedule a window, or record it. Exit 0 when nothing critical remains. Exit 2 when it does.
先收集 SEL 和 SDR，匹配短语，已解除的降一档，然后打开手册、排窗口，或只记录。没有严重项是退出码 0，还有就是 2。
Demo output / 演示输出
Stdout of python3 demo.py on the two synthetic hosts. The parser reports 59 findings, 19 critical, and exits 2. The demo process then exits 0.
python3 demo.py 对两台合成主机的标准输出。解析结果是 59 条发现、严重 19 条、退出码 2。演示进程随后以 0 结束。
BMC/IPMI fault code kit demo $ python3 scripts/ipmi_triage.py --sel samples/srv-a01_sel_elist.txt samples/srv-b02_sel_elist.txt --sdr samples/srv-a01_sdr.txt samples/srv-b02_sdr_elist.txt --out out findings: 59 critical: 19 warning: 25 info: 15 exit: 2 Host Category Severity Count Action Runbook ------------------ --------- -------- ----- ------------ ------------------------ dc1-rack-a01-srv01 battery warning 2 RB-BATTERY runbooks/RB-BATTERY.md dc1-rack-a01-srv01 battery info 1 RB-BATTERY runbooks/RB-BATTERY.md dc1-rack-a01-srv01 bmc critical 1 RB-BMC runbooks/RB-BMC.md dc1-rack-a01-srv01 bmc info 1 RB-BMC runbooks/RB-BMC.md dc1-rack-a01-srv01 cpu critical 1 RB-CPU runbooks/RB-CPU.md dc1-rack-a01-srv01 cpu warning 1 RB-CPU runbooks/RB-CPU.md dc1-rack-a01-srv01 fan critical 2 RB-FAN runbooks/RB-FAN.md dc1-rack-a01-srv01 fan warning 2 RB-FAN runbooks/RB-FAN.md dc1-rack-a01-srv01 fan info 1 RB-FAN runbooks/RB-FAN.md dc1-rack-a01-srv01 intrusion critical 1 RB-INTRUSION runbooks/RB-INTRUSION.md dc1-rack-a01-srv01 memory critical 1 RB-MEM-ECC runbooks/RB-MEM-ECC.md dc1-rack-a01-srv01 memory warning 1 RB-MEM-ECC runbooks/RB-MEM-ECC.md dc1-rack-a01-srv01 pcie critical 1 RB-PCIE runbooks/RB-PCIE.md dc1-rack-a01-srv01 pcie warning 1 RB-PCIE runbooks/RB-PCIE.md dc1-rack-a01-srv01 power critical 2 RB-PSU runbooks/RB-PSU.md dc1-rack-a01-srv01 power warning 3 RB-PSU runbooks/RB-PSU.md dc1-rack-a01-srv01 power info 2 RB-PSU runbooks/RB-PSU.md dc1-rack-a01-srv01 sel critical 1 RB-SEL-FULL runbooks/RB-SEL-FULL.md dc1-rack-a01-srv01 sel warning 1 RB-SEL-FULL runbooks/RB-SEL-FULL.md dc1-rack-a01-srv01 sel info 1 RB-SEL-FULL runbooks/RB-SEL-FULL.md dc1-rack-a01-srv01 thermal critical 2 RB-THERMAL runbooks/RB-THERMAL.md dc1-rack-a01-srv01 thermal warning 4 RB-THERMAL runbooks/RB-THERMAL.md dc1-rack-a01-srv01 thermal info 1 RB-THERMAL runbooks/RB-THERMAL.md dc1-rack-a01-srv01 unknown info 1 - dc1-rack-a01-srv01 watchdog critical 2 RB-WATCHDOG runbooks/RB-WATCHDOG.md dc1-rack-a01-srv01 watchdog info 1 RB-WATCHDOG runbooks/RB-WATCHDOG.md dc1-rack-b02-srv02 battery warning 1 RB-BATTERY runbooks/RB-BATTERY.md dc1-rack-b02-srv02 bmc warning 1 RB-BMC runbooks/RB-BMC.md dc1-rack-b02-srv02 bmc info 1 RB-BMC runbooks/RB-BMC.md dc1-rack-b02-srv02 cpu critical 1 RB-CPU runbooks/RB-CPU.md dc1-rack-b02-srv02 cpu warning 1 RB-CPU runbooks/RB-CPU.md dc1-rack-b02-srv02 fan warning 2 RB-FAN runbooks/RB-FAN.md dc1-rack-b02-srv02 memory info 1 RB-MEM-ECC runbooks/RB-MEM-ECC.md dc1-rack-b02-srv02 pcie warning 1 RB-PCIE runbooks/RB-PCIE.md dc1-rack-b02-srv02 power critical 2 RB-PSU runbooks/RB-PSU.md dc1-rack-b02-srv02 power warning 2 RB-PSU runbooks/RB-PSU.md dc1-rack-b02-srv02 power info 1 RB-PSU runbooks/RB-PSU.md dc1-rack-b02-srv02 storage critical 1 RB-STORAGE runbooks/RB-STORAGE.md dc1-rack-b02-srv02 storage warning 1 RB-STORAGE runbooks/RB-STORAGE.md dc1-rack-b02-srv02 thermal info 1 RB-THERMAL runbooks/RB-THERMAL.md dc1-rack-b02-srv02 watchdog critical 1 RB-WATCHDOG runbooks/RB-WATCHDOG.md dc1-rack-b02-srv02 watchdog warning 1 RB-WATCHDOG runbooks/RB-WATCHDOG.md dc1-rack-b02-srv02 watchdog info 2 RB-WATCHDOG runbooks/RB-WATCHDOG.md recommended: - dc1-rack-a01-srv01 bmc critical x1 what: The management controller is off-line action: Save the SEL first. Reset the management controller only after the log is stored. runbook: runbooks/RB-BMC.md - dc1-rack-a01-srv01 cpu critical x1 what: The processor reported an internal error action: Save the SEL. Treat a repeated IERR, thermal trip, or machine check as a board fault. runbook: runbooks/RB-CPU.md - dc1-rack-a01-srv01 fan critical x2 what: Fan speed crossed the lower critical mark action: Confirm the fan is seated and spinning. Swap it if it stays below the low mark. runbook: runbooks/RB-FAN.md - dc1-rack-a01-srv01 intrusion critical x1 what: The chassis was opened action: Confirm someone was in the rack, close the chassis, and clear the latch on purpose. runbook: runbooks/RB-INTRUSION.md - dc1-rack-a01-srv01 memory critical x1 what: Memory logged an uncorrectable ECC error action: Map the DIMM, reseat once, and replace it if uncorrectable errors return. runbook: runbooks/RB-MEM-ECC.md - dc1-rack-a01-srv01 pcie critical x1 what: A PCI system error was signaled action: Note the slot, reseat the card, and move it before calling the board failed. runbook: runbooks/RB-PCIE.md - dc1-rack-a01-srv01 power critical x2 what: A power supply lost AC input action: Check the feed, the supply LED, and redundancy before pulling a supply. runbook: runbooks/RB-PSU.md - dc1-rack-a01-srv01 sel critical x1 what: The system event log is full action: Export the SEL, store the file, and only then clear the log. runbook: runbooks/RB-SEL-FULL.md - dc1-rack-a01-srv01 thermal critical x2 what: Temperature crossed the upper critical mark action: Check inlet temperature, fan speed, and blanking before replacing a heat sink. runbook: runbooks/RB-THERMAL.md - dc1-rack-a01-srv01 watchdog critical x2 what: The watchdog timer expired action: Save the console. Reboot only after deciding the OS hung rather than a firmware loop. runbook: runbooks/RB-WATCHDOG.md - dc1-rack-b02-srv02 cpu critical x1 what: The processor shut down on a thermal trip action: Save the SEL. Treat a repeated IERR, thermal trip, or machine check as a board fault. runbook: runbooks/RB-CPU.md - dc1-rack-b02-srv02 power critical x2 what: Power redundancy is gone action: Check the feed, the supply LED, and redundancy before pulling a supply. runbook: runbooks/RB-PSU.md - dc1-rack-b02-srv02 storage critical x1 what: A drive reported a fault action: Name the bay and replace a faulted drive before a second member fails. runbook: runbooks/RB-STORAGE.md - dc1-rack-b02-srv02 watchdog critical x1 what: The watchdog forced a hard reset action: Save the console. Reboot only after deciding the OS hung rather than a firmware loop. runbook: runbooks/RB-WATCHDOG.md wrote out/report.md (17187 bytes) wrote out/report.csv (22882 bytes) wrote out/report.json (71267 bytes) triage exit 2 demo complete
Deliverables / 交付物
- Bilingual SEL quick reference and twelve fault categories. 中英速查和十二类故障。
- Code table, 81 rows, each with an action id. 81 行码表，每行有手册编号。
- Twelve bilingual runbooks: symptoms, ipmitool checks, steps, escalation and RMA, ticket note. 十二本中英手册。
- Triage flowchart (Mermaid, SVG, PNG). 分诊流程图。
- SEL/SDR parser with markdown, CSV, and JSON reports. SEL/SDR 解析和三份报告。
- Synthetic two-host sample,
demo.py, unit tests, and an acceptance script. 两台合成主机、演示、单元测试和验收脚本。
The free lite repo keeps the parser, a 12-row thermal and power subset, the flowchart, and RB-THERMAL plus RB-PSU. 免费精简仓库保留解析器、十二行温度和电源子集、流程图，以及这两本手册。
Delivery time / 交付时间
Instant download. Setup about 5 minutes. Adapting the code table and runbooks to your fleet about 30 minutes.
下载即得。安装并跑通大约 5 分钟。把码表和手册改成你的机器大约 30 分钟。
Pay what you want, $1 or more. Suggested price is $5. 愿付多少付多少，1 美元起，建议 5 美元。
Links / 链接
GitHub: bmc-ipmi-kit-lite Gumroad: full kit
Full n8n DevOps pack, $9 / 完整 n8n DevOps 包 9 美元: https://caoyadong.gumroad.com/l/yvmcqj
DC Inspection & Asset Register Kit / 数据中心巡检与资产台账: https://caoyadong.gumroad.com/l/hxlops