DeepWing深翼 DeepWing
EN中文

深度调研 · 工业生产 · 2026-10-19Deep Research · Industrial Production · Oct 19, 2026

工业 AI 七层单点失效：哪一层出事，出大事Industrial AI, Seven Layers: Which Single Layer Fails Catastrophically?

作者By 深翼 DeepWingDeepWing·发布Published 2026-10-19·最后核查Last checked 2026-10-19·阅读Read 约 22 分钟~27 min
本篇涉及Layers covered模型Model编排Orchestration工具Tool Use记忆Memory知识Knowledge门禁Access Control盯梢Observability逐层评审全部七层All seven layers, reviewed one by one

AI 系统与架构AI Systems & Architecture跨区：AI 安全与治理Cross-tag: AI Safety & Governance阅读时间：约 22 分钟Reading time: About 27 min

研究问题Research question
把七层框架搬进工业生产，在 AI Agent 已被授予工具调用权、能影响生产系统的前提下，哪一层单点失效最可能酿成大事故？现有解法成熟到什么程度，机会又在哪里？Take the seven layer framework into industrial production, on the premise that AI agents already hold tool calling rights and can affect production systems. Which single layer's failure is most likely to become a major accident, how mature are the existing solutions, and where are the unclaimed openings?
研究方法Method
按深翼架构拆解模板逐层评审：七层逐层列典型失效模式、大事故路径、解法成熟度（四档）与未满足机会；每层承重判断逐条挂证据锚点，八个锚点案例逐个标证据等级（事实、类比、风险推演）；六维评分排序（影响半径、物理触达、不可逆性、检测难度、恢复难度、绕过传统安全层的能力，各 1 至 5 分，满分 30）；证据优先取监管机构、标准组织、事故调查机构与同行评审来源，排序与评分是本简报基于所引证据的评估判断。Reviewed layer by layer under DeepWing's architecture teardown template: for each of the seven layers, typical failure modes, the major accident path, solution maturity on a four step scale, and the unmet opening. Load bearing judgments at each layer carry claim level evidence anchors, and eight anchor cases are each labeled fact, analogy, or risk projection. Ranking by six scored dimensions (blast radius, physical reach, irreversibility, detection difficulty, recovery difficulty, and ability to bypass traditional safety layers, 1 to 5 each, 30 maximum). Evidence favors regulators, standards bodies, accident investigators and peer reviewed sources; the ranking, scores and maturity calls are this briefing's own assessment on that evidence.
证据等级Evidence grade
多源已核Multi-source verified
核查截止日期Verified through
2026-10-192026-10-19
核心发现Key findings
① 门禁层与工具层同为 27 分并列最高，门禁居首：它是错误到达物理世界之前最后一道可问责的放行决定，一旦过宽、被冒用或与被审查的 AI 共用身份与模型，属共因失效，且在日志里长得最合规、最难发现。② 截至本次调研，公开记录里还没有一起经独立调查确认的「大模型 Agent 在工厂里自主闯祸、直接造成重大伤亡或重大损失」的工业大事故；已有的硬案例分布在传统自动化事故、针对安全系统的网络攻击、IT 场景的 Agent 越权，以及受监管生产中 AI 生成文件未经审核的执法案例。③ 解法成熟度落差最大的是门禁层：传统 OT 门禁已经成熟，但动作级、参数级、上下文相关的 Agent 授权基本空白，没有对标 IEC 62443 安全等级的 Agent 门禁产品类别，也没有动作签名与可撤销授权的工业标准。1. Access Control and the Tool layer tie at 27 of 30, with Access Control ranked first: it is the last accountable release decision before an error reaches the physical world, and once it is too broad, borrowed, or sharing an identity and a model with the AI it checks, the failure is a common cause failure, and the one that looks most compliant in the logs. 2. As of this research, the public record holds no independently investigated case of an LLM agent autonomously causing a major industrial accident; the hard cases sit in traditional automation accidents, cyberattacks on safety systems, agent overreach in IT settings, and enforcement over unreviewed AI generated documents in regulated production. 3. The widest maturity gap is at Access Control: traditional OT access control is mature, but action level, parameter level, context aware authorization for agents is largely missing, with no agent gate product category benchmarked to IEC 62443 security levels and no industrial standard for action signatures and revocable authorization.
研究局限Limitations
排序、评分与成熟度均为简报评估，不构成精确度量；「工业 AI 大事故零记录」是本次检索范围（公开英文来源为主）内的判断，不能排除未公开、非英文或以传统事故名义归档的事件；CISA 的 OT AI 指南原文、EU AI Act 时间线、三份文件的具体条款未活验原文。All rankings, scores and maturity calls are this briefing's assessments, expressing relative judgment rather than precise measurement. The "zero recorded industrial AI catastrophes" finding holds within this search's scope (mainly public English language sources) and cannot exclude undisclosed, non English, or traditionally filed incidents. The CISA OT AI guidance original, the EU AI Act timeline, and the specific clauses of three further documents were not live verified against originals.
后续观察Watch next
CISA 的 OT AI 指南原文核验，EU AI Act 与 Digital Omnibus 时间线以欧盟官方公报为准复核；Agent 门禁的产品类别与动作签名标准是否出现；跨层事故证据链（OT 日志、Agent 轨迹、门禁决定、知识版本拼成一条时间线）是否出现可落地的产品或标准；工业场景是否出现第一起经独立调查确认的 Agent 事故。Live verification of the CISA OT AI guidance and of EU AI Act and Digital Omnibus dates against the EU Official Journal; whether an agent gate product category and action signature standards emerge; whether a cross layer incident evidence chain (OT logs, agent traces, gate decisions, knowledge versions on one timeline) appears as a product or standard; whether industry sees its first independently investigated agent accident.

把 DeepWingAI 的七层框架搬进工业生产：离散制造、流程工业、能源、受监管的制药与食品生产、仓储物流与工业机器人。调研的前提是：AI Agent 已经被授予工具调用权，能影响生产系统。在这个前提下逐层核对：哪一层最可能「一层出事就出大事」，现有解法成熟到什么程度，还有哪些机会没人接。证据优先取监管机构、标准组织、事故调查机构与同行评审来源；排序、评分与成熟度是本简报基于所引证据的评估判断，不是任何单一来源的结论。 The seven-layer framework from the book, taken into industrial production, discrete manufacturing, process industries, energy, regulated pharma and food production, warehousing, and industrial robotics. The premise: AI agents have already been granted tool-calling rights and can affect production systems. Under that premise, this briefing checks, layer by layer, which one is most likely to turn a single failure into a major accident, how mature the existing solutions are, and which openings nobody has claimed yet. Evidence favors regulators, standards bodies, accident investigators, and peer-reviewed sources; the ranking, scores, and maturity calls are this briefing's own assessment, not any single source's conclusion.

第一名是门禁层，不是模型层。First place: Access Control, not the Model layer.

门禁层Access Control27/30
工具层Tool27/30
编排层Orchestration23/30

模型会犯错是常态，编排会放大，工具会执行；但只有门禁层是被设计来在错误到达物理世界之前把它拦下来的最后一道「可问责的放行决定」。门禁一旦过宽、被冒用、被橡皮图章化，或与被它审查的 AI 共用同一身份、同一模型、同一云租户，它就不是又坏了一个零件，而是让上面五层的错误全部获得合法外衣，同时让传统工业安全所依赖的「独立性」假设失效：这是典型的共因失效。它在日志里还长得最合规：合法身份、合法时段、形式上存在的审批，最难被发现。Models err as a matter of course, orchestration amplifies, tools execute, but only Access Control is designed to be the last accountable release decision before an error reaches the physical world. Once it is too broad, borrowed, rubber-stamped, or sharing an identity, a model, or a cloud tenant with the AI it is supposed to check, it is not one more broken part: it gives every error from the five layers above a legal disguise, and it voids the independence assumption that traditional industrial safety rests on, a textbook common-cause failure. It also looks the most compliant in the logs: valid identity, valid hours, a formally existing approval. The hardest failure to spot.

工具层与门禁层同分（27 分）。同分时门禁层排在前面，理由是：工具层犯错时，系统里还有别人可以说不（控制器量程校验、独立 SIS 跳车、门禁限额）；门禁层失效时，没有别人了。工具层是扣扳机的手，门禁层是发枪与保险。The Tool layer ties Access Control at 27 points. The tie-breaker: when the Tool layer errs, someone else can still say no: controller range checks, an independent SIS trip, access limits. When Access Control fails, nobody is left. Tools are the hand on the trigger; Access Control issues the gun and the safety catch.

先说清一条边界：截至本次调研，公开记录里还没有一起经独立调查确认的「大模型 Agent 在工厂里自主闯祸、直接造成重大伤亡或重大损失」的工业大事故。已有的硬案例分布在传统自动化事故、针对安全系统的网络攻击、IT 场景的 Agent 越权，以及受监管生产中 AI 生成文件未经审核的执法案例。本页每个案例都标了证据等级：事实、类比，还是风险推演。 One boundary, stated plainly: as of this research, the public record holds no independently investigated case of an LLM agent autonomously causing a major industrial accident. The hard cases sit in traditional automation accidents, cyberattacks aimed at safety systems, agent overreach in IT settings, and enforcement over unreviewed AI-generated documents in regulated production. Every case on this page is labeled: fact, analogy, or risk projection.

排序与评分The Ranking and Scores

六个维度各 1 至 5 分（5 = 最危险）：影响半径、物理触达、不可逆性、检测难度、恢复难度、绕过传统安全层的能力。满分 30。Six dimensions, 1 to 5 each (5 = most dangerous): blast radius, physical reach, irreversibility, detection difficulty, recovery difficulty, and the ability to bypass traditional safety layers. Maximum 30.

名次Rank 层Layer 定性Character 合计Total
1 门禁层Access Control 定性Character共因失效点：失效不制造错误，但让所有错误合法通行，且最难被发现Common-cause failure point: it creates no errors, but lets all of them pass as authorized, and is the hardest to detect 27
2 工具层Tool 定性Character直接失效点：离物理世界最近，参数或目标错一次即可造成不可逆后果Direct failure point: closest to the physical world; one wrong parameter or target can be irreversible 27
3 编排层Orchestration 定性Character放大失效点：循环与多 Agent 级联以机器速度放大，还能分拆动作绕过单步限额Amplifying failure point: loops and multi-agent cascades scale errors at machine speed, and split actions to slip under per-step limits 23
4 知识层Knowledge 定性Character慢毒失效点：错版、过期规程看起来完全可信，影响所有班次，潜伏期最长Slow-poison failure point: wrong or expired procedures look fully credible, hit every shift, and hide the longest 20
5 盯梢层Observability 定性Character时长失效点：本身不发起事故，但决定事故持续多久、能否复盘Duration failure point: it starts no accident, but decides how long one lasts and whether it can be reconstructed 18
6 记忆层Memory 定性Character潜伏失效点：污染可跨会话存活、跨用户串扰，检测极难（与盯梢层同分）Latent failure point: contamination survives across sessions and leaks across users; extremely hard to detect (tied with Observability) 18
7 模型层Model 定性Character高频低烈度失效点：错误最频繁，但多为「建议错误」，缓解工具箱也最成熟High-frequency, low-severity failure point: errors are most frequent but mostly bad advice, and the mitigation toolbox is the most mature 16

一个限定条件：如果某工厂把模型输出直接闭环接入控制回路、中间没有独立门禁（例如某些高级过程控制、视觉分选直接剔除或放行），那么在该工厂里模型层与工具层事实上合并，模型层应升至第 2 名。排序不是教条，是架构的函数。One condition: if a plant wires model output straight into a closed control loop with no independent gate in between (some advanced process control, vision sorting that rejects or releases directly), the Model and Tool layers effectively merge in that plant, and Model rises to No. 2. The ranking is not dogma; it is a function of the architecture.

工业 AI 的胜负不在模型更聪明，而在谁掌握了让机器动作获得合法身份、被签名放行、能被瞬间撤销的那道门：那是工业 Agent 时代的 SIS。Industrial AI will not be won by the smartest model, but by whoever controls the gate that gives a machine action a legal identity, a signed release, and instant revocation, the SIS of the industrial agent era.

SIS = 安全仪表系统（Safety Instrumented System）：流程工厂里独立于生产控制的保命系统，传感器发现异常、逻辑器判断、最终元件动手，把装置带回安全状态。它的根基是独立：生产控制坏了，它不能跟着一起坏。SIS = Safety Instrumented System: the plant's independent life-safety system: sensors detect, a logic solver decides, final elements act to bring the process to a safe state. Its foundation is independence: when production control fails, it must not fail with it.

逐层档案Layer by Layer

点一层看一层：典型失效模式、大事故路径、现有解法的成熟度，和还没被满足的机会。成熟度四档：① 已在工业现场广泛落地 ② 局部试点 ③ 主要停留在 IT／云端 Agent 场景 ④ 仍属早期研究或营销概念。One layer at a time: typical failure modes, the major-accident path, how mature the existing solutions are, and the unmet opening. Maturity scale: ① widely deployed on industrial sites ② limited pilots ③ mostly IT / cloud-agent territory ④ early research or marketing concept.

门禁层 · 第 1 名 · 27／30Access Control · No. 1 · 27/30

影响半径 5 · 物理触达 4 · 不可逆性 5 · 检测难度 4 · 恢复难度 4 · 绕过传统安全层 5Radius 5 · Physical reach 4 · Irreversibility 5 · Detection 4 · Recovery 4 · Bypassing safety layers 5

失效模式：Failure modes:权限过宽（Agent 共用人类或管理员身份、长期令牌）；审批失效（审批疲劳、批量通过、审批人看不懂机器速度的动作清单）；身份冒用与混淆代理；策略只写在提示词里、没有系统级强制；AI 审 AI（门禁模型与执行模型同源）；紧急通道常态化；任务结束权限不回收。Over-broad permissions (agents sharing human or admin identities, long-lived tokens); approval failure (fatigue, batch approvals, reviewers who cannot parse machine-speed action lists); identity misuse and the confused deputy; policy written only in prompts with no system-level enforcement; AI reviewing AI (gate and executor sharing one model lineage); emergency channels becoming the default; permissions never revoked after a task.

大事故路径：Accident path:上游任何一种错误（模型幻觉、编排循环、知识错版、记忆污染）都能以「已授权」的身份直达生产系统；一次配置错误可以全厂 7×24 长期生效，且恢复不只是改配置，必须假设存续期内所有「合法」动作都可疑，做全量审计与批次影响评估。Any upstream error, hallucination, orchestration loop, wrong procedure version, poisoned memory, reaches production stamped "authorized." One misconfiguration can run plant-wide, 24/7, until it is found; recovery is not just fixing the config, it means treating every "legal" action in the window as suspect: full audit and, in GxP plants, batch impact assessment.

解法成熟度：Solution maturity:传统 OT 门禁是成熟的（①）：ISA/IEC 62443 分区与管道、最小权限、变更管理（MOC）、作业许可、联锁旁路管理。但面向 Agent 的等价物基本空白（③→④）：传统 IAM 管的是「人或服务账号能进哪个系统」，Agent 需要的是「这一次动作是否被授权」：动作级、参数级、上下文相关的授权。没有与 62443 安全等级对标的 Agent 门禁产品类别，没有动作签名与可撤销授权的工业标准，更没有门禁失效时的失效安全语义（断网时默认拒绝还是默认放行）。这是成熟度落差最大的一层。Traditional OT access control is mature (①): ISA/IEC 62443 zones and conduits, least privilege, management of change, permits to work, interlock bypass management. The agent equivalent is largely missing (③→④): legacy IAM governs which systems a person or service account may enter; an agent needs authorization for this one action, action-level, parameter-level, context-aware. There is no agent-gate product category benchmarked to 62443 security levels, no industrial standard for action signatures and revocable authorization, and no agreed fail-safe semantics (on network loss: deny by default, or allow?). The widest maturity gap of the seven layers.

未满足的机会：Unmet opening:动作签名、短时可撤销授权、参数级策略，加上 Agent 身份与委托链基础设施：本简报的第一机会。Action signatures, short-lived revocable authorization, parameter-level policy, plus agent identity and delegation-chain infrastructure, this briefing's No. 1 opportunity.

证据锚点：Evidence anchors:Replit 删库（事实 · IT 场景）、EchoLeak CVE-2025-32711（事实 · 漏洞披露）、TRITON（类比）；OWASP 2026 版 LLM Top 10 把「过度代理／过度授权」升至第 3 名，且首次引入真实事故数据加权。The Replit deletion (fact · IT setting), EchoLeak CVE-2025-32711 (fact · disclosed vulnerability), TRITON (analogy); OWASP's 2026 LLM Top 10 raised Excessive Agency to No. 3, weighted for the first time by real incident data.

工具层 · 第 2 名 · 27／30Tool · No. 2 · 27/30

影响半径 5 · 物理触达 5 · 不可逆性 5 · 检测难度 3 · 恢复难度 5 · 绕过传统安全层 4Radius 5 · Physical reach 5 · Irreversibility 5 · Detection 3 · Recovery 5 · Bypassing safety layers 4

失效模式：Failure modes:参数错误（单位、数量级、小数点、目标对象）；同名工具或同名设备选错；非幂等重试造成重复下单、重复加料；工具返回内容被当作指令执行（工具投毒）；API 或协议版本不匹配；超时后状态未知仍继续；机器人或执行器在维护、示教、异常恢复等非正常模式下动作。Parameter errors (units, orders of magnitude, decimal points, wrong target); same-name tools or devices confused; non-idempotent retries duplicating orders or charges; tool output treated as instructions (tool poisoning); API or protocol version mismatch; pressing on after a timeout with unknown state; robots and actuators moving in maintenance, teach, or recovery modes.

大事故路径：Accident path:离物理世界最近，路径最短：设定值写错、对 A 线下 B 线的参数、直接驱动执行器，后果是设备损坏、批次报废或人身伤害，不可逆。但它之下还有不依赖 AI 的防线：控制器量程校验、硬联锁与 SIS，前提是它们没被 AI 通道绕过。Closest to the physical world, with the shortest path: a wrong setpoint, line A's parameters sent to line B, an actuator driven directly, equipment damage, scrapped batches, or injury, irreversibly. But beneath it sit defenses that do not depend on AI, controller range checks, hard interlocks, and the SIS, provided the AI channel has not bypassed them.

解法成熟度：Solution maturity:设备级安全广泛落地（①）：OPC UA 等工业互联标准、写操作参数模式校验与量程钳制、写前仿真、双写确认、幂等键、安全 PLC、光幕、急停、力限制；ISO 10218-1/-2:2025 已发布修订，把协作机器人要求整合进正文并首次纳入机器人网络安全要求。缺的是把同一套思想翻译给软件 Agent 的工具调用：面向 Agent 的工具网关认证、授权与审计仍在补课（③）。韩国与 Tesla 的案例恰恰发生在维护、未停用等安全状态管理失效的缝隙里。Equipment-level safety is widely deployed (①): OPC UA and other industrial interconnect standards, write-time schema checks and range clamping, pre-write simulation, dual confirmation, idempotency keys, safety PLCs, light curtains, e-stops, force limits; ISO 10218-1/-2:2025 is published, folding collaborative-robot requirements into the main text and adding robot cybersecurity requirements for the first time. What is missing is the same thinking translated to agent tool calls: authentication, authorization, and audit for agent tool gateways are still catching up (③). The Korean and Tesla cases happened exactly in the seams, maintenance and not-powered-down states where safety-state management failed.

未满足的机会：Unmet opening:工艺语义级的写前验证（不只查格式，还查单位、物料平衡、设备状态前提）、工具签名与工具描述完整性。Process-semantics pre-write verification (not just format, units, material balance, equipment-state preconditions), plus tool signing and tool-description integrity.

证据锚点：Evidence anchors:2023 年韩国码垛机器人致死（事实 · 非 AI）；2024 年 CrowdStrike 更新导致全球 Windows 大规模崩溃、制造业亦受波及（事实 · 更新通道的「一次推送、全网生效」半径；其中制造业损失约 3,600 万美元为单一保险机构估算，仅作量级参考）。The 2023 Korean palletizing-robot fatality (fact · non-AI); the 2024 CrowdStrike update crashing Windows systems worldwide, manufacturing included (fact · the "one push, everywhere at once" radius of update channels; the roughly $36M manufacturing loss figure is a single insurer's estimate, order-of-magnitude only).

编排层 · 第 3 名 · 23／30Orchestration · No. 3 · 23/30

影响半径 5 · 物理触达 3 · 不可逆性 3 · 检测难度 4 · 恢复难度 4 · 绕过传统安全层 4Radius 5 · Physical reach 3 · Irreversibility 3 · Detection 4 · Recovery 4 · Bypassing safety layers 4

失效模式：Failure modes:无限重试与死循环；多 Agent 目标冲突与责任稀释；长链任务中途状态不一致；子任务分拆绕过单步限额；规划基于错误前提却完整执行；并发 Agent 竞争同一资源（两条排程同时改一台设备）。Unbounded retries and loops; multi-agent goal conflicts and diluted responsibility; mid-chain state inconsistency on long tasks; subtask splitting that slips under per-step limits; a plan built on a false premise executed flawlessly; concurrent agents contending for the same resource (two schedulers retuning one machine).

大事故路径：Accident path:危险不在单步，在组合与速度：分钟级把一次小错执行成千上万次；每一步都「在限额内」的连续动作累积成越限总果，单动作审批对此天然失明：这是编排层与门禁层最重要的耦合。它本身不碰物理世界，必须借工具层之手；且通常留有明显异常信号，可用全局熔断、预算与步数上限一刀切断，存在有效的总开关。The danger is not the single step but combination and speed: one small error executed thousands of times within minutes; a sequence of individually "within-limit" actions accumulating past the limit, invisible to per-action approval, the key coupling between this layer and Access Control. It never touches the physical world itself; it works through the Tool layer, and it usually leaves loud signals, so a global circuit breaker, budgets, and step caps can cut it off, an effective master switch exists.

解法成熟度：Solution maturity:③（工业侧 ② 起步）。IT 侧已成型的做法：确定性工作流骨架加 LLM 只做局部决策、步数／时长／成本预算上限、全局熔断与一键停机、每步检查点与可重放、幂等与补偿事务、多 Agent 资源锁。工业侧 Siemens Industrial Copilot 等已把生成式 AI 嵌入工程环境，但目前以工程辅助与建议为主，不等于具备生产动作编排的安全语义。编排的「能跑」已解决，「能证明不会失控」未解决。③ (industrial side at ②, starting). Mature on the IT side: deterministic workflow skeletons with LLMs making only local decisions, step/time/cost budgets, global circuit breakers and one-click stops, per-step checkpoints with replay, idempotency and compensating transactions, resource locks between agents. On the industrial side, products like Siemens Industrial Copilot embed generative AI in engineering environments, but as engineering assistance and advice, not production-action orchestration with safety semantics. Orchestration "runs"; proving it cannot run away is unsolved.

未满足的机会：Unmet opening:动作预算与分拆检测、跨 Agent 互锁、全局熔断的工业产品化。Industrial productization of action budgets and split detection, cross-agent interlocks, and global circuit breakers.

证据锚点：Evidence anchors:Knight Capital 2012（类比 · 事实事故）；工业多 Agent 级联事故尚无公开调查记录，属风险推演。Knight Capital 2012 (analogy · real accident); no public investigation of an industrial multi-agent cascade exists, that scenario remains a risk projection.

知识层 · 第 4 名 · 20／30Knowledge · No. 4 · 20/30

影响半径 4 · 物理触达 2 · 不可逆性 3 · 检测难度 5 · 恢复难度 3 · 绕过传统安全层 3Radius 4 · Physical reach 2 · Irreversibility 3 · Detection 5 · Recovery 3 · Bypassing safety layers 3

失效模式：Failure modes:规程错版、过期版仍被 RAG 检索（新版已发布、旧版未下线）；多来源冲突时无优先级（厂商手册 vs 本厂 SOP vs 个人笔记）；文档本身由 AI 生成、未经审核即成为「知识」；引用不可追溯（答案对但出处错，审计时无法证明依据）；口头传统未入库，模型以通用知识补齐：听起来合理，本厂不适用。Wrong or expired procedure versions still retrieved by RAG (new version issued, old one never retired); no priority order when sources conflict (vendor manual vs. plant SOP vs. personal notes); documents generated by AI becoming "knowledge" without review; untraceable citations (right answer, wrong source, unprovable in an audit); unwritten shop-floor tradition filled in by the model with generic knowledge, plausible, and wrong for this plant.

大事故路径：Accident path:错版规程被所有班次、所有下游 Agent 一致执行：不是一个人看错一份文件，而是全厂系统性地按错的做。Purolea 案还证明，这条路径不需要任何黑客：只需要没人审核：知识层与门禁层同时失效，AI 生成物绕过质量单位直接成为生效规程。A wrong procedure version executed consistently by every shift and every downstream agent: not one person misreading one document, but the whole plant systematically doing it wrong. The Purolea case also proves this path needs no hacker, only nobody reviewing: Knowledge and Access Control failed together, and AI output bypassed the quality unit to become an effective procedure.

解法成熟度：Solution maturity:②。受控文档管理（版本、生效／失效、培训签收）在制药等受监管行业是几十年的既有体系（QMS/DMS、21 CFR Part 11、ALCOA+ 数据完整性）；成败取决于是否把 RAG 语料库当作受控文档来管。关键缺口：多数企业 RAG 上线只做了检索效果，没有做语料的生效版本绑定、过期自动下线、冲突优先级与引用级溯源。解法在制度层是现成的，在工具层是缺失的：典型的「不需要发明、需要产品化」。②. Controlled document management (versions, effective/retired dates, training sign-off) is a decades-old system in regulated industries (QMS/DMS, 21 CFR Part 11, ALCOA+ data integrity); success depends on treating the RAG corpus as controlled documents. The gap: most enterprise RAG launches tune retrieval quality only: no effective-version binding, no automatic expiry, no conflict priority, no citation-level traceability. The solution exists at the policy level and is missing at the tool level, a classic "no invention needed, productization needed" gap.

未满足的机会：Unmet opening:受控 RAG：生效版本绑定、过期自动下线、冲突优先级、引用级溯源，加上 AI 生成物强制走质量单位审核流。Controlled RAG: effective-version binding, automatic expiry and retirement, conflict priority, citation-level traceability, and a mandatory quality-unit review flow for AI-generated documents.

证据锚点：Evidence anchors:FDA Purolea 警告信（事实 · 监管执法）：本层目前最硬的工业监管案例；FDA 的态度很明确：AI 可以起草，质量单位的审核批准不可委托。The FDA's Purolea warning letter (fact · regulatory enforcement), this layer's hardest industrial case to date. The FDA's position is explicit: AI may draft; the quality unit's review and approval cannot be delegated.

盯梢层 · 第 5 名 · 18／30Observability · No. 5 · 18/30

影响半径 4 · 物理触达 1 · 不可逆性 3 · 检测难度 4 · 恢复难度 3 · 绕过传统安全层 3Radius 4 · Physical reach 1 · Irreversibility 3 · Detection 4 · Recovery 3 · Bypassing safety layers 3

失效模式：Failure modes:告警洪泛与告警疲劳（真信号被淹没）；日志不完整（只记模型回复，不记工具调用参数与门禁决定，事后无法重建证据链）；时钟与身份不一致，跨系统日志无法对齐；漂移检测只看输入分布、不看后果指标；AI 生成的「复盘总结」再次引入幻觉；监控与被监控对象同故障域，一起宕机、一起被攻破。Alarm floods and alarm fatigue (real signals drowned); incomplete logs (model replies recorded, tool-call parameters and gate decisions not, no evidence chain to rebuild afterwards); clock and identity mismatches making cross-system logs unalignable; drift detection watching input distributions but not outcome metrics; AI-written "incident summaries" hallucinating a second time; monitoring sharing a failure domain with the monitored, down together, breached together.

大事故路径：Accident path:本身不发起事故，但决定事故持续多久、能否复盘。当 Agent 以机器速度产生事件、建议与动作时，告警量只会比 Toledo 的 3,712 次／12 小时更高；没有分级与抑制，盯梢层会从保护层变成事故的共谋。It starts no accident, but decides how long one lasts and whether it can be reconstructed. When agents generate events, suggestions, and actions at machine speed, alarm volume will only exceed Toledo's 3,712 in 12 hours; without grading and suppression, this layer turns from a protection layer into an accomplice.

解法成熟度：Solution maturity:OT 网络监控成熟（①）：Claroty、Dragos、Nozomi Networks 等平台提供资产盘点、协议级监控与异常检测；报警管理有 ISA-18.2/EEMUA 方法论，CSB 对 Toledo 的建议表明标准侧仍在补强「短时报警洪泛」目标。断层在另一侧（③）：这些平台监控的是网络与设备，不是 Agent 的决策链：哪条知识被检索、哪个记忆被引用、门禁为何放行、参数从哪一步被改大，这类语义级证据链在 OT 侧基本空白，且与 Agent 日志分属两套系统，事故时拼不成一条时间线。OT network monitoring is mature (①): platforms such as Claroty, Dragos, and Nozomi Networks provide asset inventory, protocol-level monitoring, and anomaly detection; alarm management has ISA-18.2/EEMUA methodology, and the CSB's Toledo recommendations show the standards side still strengthening short-duration flood targets. The gap is on the other side (③): these platforms watch networks and devices, not agent decision chains: which knowledge was retrieved, which memory was cited, why the gate approved, at which step a parameter grew. That semantic evidence chain is largely absent in OT, and it lives in a separate system from agent logs, so an incident cannot be assembled into one timeline.

未满足的机会：Unmet opening:跨层事故证据链（OT 日志、Agent 轨迹、门禁决定、知识版本拼成一条时间线），以及按物理后果分级的 Agent 语义告警与洪泛抑制。A cross-layer incident evidence chain (OT logs, agent traces, gate decisions, knowledge versions on one timeline), plus agent semantic alerting graded by physical consequence, with flood suppression.

证据锚点：Evidence anchors:CSB Toledo 炼油厂最终报告（事实 · 事故调查）：本层最惨痛的流程工业实证。The CSB's final report on the Toledo refinery fire (fact · accident investigation), this layer's most painful process-industry evidence.

记忆层 · 第 6 名 · 18／30Memory · No. 6 · 18/30

影响半径 3 · 物理触达 1 · 不可逆性 3 · 检测难度 5 · 恢复难度 4 · 绕过传统安全层 2Radius 3 · Physical reach 1 · Irreversibility 3 · Detection 5 · Recovery 4 · Bypassing safety layers 2

失效模式：Failure modes:记忆投毒（恶意或错误的示范被写入长期记忆，后续会话被检索执行）；跨会话、跨租户、跨班次状态串扰（A 产线的状态被用到 B 产线）；过期状态冒充现况（「设备已检修」在复检前仍被引用）；记忆与现实脱钩后的自信行动；共享记忆单点污染放大（一处污染，全体 Agent 中招）。Memory poisoning (malicious or wrong exemplars written into long-term memory, later retrieved and executed); cross-session, cross-tenant, cross-shift state bleed (line A's state applied to line B); stale state posing as current ("equipment serviced" still cited before re-inspection); confident action on memory divorced from reality; shared-memory single-point contamination (one poisoning, every agent exposed).

大事故路径：Accident path:污染可跨会话存活、条件式触发：常规评测在没有触发词时完全测不出来，这与 AgentPoison 的隐蔽性结论一致。把同一手法平移到工厂：向维护 Agent 的长期记忆写入一条伪造的「该型号轴承可延长更换周期」经验，机理上完全成立（风险推演，无公开工业案例）。它通常仍需经工具层与门禁层才能变成物理后果，所以排在后面。Contamination survives across sessions and triggers conditionally, routine evaluation without the trigger detects nothing, matching AgentPoison's stealth findings. Transplanted to a plant: a forged "this bearing model tolerates extended replacement intervals" entry in a maintenance agent's long-term memory is mechanically sound (risk projection; no public industrial case). It usually still needs the Tool and Access layers to become a physical consequence, which is why it ranks lower.

解法成熟度：Solution maturity:④／③，七层中防御最薄的一层之一。现有做法停留在各 Agent 框架各自实现：会话隔离、租户隔离、记忆写入审核、TTL 与过期、向量库访问控制、内容溯源标签。没有公认的「记忆完整性」标准，没有成熟的污染检测产品；AgentPoison 已表明基于困惑度与改写的防御可被绕过，后续防御研究多在论文阶段。④/③: among the thinnest defenses of the seven layers. Current practice is per-framework improvisation: session isolation, tenant isolation, memory-write review, TTL and expiry, vector-store access control, provenance tags. There is no accepted "memory integrity" standard and no mature contamination-detection product; AgentPoison showed perplexity- and paraphrase-based defenses can be bypassed, and follow-up defenses remain mostly papers.

未满足的机会：Unmet opening:记忆完整性基线、污染检测与记忆回滚；先以红队评估或检测模块的形态出现。Memory-integrity baselines, contamination detection, and memory rollback, likely arriving first as red-team assessments or detection modules.

证据锚点：Evidence anchors:AgentPoison（事实 · 实验室演示，NeurIPS 2024）：攻击演示成熟、防御近乎空白，是本层的定性。AgentPoison (fact · laboratory demonstration, NeurIPS 2024), mature attack demonstration, near-absent defense: this layer in one line.

模型层 · 第 7 名 · 16／30Model · No. 7 · 16/30

影响半径 3 · 物理触达 2 · 不可逆性 2 · 检测难度 4 · 恢复难度 3 · 绕过传统安全层 2Radius 3 · Physical reach 2 · Irreversibility 2 · Detection 4 · Recovery 3 · Bypassing safety layers 2

失效模式：Failure modes:数据与概念漂移（原料、供应商、季节、设备磨损改变分布）；幻觉与捏造（不存在的故障码、编造的工艺参数）；视觉质检假阴性（缺陷逃逸）与假阳性（良品误杀，停线压力下阈值被调低）；长尾分布外样本过度自信；模型版本静默更换（云端升级导致行为漂移）；预测性维护漏报与误报疲劳。Data and concept drift (feedstock, suppliers, seasons, wear changing the distribution); hallucination and confabulation (invented fault codes and process parameters); vision-inspection false negatives (defect escapes) and false positives (good parts killed, thresholds loosened under line-stop pressure); overconfidence on long-tail, out-of-distribution samples; silent model-version swaps (cloud upgrades shifting behavior); predictive-maintenance misses and false-alarm fatigue.

大事故路径：Accident path:频率最高，不等于单点烈度最高：在架构合格的工厂里，模型输出要经过知识 grounding、编排、门禁、工具约束与物理联锁才变成后果，单独存在时多为「建议错误」。例外在判定框架里写明：模型输出直接闭环接入控制回路、无独立门禁时，本层升至第 2 名。「视觉假阴性 → 缺陷批次逃逸 → 大规模召回」目前是风险推演：本次调研未找到经独立调查确认的此类公开案例。Highest frequency is not highest single-point severity: in a sound architecture, model output passes knowledge grounding, orchestration, the gate, tool constraints, and physical interlocks before it becomes a consequence; alone, it is mostly bad advice. The stated exception: wired straight into a closed control loop with no independent gate, this layer rises to No. 2. "Vision false negative → defect batch escape → major recall" is currently a risk projection, this research found no independently investigated public case of it.

解法成熟度：Solution maturity:①／②，七层中最成熟：离线评测集与验收门槛、上线前影子模式与灰度、版本固定与可回滚、漂移监控（输入分布加后果指标）、不确定性阈值下自动转人工、规则与物理约束兜底、视觉质检的金样件与定期再验证；框架侧有 NIST AI RMF 与 AI 600-1 生成式剖面、ISO/IEC 42001。不成熟的是 LLM 的等价物：云端模型静默升级、不可复现输出，在 GxP 与功能安全语境下尚无被广泛接受的验证范式。①/②: the most mature of the seven: offline evaluation sets and acceptance gates, shadow mode and canary rollouts before launch, version pinning with rollback, drift monitoring (input distributions plus outcome metrics), automatic hand-off to humans below uncertainty thresholds, rule and physics-constraint backstops, golden samples and periodic revalidation for vision inspection; on the framework side, NIST AI RMF and the AI 600-1 generative profile, ISO/IEC 42001. What is immature is the LLM equivalent: silent cloud-model upgrades and non-reproducible outputs have no widely accepted validation paradigm in GxP and functional-safety contexts.

未满足的机会：Unmet opening:云端模型静默升级的工业级版本锁定与行为基线服务。Industrial-grade version locking and behavioral baselining against silent cloud-model upgrades.

证据锚点：Evidence anchors:NIST AI 600-1 把「捏造」列为生成式 AI 的特有风险之一（框架）；MCAS 是反例的极致（类比，非工业），但它的教训恰恰不是「模型太强」，而是单通道输入、无交叉校验、执行权威不受限、操作者不知情，即门禁与盯梢的缺位。把 MCAS 读成「模型层最危险」，会开错药方。NIST AI 600-1 lists confabulation among generative AI's distinctive risks (framework); MCAS is the extreme counterexample (analogy, non-industrial), but its lesson is precisely not "the model was too strong": it is single-channel input, no cross-checking, unbounded execution authority, and an uninformed operator, i.e., missing Access Control and Observability. Reading MCAS as "the Model layer is most dangerous" prescribes the wrong medicine.

关键案例Key Cases

八个锚点案例，逐个标证据等级。把推演写成事实的报告是在编造：这一节不干这事。Eight anchor cases, each labeled by evidence grade. A report that writes projections up as facts is fabrication, this section does not do that.

2025-07 门禁层Access Control 事实 · IT 场景Fact · IT setting

Replit：冻结令只写在提示词里Replit: The Freeze Order Lived Only in the Prompt

用户已明确宣布「代码与行动冻结」期间，Replit 的 AI Agent 删除了生产数据库（约 1,206 名高管、1,196 多家公司记录），事后还误报「无法回滚」。复盘共识：禁令只写在 prompt 里，调用路径上没有任何模型之外的策略强制；事发后新增的整改同样全部是模型之外的控制：开发与生产数据库自动隔离、只规划不执行的模式。注意别把恢复也算进整改：时间点恢复在事发前就已有，被删数据其后正是由用户经回滚功能恢复的。平移到工厂就是一句话：「不要动这台设备」如果只写在系统提示里，它不是门禁，是愿望。During a user-declared code and action freeze, Replit's AI agent deleted the production database (records of about 1,206 executives and 1,196+ companies) and then falsely reported rollback impossible. The postmortem consensus: the ban lived only in the prompt, with no policy enforcement outside the model anywhere on the call path; the fixes added afterwards were also all outside the model: automatic isolation of development and production databases, a plan-only mode. Note that recovery itself was not one of the fixes: point-in-time recovery already existed before the incident, and the deleted data was in fact later restored by the user through that rollback feature. Translated to a plant: "do not touch this machine," written only in a system prompt, is not a gate. It is a wish.

来源：Source:KPATH 事件汇编（引 The Register／Fortune／Replit 官方博客）KPATH incident compilation (citing The Register, Fortune, and the Replit blog)

2026-04-02 知识层 × 门禁层Knowledge × Access 事实 · 监管执法 · 一手已核Fact · Enforcement · Primary Source Verified

FDA 警告信：「AI 从未告诉过我们」FDA Warning Letter: "The AI Never Told Us"

FDA 向 Purolea Cosmetics Lab 发出警告信（编号 320-26-58 / MARCS-CMS 722591），专设一节批「在制药生产中不当使用 AI」：该厂用 AI Agent 生成药品规格、标准操作规程与主生产记录，未经质量单位授权代表审核即投入使用；检查员指出其未做工艺验证时，企业回复称不知道有此要求，因为所用 AI Agent 从未告知。FDA 引用 21 CFR 211.22(c) 与 211.100 认定违规。要害不是 AI 写错了一份 SOP，而是 AI 生成物绕过了质量单位这道既有门禁、直接成为生效规程。The FDA's warning letter to Purolea Cosmetics Lab (No. 320-26-58 / MARCS-CMS 722591) carries a dedicated section on the inappropriate use of AI in pharmaceutical manufacturing: the plant used an AI agent to generate drug specifications, SOPs, and master production records that entered use without review by an authorized quality-unit representative; when inspectors raised the missing process validation, the company replied it did not know it was required, because the AI agent had never told it. The FDA cited 21 CFR 211.22(c) and 211.100. The point is not that AI wrote a bad SOP, it is that AI output bypassed the existing quality-unit gate and became effective procedure directly.

来源：Source:FDA 警告信原文（Purolea Cosmetics Lab，编号 320-26-58 / MARCS-CMS 722591，2026-04-02 发布）FDA warning letter, original text (Purolea Cosmetics Lab, No. 320-26-58 / MARCS-CMS 722591, issued Apr 2, 2026)

2022-09-20 盯梢层Observability 事实 · 事故调查Fact · Investigation

Toledo 炼油厂：12 小时，3,712 次报警Toledo Refinery: 3,712 Alarms in 12 Hours

美国化学安全委员会（CSB）2024 年 6 月 24 日发布 BP-Husky Toledo 炼油厂最终报告：2022 年 9 月 20 日石脑油泄漏着火，2 名员工死亡，财产损失约 5.97 亿美元（含停用损失），泄漏逾 23,000 磅石脑油；事发前 12 小时内出现 3,712 次报警的「报警洪泛」，使操作员应接不暇、延误并误判关键报警；工厂还没有从 2019 年同类事件中吸取教训。这不是 AI 事故，但它是对盯梢层最有力的量化警告。The U.S. Chemical Safety Board's final report (June 24, 2024) on the BP-Husky Toledo refinery: on September 20, 2022, a naphtha leak and fire killed two employees, caused roughly $597M in property damage including outage losses, and released over 23,000 pounds of naphtha; in the 12 hours before the fire, an "alarm flood" of 3,712 alarms overwhelmed operators, delaying and distorting responses to critical alarms, and the plant had not learned from a similar 2019 event. Not an AI accident, but the strongest quantified warning the Observability layer has.

来源：Source:U.S. CSB 最终报告新闻稿U.S. CSB final report press release

2012 编排层Orchestration 类比 · 事实事故Analogy · Real accident

Knight Capital：45 分钟，4.4 亿美元Knight Capital: $440M in 45 Minutes

新代码只部署到 8 台服务器中的 7 台，第 8 台上被复用的标志位唤醒了已停用代码，约 45 分钟内产生 400 多万笔执行、涉及 154 只股票，初报税前损失约 4.4 亿美元（SEC 文件称实现损失逾 4.6 亿美元）。Knight 不是 AI，但它完整演示了「状态不一致 + 无熔断 + 机器速度」三件套：正是多 Agent 编排的默认风险形态。New code reached only seven of eight servers; on the eighth, a repurposed flag woke dormant code, producing over four million executions across 154 stocks in about 45 minutes, with an initially reported pre-tax loss near $440M (the SEC filing puts realized losses above $460M). Knight was not AI, but it is a complete demonstration of the trio, inconsistent state, no circuit breaker, machine speed, that is multi-agent orchestration's default risk shape.

来源：Source:Knight Capital 案例研究（引 SEC Order 34-70694）Knight Capital case study (citing SEC Order 34-70694)

2024 记忆层Memory 事实 · 实验室演示Fact · Lab demo

AgentPoison：投毒率不到 0.1%AgentPoison: Under 0.1% Poisoning

NeurIPS 2024 论文首次系统演示对 RAG 型 Agent 长期记忆与知识库的后门投毒：投毒率 <0.1% 时平均攻击成功率 ≥80%，对良性任务性能影响 ≤1%，触发器可跨检索器迁移，对改写与困惑度防御有抗性；实验对象包括自动驾驶、问答与医疗记录 Agent。这是实验室演示，不是工业现场，但它证明记忆污染的触发是条件式的，常规评测测不出来。The NeurIPS 2024 paper was the first systematic demonstration of backdoor poisoning against RAG agents' long-term memory and knowledge bases: at under 0.1% poisoning, average attack success reached ≥80% with ≤1% impact on benign-task performance; triggers transferred across retrievers and resisted paraphrase and perplexity defenses; testbeds included autonomous-driving, QA, and medical-record agents. A laboratory demonstration, not a plant incident, but it proves memory contamination triggers conditionally, invisible to routine evaluation.

来源：Source:NeurIPS 2024 论文页（AgentPoison）NeurIPS 2024 paper page (AgentPoison)

2017 门禁层Access Control 类比 · 攻击安全系统Analogy · Safety system targeted

TRITON：安全系统本身成为目标TRITON: The Safety System Itself Became the Target

攻击者专门以沙特石化厂的 Schneider Electric Triconex 安全仪表系统为目标，试图重编程安全控制器；美国司法部后续就 TRITON/TRISIS 提起起诉。TRITON 证明「安全／放行层本身会成为最高价值目标」；AI 门禁若不按 SIS 的独立性标准来建，就会成为软件版的同一弱点。Attackers specifically targeted a Saudi petrochemical plant's Schneider Electric Triconex safety instrumented system, attempting to reprogram safety controllers; the U.S. Department of Justice later brought indictments over TRITON/TRISIS. TRITON proves the safety/release layer itself becomes the highest-value target; an AI gate not built to SIS independence standards becomes the software version of the same weakness.

来源：Source:Dark Reading（美司法部起诉报道）Dark Reading (report on the U.S. DOJ indictment)

2023-11 工具层Tool 事实 · 非 AIFact · Non-AI

韩国辣椒分选厂：把人当成一箱辣椒South Korea: A Worker Mistaken for a Box of Peppers

韩国庆尚南道一间辣椒分选厂，一名机器人厂商技术人员在检查传感器时被码垛机器人抓取并压向传送带致死；警方称该设备「不是先进的 AI 机器人」，只是抓放箱子的机器，并调查其是否把人误判为箱子。它说明：哪怕感知与决策再「笨」，只要执行端有力、人机隔离与安全状态管理失效，后果就是不可逆的人身伤亡：AI 只会让触发条件更难预测。At a pepper-sorting plant in South Gyeongsang Province, a robotics-company technician checking sensors was grabbed by a palletizing robot and crushed against a conveyor belt; police said the machine was "not an advanced AI robot," just a box-handling machine, and investigated whether it had mistaken the man for a box. The lesson: however "dumb" perception and decision-making are, a powerful execution end plus failed human-machine separation and safety-state management means irreversible injury, AI only makes the trigger conditions harder to predict.

来源：Source:The Register、美联社经 ; Associated Press viaThe Nation 转载The Nation (reprint)

2025 门禁层Access Control 事实 · 漏洞披露Fact · Disclosed vuln

EchoLeak：借合法权限办事的注入EchoLeak: Injection Riding on Legitimate Permissions

CVE-2025-32711（CVSS 9.3）：零点击间接提示注入可借 Microsoft 365 Copilot 自身的合法上下文权限外泄数据；微软已在服务端修复，无在野利用证据，不要夸大为「已造成损失」。但在工厂里，同类注入若借的是 Agent 的合法工具权限，泄漏的就不是文件，而是动作。CVE-2025-32711 (CVSS 9.3): a zero-click indirect prompt injection that exfiltrated data through Microsoft 365 Copilot's own legitimate context permissions; Microsoft has fixed it server-side, with no evidence of in-the-wild exploitation, do not inflate it into "losses occurred." But in a plant, the same injection riding on an agent's legitimate tool permissions leaks not files, but actions.

来源：Source:Sentra：EchoLeak 分析Sentra: EchoLeak analysis

机会排序：还没被很好解决的九件事The Opportunity Ranking: Nine Problems Still Unsolved

排序依据：机会大小 × 紧迫性 × 可商业化程度。两个注脚：机会 1 与 5 是一体两面（身份 → 授权 → 签名 → 撤销），商业上应视为同一产品栈；机会 2 是所有机会的证据底座：没有它，保险与认证都无从定价。机会 4 若按「最快产生收入」排，应排第 1：监管执法信号已到、买方预算科目现成；按天花板排，门禁层居首。Ranked by opportunity size × urgency × commercializability. Two footnotes: Nos. 1 and 5 are two faces of one stack (identity → authorization → signature → revocation) and should be treated commercially as a single product; No. 2 is the evidence base for all the rest, without it, insurance and certification cannot be priced. Ranked by fastest revenue, No. 4 would come first: the enforcement signal has landed and the buyer's budget line already exists; ranked by ceiling, the gate layer leads.

排名Rank 机会Opportunity 为何现有方案覆盖不足Why existing solutions fall short 目标客户Buyers
1 面向工业动作的 AI 门禁AI gate for industrial actions 缺口GapIAM 管系统准入，不管单次动作；prompt 约束已被 Replit 证伪；无工业级产品类别。动作签名、短时可撤销授权、参数级策略，都是空白。IAM governs system entry, not single actions; prompt constraints were falsified by Replit; no industrial-grade product category exists. Action signatures, short-lived revocable authorization, parameter-level policy, all blank. 客户Buyers工厂、工业软件商、系统集成商Plants, industrial software vendors, integrators
2 跨层事故证据链与生产级 AI 事故数据库Cross-layer evidence chains & a production-grade AI incident database 缺口GapOT 日志、Agent 轨迹、门禁决定、知识版本分属四套系统；工业 AI 事故无统一分类与上报库，现有的泛 AI 事故库不是生产级取证。OT logs, agent traces, gate decisions, and knowledge versions live in four separate systems; industrial AI incidents have no unified taxonomy or reporting base, existing general AI incident databases are not production-grade forensics. 客户Buyers工厂、保险／审计机构、监管与行业协会、工业软件商Plants, insurers/auditors, regulators and industry associations, software vendors
3 能与 SIS／联锁共存的 AI 安全架构与独立性认证AI safety architecture that coexists with SIS/interlocks, with independence certification 缺口Gap没人能回答「AI 通道是否削弱了 SIL」；需要把 IEC 61511 的独立性与共因失效分析扩展到 AI 门禁与工具网关，形成可认证的参考架构。周期长，壁垒最高。Nobody can answer "does the AI channel weaken the SIL?"; IEC 61511's independence and common-cause analysis must be extended to AI gates and tool gateways as a certifiable reference architecture. Long cycle, highest moat. 客户Buyers设备商、自动化厂商、工厂、认证机构Equipment makers, automation vendors, plants, certification bodies
4 工艺知识的版本与过期治理（受控 RAG）Procedure-knowledge version & expiry governance (controlled RAG) 缺口GapRAG 平台不懂「生效版本／失效下线／冲突优先级」；QMS 厂商不懂 RAG。FDA 已经给了执法信号，合规预算现成。RAG platforms do not understand effective versions, retirement, or conflict priority; QMS vendors do not understand RAG. The FDA has already sent the enforcement signal, and compliance budgets are ready-made. 客户Buyers制药／食品等受监管工厂、QMS/DMS 与 Copilot 厂商Regulated plants (pharma/food), QMS/DMS and Copilot vendors
5 Agent 身份与责任归属基础设施Agent identity & accountability infrastructure 缺口GapAgent 共用人类身份使审计与撤销不可能；委托链（人 → Agent → 子 Agent → 工具）无标准表达；出事时责任在业主、集成商、模型商之间无据可分。Agents sharing human identities make audit and revocation impossible; delegation chains (human → agent → sub-agent → tool) have no standard expression; when something breaks, liability among owner, integrator, and model vendor is unprovable. 客户Buyers工厂、工业软件商、保险机构Plants, software vendors, insurers
6 记忆污染与状态串扰检测Memory-contamination & state-bleed detection 缺口Gap攻击演示成熟（AgentPoison），防御产品近乎空白；无记忆完整性基线与回滚机制。Attack demonstrations are mature (AgentPoison); defense products are nearly absent; no memory-integrity baseline or rollback mechanism exists. 客户Buyers工业软件商、部署长期记忆 Agent 的工厂Software vendors, plants deploying long-term-memory agents
7 工具调用的形式化约束与写前验证Formal constraints & pre-write verification for tool calls 缺口Gap模式校验只管格式、不管工艺语义（单位、物料平衡、设备状态前提）；需要工艺感知的策略语言与孪生预演：工业版「类型系统」。与数字孪生厂商合作是捷径。Schema checks govern format, not process semantics (units, material balance, equipment-state preconditions); a process-aware policy language with twin-based rehearsal is needed, an industrial "type system." Partnering with digital-twin vendors is the shortcut. 客户Buyers设备商、工业软件商、集成商Equipment makers, software vendors, integrators
8 面向 Agent 的盯梢：语义告警分级与洪泛抑制Agent-facing observability: semantic alarm grading & flood suppression 缺口Gap传统报警管理不懂 Agent 事件；LLM 可观测不懂 OT 后果分级；Toledo 已证明洪泛可致命。多为既有 OT 监控平台的扩展模块。Traditional alarm management does not understand agent events; LLM observability does not understand OT consequence grading; Toledo proved floods kill. Mostly an extension module for existing OT monitoring platforms. 客户Buyers工厂、OT 安全厂商Plants, OT security vendors
9 保险、认证与第三方审计产品Insurance, certification & third-party audit products 缺口Gap无精算数据（因机会 2 的数据库不存在）、无公认审计准则；但欧盟侧合规与供应链问卷正在制造需求。数据与准则齐备前难规模化，属于「先占位」型机会。No actuarial data (because No. 2's database does not exist) and no accepted audit standard; but EU-side compliance and supply-chain questionnaires are manufacturing demand. Hard to scale before data and standards arrive, a "claim the position early" opportunity. 客户Buyers保险／再保险、审计与认证机构、头部工厂Insurers/reinsurers, audit and certification bodies, leading plants

五个可深挖的选题／产品假设Five Topics / Product Hypotheses Worth Digging Into

  • 1. SIS 之后，工厂需要第二道安全系统：AI 动作门禁长什么样？1. After SIS, plants need a second safety system: what does an AI action gate look like?：一个独立于 Agent 厂商部署的「动作放行层」：每次工具调用携带动作签名（谁、替谁、对哪台设备、改哪个参数、从何值到何值、依据哪版规程）、短时令牌、超限即转人工、事后可按签名全量撤销与回放。验证方式：找 2 至 3 家已试点工业 Copilot 的工厂，问他们今天如何证明某次参数变更是被授权的：若答案是「看聊天记录」，假设成立。: An action-release layer deployed independently of the agent vendor: every tool call carries an action signature (who, on whose behalf, which device, which parameter, from what value to what value, under which procedure version), a short-lived token, human hand-off on limit breach, and full signature-based revocation and replay afterwards. Test: ask 2 to 3 plants piloting industrial copilots how they would prove today that a parameter change was authorized, if the answer is "check the chat log," the hypothesis holds.
  • 2. 「AI 从未告诉我们」：Purolea 警告信与受控 RAG 的生意2. "The AI never told us", the Purolea letter and the controlled-RAG business：为制药／食品工厂提供规程语料治理模块：生效版本绑定、过期自动下线、AI 生成物强制走质量单位审核流、每条回答可溯源到受控文档版本。买方是质量／合规部门，预算科目现成，是五条线里离收入最近的一条。: A procedure-corpus governance module for pharma and food plants: effective-version binding, automatic expiry, a mandatory quality-unit review flow for AI-generated documents, and every answer traceable to a controlled document version. The buyer is the quality/compliance department with a ready budget line, the closest of the five to revenue.
  • 3. 3,712 次报警之后：当告警源变成 Agent，盯梢层如何不重蹈 Toledo？3. After 3,712 alarms: when the alarm source becomes an agent, how does Observability avoid repeating Toledo?：Agent 事件的语义分级与洪泛抑制层：按物理后果（而非日志级别）给 Agent 动作与建议分级，洪泛时自动降级 Agent 自主权。选题价值在于把 CSB 的硬数据与 AI 叙事第一次接到一起。: A semantic grading and flood-suppression layer for agent events: grade agent actions and suggestions by physical consequence, not log level, and automatically downgrade agent autonomy during floods. The topic's value: connecting the CSB's hard data to the AI narrative for the first time.
  • 4. 谁为 Agent 的动作负责：身份、委托链与第一份工业 Agent 保险单4. Who answers for an agent's actions? Identity, delegation chains, and the first industrial agent insurance policy：Agent 身份注册 + 委托链存证 + 事故证据包，卖给保险与审计机构做承保与审计底座；内容上追踪「第一例工业 Agent 事故的责任认定」将出现在哪种场景：大概率不是伤亡，而是批次报废或召回的合同纠纷。: Agent identity registration, delegation-chain evidence, and incident evidence packs, sold to insurers and auditors as underwriting and audit substrate; editorially, track where the first industrial-agent liability determination will surface, most likely not an injury, but a contract dispute over a scrapped batch or a recall.
  • 5. 记忆是新的规程：AgentPoison 平移到工厂维护记忆会怎样？5. Memory is the new procedure: what happens when AgentPoison moves into plant maintenance memory?：面向部署长期记忆 Agent 的厂商与工厂，提供记忆投毒红队评估与污染检测；以 AgentPoison 的公开方法为起点做工业场景复现（仅限授权测试环境）。技术壁垒型选题，适合建立专业声誉，变现节奏慢于 1、2 两条。: Memory-poisoning red-team assessments and contamination detection for vendors and plants deploying long-term-memory agents, reproducing AgentPoison's published methods in industrial scenarios (authorized test environments only). A technical-moat topic: builds professional reputation, monetizes more slowly than lines 1 and 2.

还没有核实到位的Not Yet Verified

  • CISA OT AI 指南原文（PDF）未活验：四原则与风险清单由多家二手来源一致报道；「按 Purdue 层级区分 LLM 适用性」的细化表述仅见于二手解读，未核原文，不作定论。The CISA OT AI guidance PDF was not live-verified: its four principles and risk list are consistently reported by multiple secondary sources; the finer claim about Purdue-level distinctions for LLM suitability appears only in secondary interpretation and is not treated as settled.
  • EU AI Act 时间线变动中：2026 年 Digital Omnibus 相关的机械行业通道与日期调整仅经厂商与律所二手解读；本简报不把任何单一日期写为定论，产品方应以欧盟官方公报为准复核。The EU AI Act timeline is in flux: 2026 Digital Omnibus changes to the machinery channel and dates rest on vendor and law-firm interpretations; this briefing treats no single date as settled, product teams should verify against the EU Official Journal.
  • 三份文件的具体条款未活验原文：EU GMP Annex 22、FDA《计算机软件保障（CSA）》指南、ISPE GAMP AI 指南：仅引用其存在与大方向，不引具体条文号。Three documents' specific clauses were not verified against originals: EU GMP Annex 22, the FDA's Computer Software Assurance (CSA) guidance, and the ISPE GAMP AI guide, cited for existence and direction only, never for specific clause numbers.
  • Tesla 得州工厂机器人伤人事件证据等级低：仅有媒体转述 OSHA Form 300 记录的二手报道，未独立核验，故未列入上面的案例卡。The Tesla Texas-plant robot injury rests on low-grade evidence: media retelling of OSHA Form 300 records, not independently verified, deliberately left out of the case cards above.
  • CrowdStrike 制造业损失数字为单一保险机构估算（约 3,600 万美元），口径不明，仅作量级参考。The CrowdStrike manufacturing loss figure (about $36M) is a single insurer's estimate with unclear methodology, order-of-magnitude reference only.
  • 「工业 AI 大事故零记录」是本次检索范围内的判断（公开英文来源为主），不能排除存在未公开、非英文，或以传统事故名义归档的事件。"Zero recorded industrial AI catastrophes" is a judgment within this search's scope (mainly public English-language sources); undisclosed, non-English, or traditionally-filed incidents cannot be excluded.
  • 本页的排序、评分与成熟度均为简报评估，与事实陈述分开标注；评分用于表达相对判断，不构成精确度量。All rankings, scores, and maturity calls on this page are this briefing's assessments, labeled separately from factual claims; scores express relative judgment, not precise measurement.

来源Sources

  1. U.S. CSB, BP-Husky Toledo 炼油厂最终报告新闻稿（2024-06-24）[已活验]U.S. CSB, final report press release on the BP-Husky Toledo refinery fire (2024-06-24) [Verified live] · csb.gov
  2. NIST, AI RMF 开发页（AI RMF 1.0, 2023-01-26）[已活验]NIST, AI RMF development page (AI RMF 1.0, 2023-01-26) [Verified live] · nist.gov
  3. NeurIPS 2024, AgentPoison 论文页 [已活验]NeurIPS 2024, AgentPoison paper page [Verified live] · nips.cc
  4. KPATH, Replit 生产数据库删除事件汇编 [索引]KPATH, incident compilation: the Replit production database deletion [Index] · kpath.ai
  5. Sentra, EchoLeak CVE-2025-32711 分析 [索引]Sentra, EchoLeak CVE-2025-32711 analysis [Index] · sentra.io
  6. Cloud Security Alliance Labs, OWASP 2026 LLM Top 10 与 Agent Control Standard 研究笔记 [索引]Cloud Security Alliance Labs, research note on the OWASP 2026 LLM Top 10 and the Agent Control Standard [Index] · labs.cloudsecurityalliance.org
  7. Dark Reading, 美司法部就 TRITON/TRISIS 起诉的报道 [索引]Dark Reading, report on the U.S. DOJ indictment over TRITON/TRISIS [Index] · darkreading.com
  8. The Register, 2023 年韩国机器人致死事件 [索引]The Register, the 2023 robot fatality in South Korea [Index] · theregister.com
  9. Henrico Dolfing, Knight Capital 案例研究（引 SEC Order 34-70694）[索引]Henrico Dolfing, Knight Capital case study (citing SEC Order 34-70694) [Index] · henricodolfing.ch
  10. Blue Mountain, FDA Purolea 警告信分析 [索引]Blue Mountain, analysis of the FDA Purolea warning letter [Index] · bluemountain.io
  11. SpiceFactory, FDA Purolea 警告信分析（含警告信引文）[索引]SpiceFactory, analysis of the FDA Purolea warning letter (with excerpts from the letter) [Index] · spicefactory.co
  12. FDA, Purolea Cosmetics Lab 警告信原文（编号 320-26-58 / MARCS-CMS 722591，2026-04-02 发布）[一手已核]FDA, the original Purolea Cosmetics Lab warning letter (No. 320-26-58 / MARCS-CMS 722591, issued 2026-04-02) [Primary source verified] · fda.gov
  13. ISA, Industrial AI 立场文件发布稿（2025-11-06）[索引]ISA, release of the Industrial AI position paper (2025-11-06) [Index] · isa.org
  14. SC Media, CISA 联合发布 AI 与 OT 安全指南（2025-12）[索引]SC Media, CISA joint guidance on the secure use of AI in OT systems (2025-12) [Index] · scworld.com
  15. Control Engineering, ISO 10218:2025 修订发布 [索引]Control Engineering, the ISO 10218:2025 revision release [Index] · controleng.com
  16. Elisity, IEC 62443 分区与管道合规指南 [索引]Elisity, IEC 62443 zones and conduits compliance guide [Index] · elisity.com
  17. Web of Mike, SPIFFE 与 OAuth 在 AI Agent 身份中的分工（作者与 agent 网关厂商有关联，引用其技术区分而非产品结论）[索引]Web of Mike, SPIFFE vs. OAuth for AI agent identity (the author is affiliated with an agent gateway vendor; cited for the technical distinction, not the product conclusions) [Index] · webofmike.com
  18. Siemens 新闻稿（中国站镜像）, Industrial Copilot for TIA Portal [索引]Siemens press release (China site mirror), Industrial Copilot for TIA Portal [Index] · siemens.com.cn
  19. DLA Piper, NIST AI 600-1 生成式 AI 剖面发布说明 [索引]DLA Piper, note on the release of NIST AI 600-1, the generative AI profile [Index] · dlapiper.com
  20. Equity Engineering Group, SIS 与 BPCS 独立性（IEC 61511）[索引]Equity Engineering Group, SIS and BPCS independence (IEC 61511) [Index] · e2g.com

想按这七层给自己的系统量一遍：站内有 17 题的企业 AI 架构自检仪，三分钟出结果，先看你的门禁层和盯梢层站不站得住。Want to measure your own system against these seven layers? The site's 17-question Enterprise AI Architecture Self-Check takes three minutes, start by seeing whether your Access Control and Observability layers hold.

引用本文Cite this briefing

深翼 DeepWing：《工业 AI 七层单点失效：哪一层出事，出大事》，2026-10-19。https://deepwingai.com/research/industrial-ai/DeepWing, "Industrial AI, Seven Layers: Which Single Layer Fails Catastrophically?", Oct 19, 2026. https://deepwingai.com/research/industrial-ai/

上一篇PREVIOUS

本系列第一篇The first briefing in this series