Integration rigs in aerospace hook up all the hardware, software, and emulated conditions essentially testing before flight test. I’d run the test, document the result, close the action. The system meets the test case, the test case meets the requirement. Pass. Move on. If it failed, write it up and ship it back to design.

Years later I moved into cockpit design. Same company, different role. And those same test reports, some of them authored by me, started landing on my desk.

That’s when my perspective started to shift. A test pilot I worked with helped too, if you’ve read my previous post.

The reports were sound. The tests had been run properly. The system had met the requirement. But sitting on the other side of that process, looking at what we were actually trying to build and how a pilot would use it, I could see the gap. I had been trying to fix to meet the requirement as I understood it as a test engineer without context of the wider system, the operator, the HMI. I had spent years doing verification without fully understanding validation.

It’s worth pausing on that distinction, because it’s the thread that runs through everything that follows.

Verification asks: does the system do what we specified? Validation asks: did we specify the right thing?

With requirements and verification at the forefront, validation can take a back seat. That’s where the gap opens.

𝗢𝗽𝗲𝗿𝗮𝘁𝗼𝗿𝘀 𝗱𝗼𝗻’𝘁 𝗿𝗲𝗮𝗱 𝘁𝗵𝗲 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻

This became especially clear when training members of the armed forces on a new piece of kit. Within a week they had surfaced issues that six months of in-house testing hadn’t considered.

Were they being rough with it? Yes. But they saw it as a tool. We had been treating it as scientific equipment. We were incentivised to protect the kit, to keep testing, to prevent cost. They only cared whether it did the job under pressure. Both perspectives are valid. Only one was in the test programme.

This is the human dimension of the validation gap that rarely gets written about. Engineers are protective of the systems they build. Their incentives are different during development. But that produces test assumptions that reflect design intent, not operational use. The operator’s view, that the kit is a means to an end, used hard in conditions nobody anticipated, has to be deliberately built into the programme. It doesn’t have to be a test requirement. Analysis can capture it too. But it has to be captured somewhere.

The Pentagon’s DOT&E report makes this finding year after year. Systems pass developmental test and fall short under operational conditions. The labels change, procurement problem, acquisition problem, requirements problem. But the gap underneath is consistent: the people who designed the test thought about the system differently to the people who had to use it. That is a systems issue, not an organisational one.

𝗜𝘁’𝘀 𝗮𝗹𝘀𝗼 𝘄𝗼𝗿𝘁𝗵 𝗰𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗶𝗻𝗴 𝗵𝗼𝘄 𝘃𝗲𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀 𝗰𝗼𝗺𝗽𝗼𝘂𝗻𝗱 𝘁𝗵𝗲 𝗴𝗮𝗽

Some standards are genuinely loose on what success looks like. Salt fog testing under DO-160 is a good example, there is real ambiguity about whether the test environment represents what an aircraft actually sees in service. The pass criteria exist. Whether the test maps to operational reality is a different question and it doesn’t always get asked.

The same looseness shows up in how standards get interpreted. FAA, stakeholders, suppliers interpret the same requirement through a different lens, producing different conclusions about what passing means. We always reached agreement. But agreeing on an interpretation is not the same as validating it against operational reality. The argument gets resolved. The underlying question often doesn’t.

𝗧𝗵𝗲 𝗰𝗼𝘀𝘁 𝗼𝗳 𝗳𝗶𝗻𝗱𝗶𝗻𝗴 𝗼𝘂𝘁 𝗹𝗮𝘁𝗲

My test team turned to me, on a miserable grey day on the water: “Do we have to run it again? We all know it’s fixed.”

Yes. We did. Because there was no other opportunity where all the kit was connected together, the fix had to be verified on trials. On the water. Burning day rate, vessel time, and weather window.

The F-35 programme is another well-documented example of what happens when integration environments are late or insufficiently representative. The capability existed. But not early enough, or at sufficient fidelity, to de-risk the system before the costs of late discovery compounded.

Early integration environments, System Integration Rigs representative test benches, exist to prevent exactly this. Not just cheaper testing, but earlier testing. Built up incrementally as development progresses, so issues get found when they’re cheap to fix rather than when the only option is the field.

The second loss is less obvious but just as real. Without early integration environments you don’t just find things late, you burn the field time that was supposed to give you something else. Trials are expensive, weather-dependent, and limited. Every cycle spent on a known issue is a cycle you can’t spend on the uncontrolled variable exposure that open water testing is actually for.

What you bring into expensive or unrepeatable test environments should be limited to the things that can only be tested there.

𝗧𝗵𝗲 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝘀𝗵𝗶𝗳𝘁

Over my career in aerospace, maritime and defence, I’ve seen this pattern repeat.

The validation gap persists because the conditions that create it are normal. Schedules reward delivery to specification. Budgets cut integration environments early. Operators arrive at acceptance, not at requirements.

That isn’t negligence. It’s the default.

Closing the gap requires different decisions upstream, such as how systems are defined, who is in the room when requirements are written, and which environments get funded before the field becomes the only option.

If those decisions aren’t made deliberately, operational reality makes them for you.

Usually at the worst possible time.

Not to slow things down. To make the system repeatable and auditable.

Thanks for reading.

I’m Stephen Cooke, a Principal Consultant who enjoys exploring the intersection of systems engineering, AI and autonomous systems.

These are my notes on building, testing and learning - from aerospace and maritime autonomy to manufacturing, product development and emerging technology.

My mind went spiralling. Loss of control. Collision. System failure in open water. The list got long quickly.

I took it to my colleague. Not for the first time that week, he reminded me:

“This isn’t aerospace, Steve.”

And to be fair, he was right. A ship that fails usually still floats. The crew goes home. I’d defaulted to aerospace, and the pattern didn’t fit.

A stark reminder of the difference between two engineering cultures occupying the same deck. It would not be the last.

A generation ago, aerospace went through its own version of this. The industry was moving toward genuinely software-driven flight control and working out, at scale, what it meant to verify software you cannot physically inspect.

There is a famous photo of Margaret Hamilton standing next to a stack of printed code taller than she is. It gets shared as a symbol of how primitive early software engineering was. That misses the point entirely.

Hamilton’s team did not just write the code and hope. They built in fault detection, priority scheduling, and the ability for the system to recognise when it was overloaded and shed lower-priority tasks to protect the mission-critical ones. When the Apollo 11 guidance computer began throwing alarms three minutes from the lunar surface, it was this error-recovery logic that kept the landing from failure. The standards that came later formalised thinking people like her had figured out the hard way.

That’s the part that tends not to get mentioned.

The lesson from aerospace was not “add more process.” It was “the thinking has to happen before the mission runs, not during it.”

The answer came in the form of standards we still use today. DO-178 for software. ARP4754 for systems. DO-254 for hardware. A whole architecture built around one principle: safety-critical systems must be designed with verification in mind long before the mission runs, not during it.

Getting there was not straightforward. The urgency came as much from incident as from foresight. The frameworks matured because the consequences of not having them became undeniable.

Now a similar shift is happening again. Autonomous vessels moving from research to deployment. And the tools used to build them changing just as fast. Two revolutions at once.

Uncrewed surface vessels are being trialled in defence. Underwater systems are going places humans cannot or should not go. And the software running on all of it is increasingly being written with AI assistance, compressing timelines in ways the industry has not seen before.

Both shifts are happening in an environment that is disorganised at the best of times: open water, pleasure craft, fishing boats and their gear, environmentally protected sites, sea states that vary in ways a flight corridor simply does not.

In both domains, maritime autonomy and AI-driven development, the frameworks are still being written.

That is not a criticism. It is the reality of working at the edge of what standards have caught up with.

But it matters, because the people running programmes right now are making judgement calls without a settled reference point to stand on.

The standards gap is real, and bigger than it looks

There is no DO-178C for autonomous maritime software. No ARP4754 written specifically for an uncrewed vessel.

The IMO is developing a Maritime Autonomous Surface Ships (MASS) code. It is expected to be voluntary from 2025, but not mandatory until 2032. DNV launched its AROS class notation framework at the start of 2025. The SMI Maritime Autonomous Systems Regulatory Working Group is producing guidance that programmes can actually use.

But the programmes are running now. The decisions about what level of rigour is required are being made now, without a settled answer to point to.

And this is where it matters: the gap is not simply a problem waiting to be solved.

Engineering judgement is an asset. Proportional rigour, matching verification depth to the real cost of failure, is legitimate practice. The aerospace approach does not translate wholesale.

Overly rigid requirements applied to systems with recoverable failure modes carry their own risk. They slow programmes, add cost without adding safety, and can push decisions out of sight.

The gap between capability and governance is not a reason to panic. It is a reason to think carefully.

The complicating factor is that without a shared reference point, rigour is often set implicitly. Sometimes that is appropriate. Sometimes it means the level of rigour is set by whoever is most confident in the room.

Two cultures, no right answer

When I was testing autonomous vessels, the one thing I noticed, watching myself as much as the sector, was how differently people framed risk.

Maritime engineers came from a world where ships fail, crews adapt, and you sort it out when you get back to port. That is not complacency. A ship that fails is usually recoverable. It generally still floats. The crew is relatively safe. That risk calculus is real.

Engineers, like me, from aerospace carry a different bias. We worried about different things. We asked for evidence before pressing go in ways that sometimes felt disproportionate.

I found myself caught between these two paradigms.

It is hard to argue against more rigour when the logic is sound. It is equally hard to argue against less when a programme is stalling over requirements that are not obviously reducing risk.

Without a guardrail, I found myself seesawing.

When two cultures stand in the same place with different assumptions, you end up navigating by feel more than you would like.

The maritime instinct is not wrong. But defence adds another layer to this complex equation.

A vessel that loses control at the wrong moment does not just fail operationally. It can become a liability. In a contested environment it can compromise a mission, reveal a position, cause casualties.

Underwater systems are a different problem again. When they fail, you often cannot see them, cannot recover them, and have lost the asset entirely.

The “it’s not safety critical like an aircraft” framing starts to break down when you assume open-ended risk. So it has to be contained, grounded in an agreed process.

And then there is the civilian environment where we test. Open water means sharing space with other vessels, infrastructure, and conditions that change faster than a test plan can account for.

What aerospace learned

DO-178 was not built to make engineers’ lives harder.

It was built to give the industry a shared understanding of the rigour required. A way to turn informal arguments on every programme into something consistent, auditable, and not dependent on local judgement alone.

That same argument is happening in maritime autonomy right now. It is just happening informally.

On docks, in programme reviews, in debrief rooms where people with different backgrounds quietly disagree about what they just saw, without a standard to point to.

The frameworks being built now matter because they give that argument somewhere to land.

Not a copy-paste of aerospace onto ships. The environments are different. The failure modes are different. The consequences vary by platform and mission.

But the underlying habits translate:

· Requirements traceability.

· Acceptance criteria defined before testing.

· Verification that goes beyond “it ran.”

Without those guardrails, the level of rigour on any given programme is set by internal consensus.

Companies are not trying to cut corners. But without a shared reference point, they default to what makes sense locally.

In some contexts, that is fine. In others, it is the setup for an incident that makes the urgency undeniable.

Where this is heading

Maritime autonomy in 2026 looks like technology running ahead of governance.

The instincts of the people building these systems vary depending on where they came from. The consequences of getting it wrong are not yet fully visible.

Aerospace learned its lessons through framework-building and incidents that made the urgency impossible to ignore.

The path does not have to be a replica. It will not be DO-178 at sea. It will be something leaner, more adaptive, and suited to an environment where failure is often recoverable.

But a framework is needed to chart this course.

Not to slow things down. To make the system repeatable and auditable.

Thanks for reading.

I’m Stephen Cooke, a Principal Consultant who enjoys exploring the intersection of systems engineering, AI and autonomous systems.

These are my notes on building, testing and learning—from aerospace and maritime autonomy to manufacturing, product development and emerging technology.

Subscribe now