Built on Broken Assumptions: Why Goldman Sachs Pulled the Plug on 'Vibe Coding' and Why the Glass Box Wins
Image created by Richard Aukland courtesy of aukinfo.com
In the span of a single week this November, the "vibe coding" and autonomous Black Box agent narrative took three direct hits from Wall Street, enterprise analysts, and AI researchers.
For the past two years, the industry has been sold a intoxicating fantasy: hand everyone an autonomous coding agent, step out of the loop, and watch software build itself.
Now, the bill is coming due. When you look closely at the data emerging this month, a single truth becomes undeniable: autonomous Black Box workflows do not eliminate engineering bottlenecks—they multiply them.
Here is what just happened across four real-world signals, the hidden details most people missed in the reports, and why Glass Box engineering is the only viable path forward for serious teams.
1. Goldman Sachs Ends the "Vibe Coding" Experiment
On November 7, reports surfaced via Bloomberg and The Times of India that Goldman Sachs abruptly revoked access to Claude Code and GitHub Copilot for thousands of bankers, traders, and non-technical staff. Moving forward, only the bank's 12,000 engineers and quantitative researchers ("strats") are allowed to touch the tools.
What makes this reversal so striking is how fast the dream collapsed. Just four months ago, in July, Goldman’s CIO Marco Argenti was publicly cheering on "vibe coding"—envisioning a workflow where traders would whip up their own AI-generated tools and toss them over the fence for engineering to productionize.
What actually went wrong?
The official statement cited "productivity and risk management," noting that the tools now belong "in the hands of those who are best suited to leverage them."
If you read between the lines of the report, two massive operational failures forced Goldman's hand:
- The Shadow IT & Regulatory Nightmare: According to insiders cited in the report, traders weren't just building harmless internal dashboards—they were using vibe-coded scripts to track client trading flows and model market moves and positions. In a heavily regulated financial environment, having non-engineers execute market decisions based on unverified, hallucinated code is a compliance catastrophe waiting to happen.
- The Endless Engineering Backlog: When thousands of traders can generate quick, insecure prototypes by the thousands, who has to audit, refactor, secure, and maintain them? The 12,000 engineers who already have their own roadmap to deliver!
Vibe coding does not work in a regulated environment. Letting non-engineers generate opaque prototypes doesn't save engineering time; it buries your senior engineers under an avalanche of unverified technical debt.
2. The 23% Trap: Why 40% of Agentic AI Projects Will Be Scrapped
The Goldman reversal isn't an isolated Wall Street quirk; it is a symptom of a massive enterprise ROI gap.
While Agentic AI investment surged 31.8% this year, only about 11% of organizations report measurable business value from it. Why? Because Black Box tools optimize the wrong part of the software lifecycle.
An analysis by Cognition across five months of production data from Global 1000 engineering teams running cloud agents revealed a sobering breakdown of where productive engineering hours actually go:
- 23% of engineering time is spent writing code.
- 77% goes to planning, architecture, testing, security, and deployment.
When you deploy a Black Box coding agent, you speed up less than a quarter of the engineering process while creating a massive code-review and verification backlog that actively slows down the remaining 77%.
The Hidden Cultural Debt
In a November 6 Forbes analysis on leading hybrid Human-Plus-AI teams, executive advisor Christine Grimm pointed to a string of alarming 2026 benchmarks that explain why Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027 over rising costs, unclear value, and weak risk controls:
- 56% of workers admit they have already made mistakes at work because of AI (KPMG survey).
- 51% of frontline AI initiatives have stalled out, with only a third of employees properly trained to supervise them (BCG).
- Tech-first organizations are 1.6x more likely to miss out on AI returns than human-centric ones (Deloitte 2026 survey of 9,000+ leaders).
Grimm argues that leaders must evolve from betting on raw efficiency to building a culture of earned trust—stating that a "leader must make AI use visible and transparent and its output checkable so speed never outruns integrity."
Before any agent goes live, she challenges leaders and their teams to answer sharp governance questions, including:
- "Which decision should never be made without a person, and have we captured this as policy?"
- "When an AI agent gets it wrong, whose name is on it? What is our process for holding humans accountable for agentic inputs and the work of hybrid teams?"
- "What are we asking our people to become as AI agents take on new tasks?"
3. The BOBA Benchmark: "Built on Broken Assumptions"
If anyone still believes we can solve the human review bottleneck by simply letting AI agents chain tasks together and review their own work, a landmark paper released this month by Amazon AGI researchers Alejandro Cuadron and Arpit Gupta just demolished that theory.
Titled Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone, the study exposes why traditional coding benchmarks have been lying to us. Most benchmarks test an AI agent on a single isolated request against clean, human-written code.
Real software engineering doesn't work like that. Real engineering is a continuous chain where Ticket #2 builds on the code left behind by Ticket #1.
The researchers evaluated 10 frontier coding agents (including Claude Opus 5.5, GPT-6 Astra, GLM-5.3, and Kimi K3) across 280 real-world chains of 20 consecutive requests (5,600 requests in total).
The Collapse of Unattended Agents
- Taken one request at a time, agents looked brilliant—solving up to 95% of isolated tasks.
- Chained together unattended across 20 steps, average completion collapsed to 51%.
- The stat most people missed: Only 39% of chains were completed without the agent silently missing a requirement or breaking an earlier passing test along the way! In 23% to 43% of attempts, a later request actively broke functionality that worked steps earlier.
Three Breakthrough Findings from the BOBA Paper
When you dig into the data in Sections 4 and 5 of the BOBA paper, three truths emerge that every software leader needs to understand:
A. Agents Write Tests That Share Their Own Delusions
Why do agents fail so catastrophically in chains? Surprisingly, the researchers proved it is not because AI writes inherently unbuildable code. When an agent's code is verified as correct at Step k, it serves as nearly as good a foundation for Step k+1 as human-written code (within 1 to 4 percentage points).
The collapse happens because of unverified silent misses. Coding agents write their own unit tests, see those tests pass, and declare the job "Done". Because the agent's tests share the exact same flawed assumptions as the agent's implementation code, the bug goes undetected. The next request then builds on top of broken code—and fails roughly twice as often.
B. Multi-Agent Self-Review is a Coin Flip (53%)
Can you fix this by having a second AI agent act as a Senior Reviewer to check the first agent's PR? The researchers tested all 10 frontier models on 2,180 code reviews. Overall, asked whether an implementation was correct, agents were right 53% of the time, barely better than flipping a coin.
Even worse, look at how different model families failed as reviewers (Section 4.3):
- "Lenient" Reviewers (the Claude family, GLM-5.3, Kimi K3) rubber-stamped 77% to 95% of broken code as correct.
- "Strict" Reviewers (GPT-6 Astra, GPT-6.1, GPT-5.6) rejected 41% to 58% of completely working code.
You cannot solve a Black Box verification problem by stacking another Black Box on top of it. Without human-in-the-loop visibility, multi-agent pipelines simply compound broken assumptions at machine speed.
C. What Actually Works? "Check Before You Build On It"
In Section 5.4 (Implications for Users), the BOBA researchers issue a blunt prescription that reads like the blueprint for Priset's Glass Box architecture:
- Check before you build on it: Verifying each step before the next prompt is issued completely eliminates the compounding failure rate, making the codebase as reliable as human-written foundations.
- Replace blind retries with specific human feedback: Forcing an agent to blindly retry a failed task has rapidly diminishing returns and increases code regressions by 3–9 points. Giving the agent specific, targeted feedback fixes the foundation in a single round.
4. Real-World Proof: 19% Vibe-Coded vs. 74% Engineered
Academic benchmarks and Wall Street bans tell the macro story, but what does the gap between "Vibe Coding" and true Software Engineering look like in the trenches?
Recently, Richard Aukland founder of aukinfo.com shared a brilliant real-world benchmark from a client data-pipeline project designed to extract critical organizational intelligence from online documents. (Shared here with Richard's permission).
Richard started with a vibe-coded baseline and measured how much usable, verified data it actually captured. Then, he layered in rigorous software engineering architecture and road-tested components.
The difference wasn't marginal. It was a nearly 4x leap in productivity:
| Approach | Verified Extraction Yield | Why It Performed This Way |
|---|---|---|
| Vibe-Coded Baseline | 19% | Skipped PDFs, got blocked by basic anti-bot checks, missed obfuscated text, and polluted results with unverified matches. |
| Engineered System | 74% | Built with deliberate architectural edge-cases, fallback cascades, and strict validation rules. |
Look at the engineering decisions Richard had to build above the vibe-code to achieve that 4x jump:
- Smarter extraction & validation: Wider patterns, proximity matching, and cross-validation against each organization’s actual details.
- Strict grading rules: Ensuring only genuinely useful, high-confidence results were counted.
- Handling the hard formats: Decoding hidden/obfuscated data that simple AI scrapers read straight past, and parsing PDF documents instead of skipping them.
- Resilient traversal architecture: Prioritized link-following, larger page budgets, and automatic sitemap fallbacks for sparse sites.
- Multi-tier retry cascades: Progressive fallback from browser headers → headless browser → visible browser, paired with tested safeguards against anti-bot fake pages.
- Scoped verification: Filtering subdomain searches by ownership and restricting open-web searches to verified name/location matches.
A Black Box prompt gives you the 19% happy-path demo in five minutes. Engineering gives you the remaining 55% required to survive production.
Why the Glass Box Beats the Black Box
All four of these data points: 1) Goldman Sachs' ban, 2) the 77% non-coding reality of enterprise engineering, 3) Amazon’s BOBA benchmark, and 4) Richard Aukland’s 19%-to-74% engineering gap converge on the exact same conclusion:
When you remove human visibility and architectural friction from software development, your system is built on broken assumptions.
At Priset, we rejected the "Black Box" autonomous agent hype from day one. We built our platform around a Glass Box workflow (separating reasoning transparently across Q&A, Architect, and Code modes) because enterprise teams need to answer four fundamental questions at every step of development:
- Can your developers see what the AI is doing? (No hidden assumptions or silent corner-cutting).
- Can they review its architectural decisions before code is written? (Solving the 77% planning and design bottleneck rather than just dumping hundreds of lines of unverified code into a PR).
- Can they intervene and redirect the process mid-flight? ("Checking before you build on it," exactly as the BOBA paper prescribes).
- Does the workflow build human domain expertise or erode it?
AI shouldn't be a Black Box that turns your engineers into passive spectators cleaning up broken assumptions. It should be a Glass Box that amplifies human engineering judgment, keeps accountability clear, and turns 19% prototypes into 74%+ production-grade systems.
Want to see how a Glass Box workflow eliminates review bottlenecks and helps your team become more productive? Explore Priset's Coding Assistant & Orchestration Layer or join the conversation with thousands of engineering leaders on our LinkedIn page.
Priset's Glass Box AI is available now for VS Code, Visual Studio, JetBrains and Open VSX tools like Antigravity. Experience transparent, 100x engineering velocity today.