Mythos is actually some kind of step change, and the fact that anyone with access to the model could in theory find zero day exploits with a simple prompt is ground breaking. I’d argue that a machine that can find zero day exploits at scale is proof of human cybersecurity researcher displacement. And Project Glasswing from this perspective (as well as saving compute) seems like a worthy cause.

I think we continue and will continue to underestimate this different kind of intelligence. It is truly novel to hold the entire problem to solve in a single context window and run attention over the entire context simultaneously rather than sequential human analysis. It’s clear there is a big breakthrough that will augment and partially displace labor.

Mythos is a level of performance that leads to meaningful disruption. The Mythos model found thousands of critical vulnerabilities that survived decades of review. The John Henry moment of man versus machine has already passed. The machine is likely at superhuman levels of performance, especially when it comes to information processing.

So today I want to talk about and introduce a concept that I think will become a much broader phrase in the public sphere in the coming month. And that is "Engels’ Pause.”

What is Engels’ Pause

Engels’ pause is a term coined by economic historian Robert C. Allen to describe the period from 1780 to 1840 when a curious pattern happened during the midst of the industrial revolution. British working-class wages stagnated while per-capita GDP expanded rapidly during the industrial revolution, aka the most consequential technology transition in human history to date.

The math is something like this: economists Charles Harley and Nicholas Crafts estimated per-capita growth at 46% between 1780 and 1840. Charles Feinstein found that working-class wages during the period increased by only 12%. That’s a meaningful gap for workers during one of the most transformative periods in human history. So what caused it?

This is actually a bit of a debate. The analysis suggests that Artisan workers in the domestic system were replaced by machines, often tended by children. The displacement effect was high earning middle class artisans got displaced by capital and the cheapest labor possible. The returns of this output were extremely uneven, corporate profits were captured by industrialists who reinvested them heavily into more factories and more machines.

The destruction in wages was not about unskilled workers, but rather hyper focused on a specific class of skilled artisan middle class workers who commanded a hefty premium. There actually was a bit of a golden age for handloom workers, where the premium was 100% over broader workers. This higher wage created an incentive to displace this labor rapidly. The high premium on this kind of work encouraged its destruction first.

In just one generation, handloom workers wages got halved. And as one activity after another was mechanized, hand workers experienced falling earnings because they competed against mechanized output, and then eventually a flood of displaced workers from other parts of the economy. The average wage of workers did not start to increase until after the full displacement of handiwork.

Pretty much this period of time had returns of technology exclusively accrue to capital. And I think given that we just got a new revolutionary tool that can replace previously hand-churned information work, Engels’ Pause is likely the single most powerful analogy for today and the coming decades.

The Information Artisan Class - Who’s at Risk?

Here comes the unfun part. We have to answer the question, who is the modern handloom weaver?

The information artisan class in my view is around ~70.7 million workers in the “management, professional, and related occupations” industries and represents about 43.9% of the US workforce. They account for about 40-45% of GDP, and if we add broader definitions of office & admin support as well as sales, we can push that number to ~100 million workers of the 161.3 million employed individuals in the US. This is the richest and most valuable part of the economy, and the golden age of information work might have been from 1993 to 2020.

A helpful (AI generated graphic!) is below.

I would go one step at a time over whose most at risk as well as who has the highest average wage.

From the perspective of most lucrative workers to displace, this is it. I would expect entry level roles in financial analysis, compliance, legal document review, data processing, and administration to face near term pressure at the low end. Meanwhile at the higher end I would expect targeted efforts from Claude Code for software, or Harvey for legal to displace workers at the higher end.

At one point in time, it was a relatively easy golden ticket to the middle class with a college degree in business, law, or even just understanding Excel as a nice entry point to the middle class. That “ride” is likely over, and we should expect that this is where jobs will be hurt the most. And given that the technology is diffusing faster than the industrial revolution, we should begin to expect this in years not in decades. This is an incredible risk.

But at the lowest end there is hope. The average wage for unskilled labor using AI should increase if the past is any analogy. Quite literally children equipped with AI should be as skilled or more skilled than a seasoned professional in a data entry job. This means that the wage of a professional will collapse to that of an unskilled worker, and perhaps like Engels’ pause, wages will eventually grow again.

The thing that makes this staggering is this is the majority of jobs in the US today. What would a one to one case look like for the industrial revolution to today? Let’s try to vibe this out.

Information Revolution versus the Industrial Revolution

In the 1790s, specific tasks like spinning and weaving largely got displaced with large capital investments and factory concentration. Today that analogy would be along the lines of $10-25k in tokens replacing the job of a $120k a year analyst.

Now not to be alarmist, but the second scenario to me seems a lot worse. And while I do not expect mass displacement soon, on the margin more supply of information work makes the incumbent knowledge worker’s value massively less useful quickly. And new grads? No chance. I would argue that replacing the $120k a year analyst with $10K in tokens is going to be almost a fiduciary duty, and will happen as soon as feasibly possible. And while the future looks like humans and AI working together, 1 person harnessing a single AI enabled solution to displace 4 people at the same cost is going to be massively deflationary.

Another issue is this is going to be hard to measure. Productivity is starting to tick up while layoff announcements have soared, and entry level wage growth is stalling. This doesn’t definitively prove that displacement happens yet. This is the smoke and the fire is yet to come.

The other issue is it’s going to be complicated to measure. Engels’ pause was named after him because he literally went to the streets of Manchester to count the empty cottages where weavers used to work. If an office is still full but is outputting multiples more information work, it’s hard to measure the abstract displacement. It will be tricky in the coming years to measure what is actually being displaced.

We have likely entered a new information industrial revolution, and I want to remind people how drastic the change during the initial revolution was. I expect this kind of “production increases” to be seen visibly in our economy in a relatively short amount of time (decades).

Industrial Revolution stats from Britain:

Cotton

• in 1750 Britain imported 2.5 million pounds of raw cotton, and by 1787 consumption was 22 million pounds. By 1800, 52 million pounds, and by 1850, 588 million pounds. Over a century that was a 235x increase.

• Cotton spun amounted to 5 million pounds in 1781, increasing to 56 million in 1800.

• The cotton industry rose from 0% of GNP in 1760 to 8% of GNP by 1812.

• The textile industry expanded production 50x between 1780 and 1840.

Iron

• In 1740 Britain produced 17,000 tons of iron. By the 1840s, more than 2 million tons of Iron. In 1852 Britain produced more than the rest of the world combined with 3 million tons. This was a 116x increase in a century.

Overall Economy

• The entire economy’s output 3x’ed over 80 years, most of which happened during Engels’ pause. The net impact was average income per capita doubled, and the share of farming massively fell.

• But the share of national product owned by the top 1% went from 25% in 1801, to 35% in 1848. This was an extremely uneven but meaningful spurt of economic growth.

So let’s be creative. What happens in the case of a new Engels’ pause?

Code is going to be all written by AI, and will 100x total output over the coming decades. The cost of the incumbent code will likely collapse, and software developer wages will decline in real terms.

For example: a legal draft that cost $5,000 in junior associate time will cost $50 dollars in 5 minutes, or a financial model that took an entire day of an entry level analyst will be generated for pennies. A market research report of the entire landscape for a consulting team will take a single person an afternoon.

This is exactly what happened during Engels’ pause, and it took until the railroad revolution for wages to start to increase again. I wrote meaningfully about the railroad revolution in a longer piece.

The argument of how labor became more skilled post industrial revolution is a bit murky too. Mechanical skill is a legitimate skill, but so far “prompt engineering” has been completely displaced by the models themselves. The retooling will be bumpy and uncertain.

But one thing should be clear: production should massively increase. Cotton and Iron increased 235x / 100x over a century during the industrial revolution. The entire economy of Britain 3x’d, but it all came at the cost to labor. Today I believe this could happen, but transitions will hurt.

Pretty much it seems almost impossible from first principles that some sort of new pause won’t happen. Let’s quantify what this would look like.

A New Engels’ Pause

Personally I think a new pause is inevitable. But let’s steelman the counter arguments first. First is that “productivity booms create jobs”, while this is true in the long run, the shorter run is a bit messier. New AI productivity needs to make “new tasks” that can be serviced by humans and fast enough to offset displacement. The reality is that most of the downstream cognitive tasks itself will be more consumption for models, at least if it’s focused on pure information. Maybe we all become physical turks for AI, but that transition period to make wages of that scarce is likely going to happen very slowly.

So what does a new pause look like? Well this what a new “pause” by 2040 would look like. This is pretty much twisting the Engels’ period to today.

This doesn’t look good. If you’re curious what the “permanent underclass” dialogue looks like, this is it. Workers wages only grow nominally, but the excess almost all accrues to capital. And capital will obsessively reinvest back into more capital to AI and this extends the gap.

The ending of the pause took an entire new paradigm to kick off. Railroads became a new driver of labor demand that took the slack that the industrial revolution created.

The Counter Case: Electrification

There was another case, and that’s electrification. Electrification was another technological change that literally added hours in the day for work, and the actual outcome was much better than Engels’ pause. So let’s discuss the counter-case.

There was a productivity paradox that happened in the electrification age that was hard to understand. According to Paul David, you could see electric motors everywhere but not in the productivity statistics. The reason why is this augmented labor that had already adapted to an industrial process, and electric motors often replaced steam engines. Additionally there was a quick feedback loop, it created a whole new set of tasks, namely repairing electrical infrastructure.

The first industrial revolution had already happened, and the labor economy had already adapted. It had absorbed the new tasks, and the new power of electrical equipment augmented the industrial base. What’s more, electrification took almost 30 years to show up in the productivity data! But in this case I think our base case is much closer to Engels’ rather than electrification’s impact.

Electricity complemented a previous meta, while the first industrial revolution completely replaced labor in meaningful swathes. There is a case to be made that perhaps simple pre-AI software has been augmenting human labor. But the analogy breaks down on inspection. Humans have always used tools, and it was humans and looms together that spun fabric, and software of yesteryear looks a lot more like a handloom than a replacement of a current process.

When I ask Claude to make a DCF end to end, there is no human in the loop. Versus before I had a human use a human oriented tool (excel) to create and run the analysis. I fear we are most likely in the Engels’ scenario.

Conclusions (I’m Not Trying to be a Doomer)

As of late I feel like I have been pretty bullish AI and pretty bearish everything that serves as its complement. I think that has been the right trade, and I wrote about this once in June 2025 and again in January.

I think it’s clear to me that we are going through the primary technological revolution in our lifetime. And it could very well be an overbuild, as we have seen before with railroads, electrification, and even the internet. But to dismiss the technology altogether is a mistake. I continue to believe this is likely going to look like a decade long displacement and investment cycle, and I think that the case where capital does poorly in Engels’ pause case… seems not correct.

This also increases the importance of the location of the production in my view. Britain, a small island produced half of the world’s iron by adopting industrial policies fast in 1854. The United States will be the “source” of well over 70%+ of the world’s code as it has deployed the majority of the world’s compute.

The outputs of the information economy will explode, but the headcount will not. And the AI economy today is still a small percentage of the total real economy, but if Cotton in Britain is a guide, it should become a meaningful part of total productivity soon. Capital could have an incredible decade.

Disclosure

This newsletter is published by Fabricated Knowledge and is for informational purposes only. Nothing written here constitutes investment advice, a recommendation to buy or sell any security, or an offer to transact in any financial instrument. The author holds positions in public equities, but does not hold positions in companies or sectors discussed in this newsletter. These positions may change at any time without notice.

The author is co-founder and President of SemiAnalysis LLC, a semiconductor and technology research firm. Views expressed here are the author’s own and do not represent the views of SemiAnalysis.

All information is provided “as is” without warranty of any kind. Historical analogies and forward-looking statements about technology, labor markets, and economic conditions reflect the author’s opinion and should not be relied upon as predictions of future outcomes. Do your own work.

Doug: Welcome back to the Fabricated Knowledge podcast. I have Val from WEKA once again, and we are here to talk about memory. I think from last time to now, it was actually a pretty complete podcast. We were rambling a little bit about KV cache, NAND offloading. And I think from then to now, we’ve really seen quite a different market. NAND prices have exploded. All memory prices have exploded. And I still think it’s the same story. I wanted to update everything, talk to Val, and get some more current thoughts on the whole market. Thanks for being here.

Val: It’s a pleasure. Happy to be a repeat guest.

Doug: Thanks for coming. I think we were talking about this a little bit before — let’s actually talk about the Jensen conversation, because at CES, the big news was — and people took this to the utmost extreme, everyone’s like, “everyone needs 16 terabytes per GPU.” Maybe that’s a little bit of a BlueField advertisement, but let’s just talk about the endorsement of the KV cache market overall.

Val: Absolutely. Like we were saying earlier, it was kind of a confusing topic last summer. The ecosystem wasn’t really ready for KV offloading. Agentic demand hadn’t exploded yet, even though you and I probably anticipated it back then. We’re here now, right? Agents are exploding — even pre-Claude Code. We’re all Claude Code-pilled over the past holidays. We’re seeing trillion-token consumption by AI startups now as a routine thing, daily sometimes. So the demand is there, and it’s cool to see that different vendors had different names for it before Jensen just anointed it as the “context memory” market — and particularly “context memory storage” as we extend storage beyond HBM and DRAM onto NVMe.

Doug: It’s kind of a big deal. I’m actually going to try to do this live and see what SemiAnalysis is paying in terms of OpenAI, because for a company that doesn’t have a massive engineering team — we have a few programmers for sure — but we are definitely adopting this on a pretty hardcore basis. We’re running at like half a billion tokens a day.

Val: I see a lot of organizations around that billion tokens a day, and this number is climbing rapidly. It was funny because we were just chatting about the pricing yesterday. Opus 4.6 fast above 200,000-token context is $225 per million output tokens.

Doug: Wow. I got it wrong.

Val: No, no — there’s actually two tiers. If you’re below 200K tokens and above 200K tokens. I was thinking of the above-200K tier.

Doug: Yeah. Well, let’s be clear — I think everyone at our firm uses the 1-million-token context window. That’s it. It’s so much better. It really is. It’s crazy.

And I feel like maybe that’s the logical segue — the last time we talked, we discussed essentially how you’d have an infinite context window. I think since then, the thing that’s changed is the context rot problem does seem to be real. But that doesn’t mean you don’t need more context. In fact, you need a ton of context. If 2023–2024 was about prompting, context management is the new prompting, right? Thinking about how to intelligently manage all of that.

Let’s chat about how that looks for the current workflow — multi-turn, high concurrency. How does WEKA do that? Maybe go through a basic primer for people.

Val: The big thing is concurrency. Fundamentally, there’s no such thing as a single agent. I kept saying it’s either no agents or agent swarms. And agent swarms implicitly mean a lot of concurrent subtasks happening, typically orchestrated by one or more orchestration agents. That’s where we typically want to pay the Opus price — but usually for execution, we find sometimes Opus is either more efficient or just obviously better-quality tokens.

High concurrency implicitly means we’re often accessing the same codebase, same documentation, same libraries, similar tool calling — and that results in very high KV cache utilization. There’s a really important nuance here for the audience: there are already two kinds of KV caching. There’s a **logical** kind, where you’re seeing the actual reuse in your context amongst the various parallel subtasks. And then there’s the **physical** KV caching — what the inference provider platform can actually do.

The tell is if we go to Anthropic’s prompt caching pricing page. It started off as a very simple page six or seven months ago, especially as Claude Code was launching — just “use caching, it’s cheaper.” Now it’s an encyclopedia of advice on exactly how many cache writes to pre-buy. You’ve got 5-minute tiers, which are very common across the industry, or 1-hour tiers — and nothing above. That’s a really important tell. Then of course you’ve got all sorts of arbitrage opportunities around the pricing for cache reads based on how many cache writes you’ve pre-purchased.

Doug: Let’s actually talk about that. What’s the important tell on the above-1-hour? Is it what I’m thinking — DRAM versus NAND?

Val: It’s multiple tiers. Right now, the tell — if you can’t offer more than an hour — is that there’s no cache offloading beyond DRAM. You’re not offloading to NVMe yet. You’re doing HBM, maybe DRAM pooling, maybe CXL, but that’s basically musical chairs around very finite resources. We’re still talking about 1 to 2 terabytes at most of DRAM per node. With the bigger Blackwells, if that’s what’s being used for inference, maybe collectively about 2 terabytes of HBM across the 8 GPUs. Super finite when you consider trillion-parameter model weights and million-token context windows per user.

Doug: It’s definitely going to go up. I know for the video side — but I’m not quite a believer of 16 terabytes per GPU. I think we’re not quite there. And also, on the training side, it can be very sparse. You don’t actually need that much, because you’re so compute-constrained.

Val: For traditional forward passes, yeah. Maybe another first principles topic here — context memory is not that valuable for traditional pre-training. Reinforcement learning is different. We’re doing a lot of RL loops, and inference is the critical middle part of those RL loops. But for traditional pre-training, no — context memory storage is not a thing. For RL, and particularly for inference, context memory storage is everything in an agentic era.

Doug: And it’s obviously becoming a big thing. I wanted to go back to this because honestly, I didn’t fully understand it when we had this conversation last time. Logical caching versus physical caching — could you explain the two differences?

Val: Sure. When you have multiple turns of an agent, especially an orchestration agent with multiple subtasks, you naturally build up reusable, cacheable tokens. We actually documented this — we released an open-source project called KVCacheTester from WEKA, authored by Callum Fox. He wrote an observability tool that shows you turn-by-turn the logical cache consumption for any set of prompts on any model, because it’s just a proxy. And he’s got a load generator if you want to simulate this or replay existing real-world traces.

What happens is you do naturally build up, after the system prompts, reusable tokens — cacheable tokens. You get insane reusability. You were saying you’re reaching 90%. You can reach 99%.

Doug: I actually looked at this yesterday and it wasn’t quite what I thought it was. It varies a lot. The very high caching rate — I would say it averages around the 70s. There are definitely sessions where you can see heavy usage of the same tokens over and over — boom, 90%. But when you start a new session or have a ramp-down, you can have much lower rates. Blended, I’d say north of 50%, but probably in that 60% range for the actual cache hit rate.

That’s interesting because the variability in pricing on input/output is massive. We were modeling this and it’s like — hey, if it was all uncached and you’re paying the raw price, you’re talking insane numbers. Just paying list price for input.

Val: List price, yeah. It’s like $5 per million input tokens at crazy volumes.

Doug: 100%. And then if you’re doing cache reads, it’s what — a dollar?

Val: The Anthropic cached price is 10% of list — so it’s about 50 cents. A nice 10x difference.

Doug: It’s a big deal. It’s kind of this — OK, so let’s actually talk about this. Specifically, you said this phrase: “logical versus actual.”

Val: Right. So we’re looking at — and hopefully we’ll be able to do a screenshot for people — the logical cache is exactly what we’re seeing here. Variability, but peaks of up to 90–99% KV cache rate, because certain libraries, system prompts, certain tools are very common and absolutely reasonable to reuse across multiple subtasks.

But the 1-hour limit on cache writes you can pre-buy is the tell. What it explicitly tells you is that your KV cache will be evicted after an hour. If you didn’t pay for the hour premium and only bought 5 minutes of cache writes, your KV cache gets evicted after 5 minutes. What tends to happen is either you go on the stereotypical coffee break while an agent runs, or an orchestrator’s got dozens or hundreds of subtasks and not all of them complete at the same time — there’s going to be enough lag time that you’ll easily have a 5-minute gap between subtasks. So it evicts KV cache.

Already you’re seeing the difference between logical and physical KV cache. In reality, with billions of tokens per day and agent swarms, the physical KV cache — as opposed to the logical — you’re evicting all the time. You’re certainly evicting within 5 minutes.

Doug: OK. I see what you mean. Logical versus physical. I imagine the actual token factory tries to keep HBM warm and move things as fast as possible.

Val: Yeah. And now it’s a struggle with the volumes and very finite memory. Actually, it’s about keeping DRAM warm — because HBM is fundamentally for attention computation, and DRAM has been the staging layer for many months now. We keep that warm so we can feed HBM, but we’re totally out of the ability to even keep anything warm. That’s the issue. We’re just refilling over and over again.

Doug: What do you mean by “keeping warm”? As in, it’s constantly being used? You’re saying DRAM is effectively just holding cache — sitting there waiting. It’s not actually doing read/writes.

Val: That’s correct. The Flash Attention algorithms — at least Flash Attention 4; we don’t know what happens with Rubin onwards — don’t calculate attention from DRAM. They move data into HBM, and even that gets moved into SRAM, and that’s how you ultimately do the actual Flash Attention computation. DRAM is a staging area. It’s a green room. And the green room is overloaded right now.

The only fallback is traditional NVMe storage, and it isn’t nearly fast enough to keep up with even just feeding the green room. So what you do is just keep refilling and refilling off the GPU itself, consuming tons of energy, pretty much lighting up the rack every few seconds.

Doug: I want to talk about CXL here, because — you know, memory rack guy over here. I think it’s funny and interesting because I remember writing about CXL in like ‘22, talking about hyperscalers. The original intended purpose was essentially having a giant disaggregated rack system for hyperscalers. It’s crazy to talk about because that use case doesn’t even register to me anymore. It’s useless. Pointless. And now CXL has been zombified for another vision entirely — which is essentially a hot rack for these attention mechanisms.

Val: Totally with you. Ten years ago, this was the classic solution looking for a problem. No one — everyone thought it was geeky, cool, not commercially viable.

Doug: I thought it was because the history of computing is you just disaggregate, pull it out as big as you can, making bigger and bigger elastic pools. It’s funny — we are making elastic pools, but it’s kind of the opposite. Everything is so constrained by the physical pool capacity.

Val: In many ways, it’s going back to the future over and over again. We’re creating separate networks to try to access — it’s not like a mainframe — just really enhanced I/O to complement compute and not have overallocation or starvation. My original memory of CXL way back was like, “Wow, we need to network PCIe.” But that was before Ethernet got so phenomenally fast.

Doug: That itself is a really interesting conversation. I think that wasn’t an accident — it was also because of AI. Ethernet sped up because of AI, and PCIe just isn’t going to be the — well, we’ll see.

Val: That game is over. We’re into common 800-gig ports right now, 1.6 terabits. With PCIe Gen 6 or 7? That’s done.

Doug: I mean, there are still some PCIe accelerators. But I think what’s crazy is now I’ve heard some really wild stories of what they’re doing — like brownfielding DDR4, essentially using the CXL 1.0 spec, and just plugging it into accelerators. It’s kind of crazy. In this shortage, we’re finally seeing the pull-forward demand for CXL. The solution finally has a problem. It’s just very weird.

Val: There’s a really interesting tension right now, because NVIDIA architects would really love this — and I’m not picking on NVIDIA alone, it’s just that they document this so well. They want a separate front-end network, the north-south. They want a very dedicated, untouchable east-west network for NVLink and nothing else. Now they’re saying you want a third network, a BlueField-based network for context memory storage. That’s just a lot of ports without co-packaged optics. And in a shortage, it’s one thing to pay a premium for this — at least you can get it. Can’t get this shit now.

My mind keeps going back to 2024 and one of those original DeepSeek whitepapers that said, “Fuck it, we ball — we’re just going to use all the ports for training, for RL, for inference, all the time. We’re going to dynamically allocate them, actually be elastic with the ports and not have this very hard network segregation.”

Doug: Yeah, that was one of the really impressive innovations around DeepSeek V1. They cracked the algorithm side, which is its own thing. And this is the coolest part about AI — it’s so best-of-breed. It reinvents everything over and over again.

If I had to guess, it feels like if you’re going to attach context memory, you’re going to do it on the north-south with BlueField. I know there’s a third network, but my brain just says the NIC is so close. What’s the future going forward? Do we even need to connect these things to the internet?

Val: I think the future is the DeepSeek approach. For customers that don’t have the staff or the expertise and want to buy a blueprint from a vendor that just works — it’s going to be overprovisioned. It’s going to be expensive.

Doug: Because the vendor overhead is real.

Val: For someone that’s really scrutinizing their gross margins, their hard opex — every microsecond, every micro-cent of token cost matters — you’re going to DeepSeek this. You’re going to put a crack set of engineers on it. You’re going to do congestion control on these networks, and you’re going to oversubscribe instead of overprovision. And that’s how you’re going to have really good gross margin — maybe even positive gross margin on OpenRouter for years.

Doug: What I was going to say — because everyone loves to spec things out, “Oh, this is the possible network” — but we know message sizes aren’t taking up the whole bit. There is slack. My brain goes to the power grid, actually, where everyone is obsessed with the spec for 100% utilization. There’s actually a lot of networking that’s not utilized, and being creative about it matters.

Val: Headers are a thing. Encryption is a thing. But — and this is my one shameless plug — we do pride ourselves on 4KB KV cache offloading over high-speed networks, whether they’re dedicated east-west or otherwise. We’re getting 95% line rate. On just a Hopper-class system — keeping it modest, H200 — we’re getting 3.12 terabits per second for KV cache offflow. That’s CXL-and-above speeds, which is what makes the economics of applying NVMe to a DRAM/DIMM problem so exciting. The arbitrage is massive.

Doug: But for NAND pricing, it’s up so much.

Val: Revenge of the supply chains.

Doug: Month by month, that math might change. I actually want to take it back to DeepSeek. I didn’t have time to read the nGram paper. DeepSeek V4 is probably happening this week.

Val: Maybe as we’re recording.

Doug: Maybe as we’re recording. It’s Chinese model week, guys.

Val: If you’re not familiar, they like to drop things Saturday morning their time — Friday evening Silicon Valley time — just to mess with people.

Doug: Historically, it’s been done on holidays. President’s Day is a good opportunity — I’m hoping. And then GLM-5 yesterday, pretty good. Qwen 2.5 — I’m a fan. I’m a fan of all of it. I’d love to hear what you think nGram could look like.

Val: EnGram is an indicator of absolutely the future. We just don’t know if it’s going to be this month, this quarter — definitely this year. First principles again: if inference servers are spending so much time — and we didn’t even talk about SGLang, HiCache, Dynamo, TensorRT-LLM, and all sorts of stuff AMD is going to do with ROCm — if inference servers are spending all this time just staging tokens to be processed, why don’t the models become natively aware of all this staging?

Why don’t models natively become aware that we have multiple tiers now, including some interesting NVMe tiers, and directly figure out how to allocate or load weights just in time so we’re not hogging memory? Maybe only load the weights that are going to be active for a particular subset of mixture-of-experts. Have models store static common tokens in one tier, and only keep dynamic tokens in HBM. The concept is: we have sparse models, mixtures of experts, conditional compute today — and nGram is going to be conditional memory, loading just in time.

Doug: It feels almost like speculative decoding, where the hardware is aware of what’s around it in order to pull things just in time, and it’s baked into the hardware — the hardware’s problem. In this case, the model is the hardware. So it becomes memory-aware across what is available to it. Is that the right way to think about it?

Val: Yeah, that’s the best way to describe it. The models are now memory-aware. They’re not relying on the inference servers to work around the memory limitations models have.

Doug: Because right now, you force an infrastructure decision and then the model folds into the memory space it has — versus the model being able to elastically consider, “I have this much NVMe, this much HBM, this much DRAM.” That’s really interesting. How does the mechanism actually work? That seems really ambitious. And honestly, it makes a lot of sense because I’m starting to believe that the models, at some level, are an extension of hardware.

Val: That’s exactly where it’s going. And it’s a natural DeepSeek approach, right? Always first principles. Take a look at all the limited available resources — they must be laughing now because the whole world has limited resources, not just China — and say, “Let’s absolutely extract everything we can.” If we can do it more efficiently with the attention algorithm and the forward passing within the model, why offload that optimization responsibility to the inference server at test time? Let’s do it not just at training time, but even pre-reasoning. All sorts of implications around what that means for positional embeddings and beyond.

Doug: Yeah. The V4, we’ll see.

Val: We will see. The paper is out there, so we’ll see whether they deliver.

Doug: I’m pretty interested. Honestly, the state-of-the-art dance has been a lot closer than it’s ever been. Anthropic is spiritually in the lead right now, I think, and OpenAI is now responding, but I feel like OpenAI had a whole year where their capabilities were really about pushing infrastructure and total capacity, not about pushing the bleeding edge. So now everyone’s kind of caught up. I’m really interested in this nGram thing. I’m excited to monitor the situation.

Val: Well, you’ve got to get used to the fact that a year ago OpenAI was a clear leader. Now they’re not. This is the year where one or more Chinese models will be in that top mix, where OpenAI and Anthropic will no longer be the clear leaders.

Doug: We’re already seeing the jagged frontier. Gemini continues to take consumer usage in a way that OpenAI used to have. Gemini is crushing it in the West for video and image. Opus clearly has a giant coding lead — something very special is happening there; you hear everyone saying it. And in China, the video models are insane. They’re probably already state of the art. Seedance is definitely number one right now in my view. And we’re starting to see some interesting things — I mean, I’m obsessed with the agent swarm work from Kimi.

Val: I think the models are the precursor to nGram — which is basically bringing the swarming that happens outside the model inside the model.

Doug: We’ll see. It’s funny that the swarm has to run on like a 16-node H100 cluster. Wow. Just to get this thing going.

Val: Bill Gurley says this so well — this is a sport of kings. At the end of the day, unless you’re doing a desktop local quantized model, you’ve got to be a big inference provider to be in this game.

Doug: It’s pretty crazy. The fact that in order to use a leading-edge capability, we’re talking about what was hundreds of thousands of dollars of compute just to do something at the frontier.

OK, I feel like I hit most of the topics. Is there something I’m missing?

Val: I don’t know if we want to touch on the ramifications for SaaS. I had this simplistic notion that we’d see a $20,000-per-seat agent because we saw $2,000-per-seat agents last year. It’s not shaping up that way. These crazy token costs — API costs into the thousands per day — is what we’re actually seeing.

Doug: I think everyone doesn’t want the per-seat model at the end of the day. It’s interesting — we’ll see. The seat model, if it continues to work, feels super deflationary. That’s my big-brain take I’ve been thinking about — too much volume is going to be deflationary.

Right now at SemiAnalysis, we’re all using fast mode on Opus 4.6. The context window limit is 200K on fast mode. We have a vibe-coding competition every week — whoever has the best project done wins. So yeah, we’re definitely trying to push intelligence per person.

But it’s kind of interesting — maybe this is something I wanted to talk about. Whenever these new models come out, everyone’s like, “Oh, they become so stupid.” It’s quantization, and it’s so painful.

Val: Yeah.

Doug: That’s the key topic. I really wish there was enough capacity to just do full precision — or at least FP16.

Val: It’s not a compute problem at inference time — it’s a memory problem. That’s the key thing. If you reduce the amount of redundant prefills — take that O(N) operation where N could be thousands or millions at scale, down to one — certainly per agent swarm session, because you’ve prefilled all the tokens all the agents and subtasks will need for that swarm — you now have massive savings in accelerator compute time. You can reallocate those GPUs.

We have a little slider demo where you literally detune the number of prefill nodes so you can increase the number of decode nodes. You’ve just got more parallel decode capability at your disposal that improves latency and raises token throughput.

Doug: Yeah, definitely the scale-out version of this.

Val: Meaning you can actually afford to keep high-precision FP8, or even FP16 if you really want it, because now you’re not redundantly doing all those matmuls over and over again on the prefill side.

Doug: I was going to ask — because this is definitely a conversation in the investor world — memory is having its beautiful moment. And I see nothing to stop it anytime soon. Do you think all of the value just accrues to memory? I mean, I’ve seen this story before. Memory is clearly essential — KV cache and scaling it is like an original sin of transformers. We’re definitely struggling. But will that persist?

Val: This is the key question. If we’re in a world where transformers are eating the world — diffusion-transformer hybrids, and Mamba is fundamentally still optimized transformers — as we continue to optimize, if we don’t want a Gartner hype cycle moment for every model release where the inference providers get the non-quantized version to evaluate, all the influencers and benchmarkers get the full-precision version, and then you and I a week later get the quantized version — if we want to avoid that cycle, then yes, memory will rule for a long, long time.

Memory solves this problem. Memory gives you prefill nodes back so you can do full precision, and decode doesn’t matter at that point. You’re able to get the best of all worlds. The memory wall is the thing — you’ve got to scale the memory wall, and most people can’t right now.

The tell will be OpenRouter. I love OpenRouter. They just crossed — I don’t know how many trillions of tokens.

Doug: I think it’s like 1 trillion a day.

Val: Exactly. They’re a real proxy for the industry right now. OpenRouter has been ahead of this — they’ve had, for every model and every provider, input/output pricing plus cache read and cache write pricing. Very few open-model providers even have cache reads as a product offering. And almost none — maybe one or two exceptions that prove the rule — have any kind of cache write functionality and cache write pricing. When those two columns fill up, we’re going to see an explosion of performance and efficiency and higher-quality tokens, because people won’t have to make that tradeoff.

Doug: That makes sense. Cool — anything else you want to talk about?

Val: Next pod. So much to cover.

Doug: Enough conversation for today. Thanks for listening to our high-powered conversation. And thanks for coming by, Val.

Val: We’ll talk soon. Looking forward to it. Bye.

But let’s start with one core conviction. Claude Code is the glimpse of the future. Assuming it improves, has harnesses, and can continue to scale large context windows and only become marginally more intelligent, I believe this is enough to really take us to the next state of AI. I cannot stress enough that Claude Code is the ChatGPT moment repeated. You must try it to understand.

One day, the successor to Claude Code will make a superhuman interface available to everyone. And if Tokens were TCP/IP, Claude Code is the first genuine website built in the age of AI. And this is going to hurt a large part of the software industry.

Software (Especially Seat Based) is in for a Much Rougher Ride

The environment may be rough at OpenAI, but at a traditional SaaS company, there is likely no greater whiplash than SaaS is eating the world in 2012 to Saas is screwed today. The stocks reflect it; multiple compression in the companies has been painful and will persist.

This is structural. I believe it’s time to rethink software’s value proposition, and I have what I consider the best analogy for what the future holds. Afterwards, we will digest what Software will look like as an extension of computing, because I believe that Claude Code resembles the memory hierarchy in computing. Let’s explain.

The New Model of Software

Claude Code (and subsequent innovations) clearly will change a lot about software, but the typical (and right) pushback is that you cannot use “non-deterministic software” for defined business practices. However, there is a persistent design pattern in hardware that addresses this difference: the memory hierarchy. No one can rely on anything in a computer's non-persistent memory, yet it is one of the most valuable components of the entire stack.

For those unfamiliar with computer science, there is a memory hierarchy that trades capacity and persistence for speed, and the system works because there are handoffs between levels. In the traditional stack, SRAM sits at the top; overflow is to DRAM, which is non-persistent (if you turn it off, it goes away), and then to NAND, which is persistent (if you turn it off, it persists).

I don’t think it’s worth matching the hierarchy too closely, but I believe that Claude Code and Agent Next will be the non-persistent memory stack in the compute stack. Claude Code is DRAM.

I believe that AI and software will be an extension of this, and we can already identify which layers correspond to which. The “CPU” in the hierarchy comprises raw information, and the fast memory in the hierarchy corresponds to the context window. This level of context is very fast information, not persistent, and gets cleared systematically. The output of work performed in non-persistent memory is passed to the NAND, which is stored for the long term.

Now that the code is merely an output of hardware, I believe this analogy applies.

AI Agents and their context windows are going to be the new “fast memory”, and I believe that infrastructure software is going to look a lot closer to persistent memory. It will have high value, structured output, and will be accessed and transformed at a much slower rate. I believe the way to think of software, and the “software of the future,” looks a lot more like NAND, and that is persistent, accurate, and information that needs to be stored. In software parlance, it will be the “single source of truth” that AI agents will interact with and manipulate information from.

If you’re so visually inclined, here’s a diagram of a Claude code context window being compacted over and over. Another way to think about this is that it is an identity for a compute cycle; once the task is finished, it is transferred to slower memory and continues.

Each time an AI agent's computation cycle occurs, this is a scratchpad. Each context window is a clock cycle: cached state accumulates until the cache is flushed, after which information is processed. Afterward, the entire context is discarded, leaving only the output. Computation is ephemeral, and information processing by a higher tier of computation largely abstracts away most of human reasoning.

Importantly, I think there is a world in which software doesn’t go away, but its role must change. In this analogy, data, state, and APIs will be persistent storage, akin to NAND, whereas human-oriented consumption software will likely become obsolete. All horizontal software companies oriented at human-based consumption are obsolete. The entire model will be focused on fast information processors (AI Agents), using tokens to transform them and depositing the answers back into memory. Software itself must change to support this core mechanism, as the compute engine at the top of the hierarchy is primarily nonhuman, namely an AI agent.

I believe that next-generation software companies must completely shift their business models to prepare for an AI-driven future of consumption; otherwise, they will be left behind.

Glimpses of the Future

So what does this future look like? I believe that all software must leave information work as soon as possible. I believe that the future role of software will not have much “information processing”, i.e., analysis. Claude Code or Agent-Next will be doing the information synthesis, the GUI, and the workflow. That will be ephemeral and generated for the use at hand. Anyone should be able to access the information they want in the format they want and reference the underlying data.

What I’m trying to say is that the traditional differentiation metrics will change. Faster workflows, better UIs, and smoother integrations will all become worthless, while persistent information, a la an API, will become extremely valuable. Software and infrastructure software will become the “NAND” portion of the memory hierarchy.

And since I’m going to be heavily relying on the history of memory, the last time a new competitive technology came out, it was an extinction event for the magnetic cores that DRAM replaced, and I think this is probably going to be the case for UI companies or companies like Tableau or other visualization software. Zapier / Make as connectors, UiPath, or RPA companies, etc. These are all facing an extinction-level event.

Other companies that I think could be significantly affected include Notion and Airtable. Monday, Asana, and Smartsheet are merely UIs for tasks; why should they exist? Figma could be significantly disrupted if UIs, as a concept humans create for other humans, were to disappear.

Companies that are interesting are “sources of truth,” but many of them need to change. An example might even be Salesforce, a SaaS company. I don’t think the UI is that great, and most of the custom projects are just hardening workflows in the CRM. For Salesforce to make the leap, it needs to focus its product on being consumed by an AI agent, with manipulation and maintenance, while being the best possible NAND in this stack. The problem is that Salesforce will want to try to go up the stack, and by doing so, maybe miss the shift completely.

Most SaaS companies today need to shift their business models to more closely resemble API-based models to align with the memory hierarchy of the future of software. Data’s safekeeping and longer-term storage are largely the role of software companies now, and they must learn to look much more like infrastructure software to be consumed by AI Agents. I believe that is what’s next.

This raises the question: what does this look like for the industry as a whole in the near future? I believe the next 3-5 years will be a catastrophic sea change.

Read more

But first, let’s see who actually won.

2025 Year Performance Review

Let’s ask ourselves who won this year, and the answers are a bit surprising.

If I had to put the stories of 2025 in semiconductors into a single cohesive narrative, it would be:

1. AI spending stepped down the stack. Memory and optics were the beneficiaries. The bottlenecks moved from GPUs to the things that feed and connect them.

2. Semicap finally participated. WFE was tepid, but the stocks ripped anyway. The market is pricing in what’s coming.

3. Automotive remains in purgatory. I thought it would bottom last year. I was wrong. Will it bottom this year? I’m done guessing.

4. This is the year the AI story got leverage. This remains my favorite topic, and I wrote about it in Capital Cycles and AI and referenced it in the railroads piece.

The chart below tells the story.

This year was clearly the year of memory and optics. I was pretty bullish on optics last year, and I think that definitely was the best prediction of them all so far. I also said that HBM would be really solid, and that definitely worked out.

I did, however, get really negative on everything related to the trade war and flipped out a bit too late. But at the end of the year, I am back to a believer, as I believe the most logical path is probably a bubble in the semiconductor / AI world.

2025 Predictions Scorecard

Before we continue into each big key theme, I did want to evaluate some of my “predictions”.

What Worked:

• Optics “most bullish for 2025” — LITE +331%, FN +107%, TSEM +128%

• HBM memory — MU +228%

• AI returns moderate — NVDA +35% (vs +178% in 2024)

• Intel pivot (Aug) — +75% from $21 to $37

• Rambus (MRDIMM) — +72%

• SiTime — +58%

• MPWR — +54%

• Lattice — +31%

What Didn’t:

• AMD bear — +78% (ouch)

• “Avoid NAND/HDD” — WDC +270% (missed a 3x)

• Automotive bottom-fish — ON -12%, MBLY -48% (year 3 of pain)

• Semicap “middle of road” — LRCX +139%, KLAC +93% (undercalled)

I’d say it was okay, but I definitely think I could do better. Anyway, let’s move on to this year’s themes, because I think they are noteworthy.

Memory and Optics

The bottlenecks became the winners. I was bullish on HBM last year, but I underestimated the knock-on effect: when HBM sells out, DRAM tightens. When DRAM tightens, prices rip.

People are buying DRAM for the price of a new PC; the price changes are catastrophic. I have never seen a line increase so rapidly and so sharply in semiconductor history. After the worst memory cycle, we are now in the strongest memory cycle in history. No surprise that a historic AI cycle would cause this. In optics, we are seeing a similar situation, with a complete shortage of EMLs, CW lasers, VSCELs, etc. It’s all backordered, and that’s for the massive 800g cycle. We are not even talking about the higher-performance 1.6T transceivers.

This is a perfect example of an expansion of the previous year’s historic Nvidia rise; it extends to suppliers. I think this will continue, but it’s difficult to replicate this result. I wouldn’t be surprised to see something similar to Nvidia’s performance this year, where returns moderate but are strong after a historic year.

Another signpost I have to consider is that NCNRs are ramping up in memory, and the last cycle occurred 9 months before the peak of the memory cycle. This time, however, I think it will be a much longer cycle, given the magnitude of AI.

Semicap Equipment Roared on Modest Results

I recall that, once, semicap trading at 20x earnings was a hope, and now the average semicap company trades at 20x EBITDA. Discuss multiple expansion, but there have been quite a few commodity providers that have seen even worse on the data center side, so this is not surprising.

The surprising thing is that it’s not that semicap revenue grew rapidly this year; rather, multiples expanded, and forward estimates are rising. Now that’s because stocks trade on expectations, and given that logic is in a shortage, memory pricing is ripping, it’s pretty clear that 2027 and 2028 are going to be wild years for WFE.

I think the fundamental story for Semicap is just beginning, and I expect to see pretty massive beats in the coming quarters.

Automotive is in Death Valley

Last year, I tried to call the bottom in automotive. It never happened. One way to look at this is to track the historical inventory cycles of automotive and analog companies.

We didn’t experience the drastic correction observed in other markets, and we are now in the “recovery” quadrant despite a mild correction. The market has been and continues to expect weakness, and as much as I want to bottom call again, I mostly want to give up.

Maybe that means automotive and analog rips, but, man, it’s a tough market with the continued Chinese competition. If you asked me to put a gun to my head, I’m long. But I would have said that last year, and would be dead. So your mileage may vary. It’s troughed, but will it go up? Who knows!

AI Learned about Leverage

This is the year the AI story got leverage. I wrote about this in

and again in

In the second, I called the “alcohol” of this party, and I'm going to use this analogy in the future. Oracle is drinking the hardest, borrowing $100 billion over four years for the Stargate project.OpenAI has committed $300 billion to data center buildouts, while internal documents project $115 billion in cumulative cash burn through 2029, a period executives reportedly call “the valley of death.” There’s only way to party this hard, and it’s leverage.

Meanwhile, financing became a bit more circular this year. Nvidia takes equity stakes in GPU customers. Those customers buy more GPUs. The GPUs become collateral for debt facilities. The debt buys more GPUs. Nvidia’s revenue rises. Repeat.

None of this is illegal or actually even unusual for a capital cycle. But the bond market has begun asking questions that the equity market has not. CoreWeave’s credit default swap spreads widened sharply in late 2025, but in recent weeks have relaxed. This is now a key factor in this cycle. We will have a lot more to see this coming year, especially with a potential IPO. Supply is coming to the market to meet demand.

Anyways, it’s time for the broader semiconductor cycle.

The Total Semiconductor Cycle

Let’s now turn to my favorite (and I think famous) chart about semiconductor cycle positioning. As you can tell, we are now in a huge upswing, and I believe almost every single company will make it out of the recovery cycle into the expansion cycle. The real question is, when will inventory growth outweigh revenue growth?

What’s also interesting is that Memory is still in the recovery cycle; we still aren’t even seeing inventory builds meaningfully. This is usually an “okay” time to own, but once meaningful inventory buildup begins, caution is warranted. Also, it’s surprising to see Automotive still in recovery, but not unexpected. It had previously spent the longest time in the expansion quadrant, as the 2020 cycle resulted in almost 4 years of consistent revenue growth.

2025 Valuation and Valuation Changes

The median semiconductor company is not cheap. The most costly are SITM, AEHR, ALAB, and LSCC.

Some are during their cyclical troughs, so it makes sense. Still, the most exciting situation is when a company is trading at below-average valuation and high quality (looking at you, TSM, ENTG) or when it’s an excellent company trading at average multiples. That would be AVGO, NVDA, TSEM.

But another way to frame it is, did the company become more or less expensive? This is my graph to represent this. Now, the problem is the outliers here, so I did a zoom-in.

Pretty much it’s better to be below the line, not above. It’s not surprising that companies that are performing poorly are becoming more expensive (SWKS, COHU, etc.). Still, it is surprising to see companies that perform well become cheaper, namely LITE, CRDO, and SMTC. That is usually an interesting place to fish. Additionally, the observation that NVDA trades below the line is noteworthy, and as it assumes and retains the most prominent position in the market, multiple compressions are somewhat unsurprising.

It’s cool finally seeing ASML multiple compress, and it’s funny because it’s just in time for reacceleration.

Now, the final chart I love to do is the underperforming and outperforming chart considering valuation. Historically, you want to buy cheap underperformers, but companies that are underperforming this cycle often have a particular set of issues with them.

Of the companies I see at the bottom, I think most of them are automotive, and that itself is a cycle call. Meanwhile, being cheap and outperforming is another interesting concept, and this is where memory will “shine” as EPS rises and multiple contracts are signed. Finally, it is expensive yet outperforms the competition. Nvidia and AVGO are truly incredible because they are decently priced, continue to outperform (marginally), and are “becoming the market”. Now, let’s step through each segment cycle.

Sector by Sector Overview

Now it’s time to walk through the sectors to review the inventory cycle of each. First and foremost, let’s do the most important segment, “fabless semiconductor” which really is just AI revenue now. Everyone expect AMD has meaningful amount of revenue exposure here, and you can argue it’s AMD’s “story”.

The interesting thing is that you can see the inventory cycle that occurred during the GB200 build-up partially resolve itself. While there was a point where inventory was growing faster than revenue, we are now back to a very expansionary market. This is likely to be a “fine” year for these companies, and I expect further outperformance (though probably not eye-watering). I’m still not a fan of Marvell, but at this point hell, even they deserve a bid.

AI Networking

This is probably the bottleneck and will remain the bottleneck. I really like the setup of these companies, even after the prodigious run. Odds are we will see slightly moderated outperformance given how hard it ripped year to date, but I do like share gain stories like SMTC, which should gain share at TPU.

I expect another great year in AI networking, as hilariously enough, most of the 800G and 1.6T cycle is still ahead of us, not behind us.

Meanwhile, I’ll have a separate call out for optics, which is probably the hottest segment right now. Another name not mentioned here is 5802 and AXTI, which are part of the InP and CW laser trade. These stocks are parabolic, but the demand seems unstoppable.

This is pretty much an industry on fire, and I think LITE is leading the way. I don’t really have much to say other than “wow look at them go”.

Analog / Automotive Cycle

Now this is where I think I will be spending some tokens to discuss. Look at 2022 and how the inventory cycle has continuious stayed high, despite even negative revenue. This is a chart that is showing the global buildup in inventory in 1 picture, and I think that because there never was the horrific blowup in inventory, there isn’t the parabolic rise today.

Now, the thing is, I think that will change on the margin. While MPWR, ON, ADI, TXN, NXPI, and IFX are mostly analog and power companies, I do believe that this time, surging demand for smaller parts of their businesses should pull those parts out of the trough.

The companies that might show (especially last year’s plunge) way better are the much smaller, pure-play companies. WOLF, INDI, MBLY have corrected sharply out of the trough, but still are burning through inventory. In aggregate, this is a healthy place to be.

Pretty much this is the single best setup in terms of inventory timing / the cycle. Maybe this is the year that it rallies finally, but I have been burned so many times before.

Foundry Cycle

This one, in my opinion, is somewhat imperfect. Inventory of wafers is rarely built to be busts in the way that semiconductors do, but I do think that the positioning of the foundry right now is very good.

I think there is a non-zero chance that foundry is the next bottleneck, as we know shortages in leading logic are happening at TSMC, which are overflowing to Intel and even Samsung. While AI is the story of the day, it is starting to completely fill the leading nodes at TSMC, with PC and smartphones going to other foundries as a gap fill. In some ways, this is like the HBM thesis, and Foundry is going to get the lift from AI selling out TSMC so completely.

Which brings me to my 5-second conversation on Semicap. I think that if the above is true (I believe it is), then the semicap stocks are about to have the strongest, most meaningful year in WFE history. Probably not a calendar 26 over 25, but something like 2H26 + 1H27 over 2H25 + 1H26. There’s clearly substantial demand, and we are starting to see pull-ins everywhere, including in China WFE.

Memory

We are still in the inventory burning phase, and likely will be for sometime to come. All of HBM is sold out, all of DRAM is sold out, spot is going to be probably be ripping on the back price increases / relaps.

I know that memory will have at least an incredible 6 months, if not 12. Who knows what stock prices do? This is where things get dangerous, but I believe it gets better fundamentally for some time yet. And that, of course, accrues to Semicap.

Mobile / RF

Inventory after multiple its so over and it’s so backs (you can see it in the chart) is finally so back. But now the real headwind is going to be memory pricing.

The consensus is that memory pricing is going to drive up phone prices so much that there’s really only one outcome: people buying fewer phones. I think that happens, and I think this is probably a lag to a certain extent. Inventory is cleaned up, but smartphones really are the new PC. It will likely be a decade of irrelevance before we care about phones again. This is, and will likely remain, the permanent funding shortfall.

Which Now brings me to my Predictions

Read more

This is the story of the railroad capital cycle: how America financed, built, overbought, and eventually consolidated the most transformative infrastructure of the 19th century.

Early American History of Railroads: The Civil War and Land Grants

Where does a cycle begin? For American railroads, the Civil War is a good starting point: the conflict demonstrated the value of railroads and was the first major use case for the transformative technology. Better, cheaper, and faster logistics could win wars, and railroads were partially why the North won.

The war was fought over railroads. For example, “Every major Civil War battle east of the Mississippi River took place within twenty miles of a rail line,” and one of the few critical advantages the North had over the South was a dominant standard railroad gauge and a higher quality network. Notice the redundancy of the northern roads compared to the south. And while many factors contributed to the Union victory, this was clearly one of them.

The war demonstrated the value of the railroads, but in peacetime, the railroad industry itself roared to life. At the end of the Civil War (1865), it became clear that technology would shape the American West, and railroad construction increased from ~1000 miles per year to 7000 miles per year within a few years.

But before we go into the dramatic rise (and fall, and rise, and fall) of railroads, I need to discuss one of the core mechanisms that helped finance them in the early days: land grants.

Land Grants and Railroads

The mother of most railroads was land grants. Land grants enabled the financing and construction of the first railroads. The Pacific Railroad Act of 1862, passed during the Civil War, aimed to bind California to the Union before it could contemplate secession and was the first transcontinental railroad endeavor.

Transcontinental railroads were ruinously expensive to build, with costs ranging from $45,000 to $60,000 per mile. In today’s dollars, that’s roughly $2.5 million per mile; for context, the first transcontinental cost about $1.2 billion total. The federal government was cash-strapped, but it had abundant land.

So a clever solution was created: grant land in a 10 to 40 mile corridor around a new railroad to the company itself, and parcel the land 50/50 between the railroad and the government. This would increase the land’s value, as it was in a remote location. The odd sections of the land parcels were conveyed to the railroads, which would then issue securities secured by the land.

On paper, this was elegant. Public land was sold for $1.25 per acre. Grant half to a railroad, and the resulting settlement would double the value of the government’s retained sections. This was a revenue-neutral transaction for the government, or so the theory went. Another consideration was that Manifest Destiny was viewed as an inevitable defining force of the time, and to expand West was the Zeitgiest itself.

The railroad developer, as the new landowner, had a dual mandate: that of a real estate developer and a transportation company. Sell a farmer's land, then ship his crops when the railroad arrives. The customer base was embedded in the business model and was an essential component of the railroad supply and demand.

But the reality of the land grants was a bit less rosy, according to Robert Henry, the realized price for the government was ~97.2 cents, higher than the carrying value of 12.5-23 cents per acre without transportation access. It created value, but it didn’t pay for the railroads outright.

Railroads, however, monetized their lands at much higher prices, selling them for approximately $2.81-$3.38 per acre. This was still a huge return and was the primary funding mechanism for the first railroads.

The scale of the land grants was staggering and often exaggerated. Maps like the one below frequently overstated the amount of land granted by a factor of 4x or more and depicted the entire indemnity limits rather than the actual checkerboard. The true figure was roughly 131 million acres, about 10% of the total public domain. For perspective, that’s larger than California and New York combined.

Land was sufficient to prime the pump, and the first major expansion began in 1865 and reached its zenith in 1873, when it collapsed.

The First Great Railroad Buildout Plus Meet the Railroads

Like any mania, excitement went vertical. The post-1865 period was the big bang, with Promontory Point as its focal point.

The network expanded from 35,000 miles at war’s end to 70,000 by 1873, attracting a tidal wave of foreign capital. According to Henry Parris’s “Slow Train to Paradise,” 75% of railroad securities were held by British and Dutch investors seeking growth unavailable in Europe.

The railroad’s biggest product, in many ways, was the act of fundraising itself. Insiders enriched themselves by extracting as much as possible per mile of railroad built. The goal was simple: build a railroad and pay yourself to do so.

The absurdity of the self-enrichment was visible even at the moment of the Big Bang—Promontory Summit wasn’t chosen for engineering or commercial reasons. Congress insisted on it because the Union Pacific and Central Pacific, racing for land grants, had actually built past each other. Their parallel grades ran side by side for over 200 miles, each company laying track through the same Utah territory to claim more government land. Congress finally intervened and designated Promontory as the meeting point. On May 10, 1869, seven years after the Pacific Railroad Act, the golden spike was driven.

So while I’ve been talking about the great East meets West, let’s actually meet the railroads themselves. It’s important to know the main characters themselves. I will resume the story of the first peak in the railroad buildout after I introduce the key players.

Meet the Railroads and Their Players

The railroads and the Gilded Age and Robber Barons themselves are impossible to disentangle. The men who built and looted these companies became the era’s defining figures. What follows is a brief guide to the key players and their railroads. So let’s meet the most iconic rail of the era, the Erie.

The Erie Railroad: Vanderbilt Jay Gould, Jim Fisk, Daniel Drew (The Speculators)

The Erie connected New York to the Great Lakes and was among the most strategically valuable railroads of its time. Cornelius Vanderbilt, who had built his fortune in steamships before consolidating New York’s railroads, realized that by purchasing it, he could create a larger infrastructure empire. In early 1868, he began buying Erie shares on the open market, assuming the supply was finite.

Supply was infinite. That’s because Jay Gould, Daniel Drew, and Jim Fisk, who controlled the Erie board, set up a printing press in the company’s headquarters and began issuing convertible bonds, which they immediately converted to stock and dumped on the market. For every share Vanderbilt bought, they issued more shares, and this was the first “watering” of stock. After all was said and done, the share count of the Erie increased by over 200%.

Vanderbilt secured an arrest warrant from a friendly (bought) judge. The Erie trio fled across the Hudson to New Jersey, beyond the court’s jurisdiction, with hired guards and, reportedly, cannons defending their hotel. The standoff ended in a negotiated settlement. Vanderbilt lost millions and never acquired the Erie. The Erie was a systematic example of the era, reflecting control over shares and the processes of selling and issuing stock.

Union Pacific: Bankrupt Transcontinental into Shining Success, (Thomas Durant, E.H. Harriman)

The Union Pacific still exists today, but its early history was the most scandalous of any American railroad. Congress chartered it in 1862 to build the eastern leg of the transcontinental, from Council Bluffs, Iowa to Promontory Summit. Thomas Durant, the railroad’s principal promoter, controlled the company but was primarily interested in extracting revenue rather than building efficiently.

His mechanism was the Crédit Mobilier of America. Durant and his allies owned this construction company, which contracted with Union Pacific to build the railroad at wildly inflated prices. Each time a contract with Union Pacific was signed, Credit Mobilier inflated construction costs and booked profits for its owners.

To protect the scheme, Durant distributed Crédit Mobilier shares to congressmen for free, who then controlled the railroad’s land grants and subsidies. The famous “Ox-Bow Incident” captured the incentives perfectly: Durant routed the railroad in a nine-mile loop through Nebraska for no reason other than to claim an extra $144,000 in bonds and land grants.

The scandal broke in 1872 when newspapers revealed the congressional bribery. Eight members of Congress and Vice President Schuyler Colfax were implicated. Union Pacific limped through the Panic of 1873 and into receivership. The railroad Durant built was so shoddy that it could barely operate until much later.

Enter Edward H. Harriman in 1897, after Union Pacific’s second bankruptcy. Harriman was Durant’s opposite: operationally obsessed, financially conservative, focused on the long term. He poured tens of millions into reducing grades, straightening curves, laying heavier rail, and replacing rolling stock. Costs fell. Capacity rose. Union Pacific became one of the best-run railroads in America.

Harriman was still a robber baron. However, he consolidated and operated rather than extracted. His empire would eventually rival Morgan’s. To understand how, we need to meet two more railroads he acquired.

The Central Pacific / Southern Pacific: California’s Railroad (The Big Four, E.H. Harriman)

If Durant represented Eastern corruption, California’s “Big Four” were his Western counterparts: Collis P. Huntington, Leland Stanford, Mark Hopkins, and Charles Crocker. These Sacramento merchants had no railroad experience when they backed Theodore Judah’s vision of a railroad across the Sierra Nevada in 1861. Judah died of yellow fever in 1863, crossing Panama to seek financing in New York. He never saw his railroad completed. The Big Four did, and they profited handsomely from Judah’s work.

The Central Pacific was primarily built by Chinese laborers, thousands of whom blasted tunnels and laid track through some of the most brutal terrain on the continent. The Big Four, like Durant, created their own construction company to overcharge the railroad. By the 1880s, the Central Pacific had merged into the Southern Pacific, and the combined system controlled nearly every rail mile in California. Farmers and merchants called it “the Octopus” for its stranglehold on rates. Leland Stanford, meanwhile, parlayed his railroad fortune into a governorship, a Senate seat, and a university.

The Southern Pacific eventually stretched from Portland to New Orleans, becoming the dominant railroad of the West. Harriman acquired control in 1901 and, as with Union Pacific, invested heavily in rehabilitation. For a brief period, he controlled both railroads, creating the most powerful transportation empire America had seen. The Supreme Court forced the combination apart in 1913, but by then Harriman was dead.

The Great Northern (James Hill)

James J. Hill was the anti-everything. Hill built without land grants and managed his empire differently compared to the other majors. Where they watered stock, Hill reinvested profits in the railroad. Where they extracted, he developed. The Canadian-born “Empire Builder” created the only transcontinental railroad that never went bankrupt, and he did it by being smarter, cheaper, and more patient than everyone else.

Hill started with the bankrupt St. Paul & Pacific in 1878, a distressed Minnesota railroad that more astute investors had dismissed. He rebuilt it methodically, extending west only when the existing line was profitable, selling farmland to settlers who would then ship crops on his trains. By 1889, he’d renamed it the Great Northern Railway and set his sights on Puget Sound.

His secret weapon was Marias Pass. Native American stories spoke of a low crossing through the Rockies in northern Montana, but no surveyor had found it. Hill hired John F. Stevens, who located the pass in December 1889, at an elevation of 5,215 feet, versus the Northern Pacific’s much higher crossing to the south. Hill’s route had lower grades, fewer curves, and lower operating costs. When the Great Northern reached Seattle in 1893, it was the most efficiently built transcontinental in America.

Hill’s philosophy was that a railroad’s prosperity depended on the prosperity of the territory it served. He distributed purebred bulls to farmers, promoted scientific agriculture, recruited immigrants from Scandinavia, and developed infrastructure rather than simply extracting resources. When the Panic of 1893 bankrupted the Northern Pacific, the Santa Fe, and the Union Pacific, the Great Northern continued to operate profitably.

Hill eventually gained control of the Northern Pacific and the Chicago, Burlington & Quincy, creating a northwestern empire that rivaled Harriman’s. The two titans clashed spectacularly in 1901 when Harriman attempted to seize the Northern Pacific through a stock corner, nearly crashing Wall Street before J.P. Morgan brokered a truce. More on that later.

The Atchison, Topeka and Santa Fe (Cyrus Holliday, William Barstow Strong, Fred Harvey)

The Santa Fe was built on audacity and marketing genius. Cyrus K. Holliday, a Topeka lawyer, drafted the railroad’s charter in a Kansas hotel room in 1859, dreaming of a line that would follow the old Santa Fe Trail to the Pacific.

The railroad grew through cattle towns: Newton, Wichita, Dodge City. It became the preferred shipper for Texas cattle drives heading to Kansas railheads. But the true expansion came under William Barstow Strong in the 1880s. Strong raced the Denver & Rio Grande for control of Raton Pass, the key gateway into New Mexico. He sent construction crews to occupy the pass at dawn, hours before his rivals arrived. The Santa Fe reached Los Angeles by 1887 and Chicago by 1888, becoming the only transcontinental connecting the Midwest directly to Southern California.

What made the Santa Fe legendary, though, was Fred Harvey. Beginning in 1876, Harvey built a chain of restaurants, hotels, and dining cars along the route, transforming western travel from an ordeal to an experience. The Harvey Houses served high-quality food on real china with professional service, a revolutionary innovation for the frontier. The “Harvey Girls,” young women recruited from the East to work as waitresses, became a civilizing force in railroad towns and the subject of a 1946 MGM musical. The Santa Fe focused on the customer experience.

Okay, after the long detour of players, let’s get back to the first build and bust cycle. As you can tell, we have many busts to discuss.

The Build and Bust (1873)

Throughout the original buildout, railroad securities were the hottest asset class in the world. European capital poured into America to finance the new network.

However, the land grant model created a perverse incentive: build first, address traffic later. For many railroads, the goal was to lay track and claim land. Whether the route made economic sense was secondary. The result was massive overcapacity and dozens of duplicate routes competing for the same freight.

The breaking point for this cycle came from capital markets, and not the technology. In May 1873, the Vienna Stock Exchange collapsed after Austria-Hungary demonetized silver, triggering bank failures across Europe. Investors who had financed American railroads suddenly needed liquidity. They sold shares for liquidity, and selling pressure mounted through the summer. The Credit Mobilier scandal also ended all land grants after the fraudulent allegations.

In September of 1873, Jay Cooke & Company declared bankruptcy. Cooke had financed the Union during the Civil War and emerged as America’s most prestigious banker. After the war, he bet everything on the Northern Pacific, attempting to build a second transcontinental without the subsidies that Union Pacific and Central Pacific had enjoyed. But he ran out of money as bond sales stalled. The bank collapsed on September 18.

The New York Stock Exchange closed for ten days. The Panic of 1873 was called the “Great Depression” until the 1930s claimed the title. Railroad construction fell 78% between 1872 and 1875. Eighteen thousand businesses failed. Unemployment reached 14% nationally and 25% in New York City.

Railroads suffered the worst of all. Between 1873 and 1879, half of all railroad bonds defaulted. More than a billion dollars (~134 billion dollars today) in debt went bad in 1874 alone. Union Pacific entered receivership. Northern Pacific went formally bankrupt, its line stranded incomplete in Montana. More minor roads vanished.

But the track stayed in the ground. The rolling stock continued to roll, slower now, less frantic. And when the economy finally recovered, all that infrastructure, built with European money, now wiped out, was available at fire-sale prices. The contraction lasted 65 months, the longest in American history to that point. Workers bore the brunt: wages collapsed, and the Great Railroad Strike of 1877 became one of the most violent labor actions the country had seen. The men who bought bankrupt railroads in 1878 built some of the era’s greatest fortunes.

The Second Build: From Bust to New Mania (1879-1893)

Here’s the thing about capital-intensive industries: they are cyclical. The 1873 crash was the warm-up for the second larger cycle. The expansion from 1879 to the zenith in 1893 is what we now call the Gilded Age, and the scale dwarfed that of earlier periods.

By 1882, more track was laid in a single year than had existed in the entire country in 1850. This was the core period of rapid railroad expansion, during which railroad mileage more than doubled, from 93,000 miles to 164,000 miles. At the end of this period, the United States had the most railroad track laid in the world, and it still holds that title to this day. And this time it wasn’t land grants, but Wall Street that led the way.

The Invention of American Finance: Financing Railroads

In 1871, after the Credit Mobilier scandal, land grants from the government rolled to a halt. So how did railroads accelerate despite the end of “free land”? The answer was Wall Street. And to go deeper, you have to understand that Wall Street as we know it didn’t exist before railroads. The railroads created such a funding need that they gave rise to Wall Street.

Before the advent of railroads, Wall Street barely functioned as a capital market. In 1830, there were a few dozen securities trading, mostly bank shares and state bonds. Daily volume measured in hundreds of shares. The modern stock market didn’t exist because there was nothing large enough to require it.

A textile mill at the time costs $100,000. Wealthy individuals, retained earnings, or short-term bank loans could cover that. Railroads cost millions. A hundred miles of track between two cities ran $3-5 million. A trunk line cost $20-30 million. A transcontinental cost more than the annual federal budget.

No single bank had that kind of capital. Railroads required a new solution: raising capital from thousands of investors who had never met one another, had never seen the property, and would never manage the company. The concept of modern shareholders, the capital stack, and how to finance large projects emerged in response to the funding needs of railroads.

Common stock existed before railroads, but was rarely traded. Railroad shares were the first stocks to be widely traded by a broader group of people rather than a small group of owners. So when you see the claim that “railroads were 60%+ of the market,” it is backwards. Railroads essentially gave rise to capital stock markets because of the substantial capital requirements to build them, so of course, the concentration in the 1880s was massive.

Meanwhile, corporate bonds as a mass-market security were a railroad innovation. The concept of a capital stack was developed to finance railroad construction. Investment banking as a profession emerged to sell railroad securities. The great banking houses - namely, J.P. Morgan, Kuhn Loeb, Jay Cooke, Drexel- all built their fortunes on railroads.

The ticker tape itself was designed primarily to transmit railroad stock prices! Financial journalism was invented to cover railroads. The Wall Street Journal was founded in 1889 primarily as a railroad sheet. Moody's, Standard & Poor's, and many modern financial companies were established to assess railroads.

Accounting standards were introduced to standardize the preparation of often fabricated annual reports and financial statements. Depreciation as a concept was introduced to account for the long lives of railroads. All of finance was invented as a downstream industry to support the railroads, and that itself is an incredible story. The 1880s were a railroad-dominated stock market, and bonds actually accounted for an even higher share of total issuance than stocks. The entire economy was a railroad.

And the birth of the 1880s Wall Street machine was built with a single purpose: to finance the second, much larger machine of railroads. And Wall Street’s strength had grown powerful to create something genuinely unprecedented, and that is the second great boom.

The First Great Wall Street Boom

The 1880s saw the full strength of financial engineering bent on building more railroads. The first generation of promoters, such as Durant, had used construction companies to extract value; the second group of titans realized they could extract more value from railroad securities and expectations on the railroads. Railroads issued bonds against future traffic, against branch lines not yet built, against connections to towns that didn’t exist except for on promotional maps. These were literal roads to nowhere financed by paper securities.

The entire railroad capitalization (stocks plus bonds) grew from $4.6 billion in 1876 to $10.6 billion in 1890. The incredible thing is that most of this was not actual infrastructure but financial products. One railroad historian estimated that by 1890, 40% of railroad capitalization represented “water”, or securities issued in excess of any investment in roadbed, rails, or rolling stock.

And this incredible financial machine lured back European investors en masse. Despite being burnt in 1873, foreign holdings surged to $3 billion, or a quarter of all outstanding railroad capital in America. And unlike the land-grant era, in which the government provided at least some discipline through the granting process, Wall Street financed anyone with a sufficiently compelling story and sufficient demand. If investors purchased the bonds, they would be issued. If you could sell a story and bonds, you too could potentially build a railroad, regardless of whether the railroad made any economic sense at all.

The Transcontinental Race (Again)

Despite the previous transcontinental going into bankruptcy, the urge to build another one roared again to life. In 1880, there was effectively one transcontinental, but by 1893, there were five transcontinentals.

The Northern Pacific, left incomplete and bankrupt in Montana, reorganized and finished in 1883 under Henry Villard. Henry Villard was an incredible promoter, and after completing the road promptly ran out of money. Northern Pacific would struggle for another decade.

The Sante Fe, under William Barstow Strong, reached Los Angeles in 1887. Strong acquired his way eastward, reaching Chicago in 1888. The Santa Fe connected the Great Lakes to the Pacific.

James Hill’s Great Northern reached Seattle in 1893, and was the only transcontinental built without land grants and without meaningful leverage. More on Hill later.

The Denver & Rio Grande, the Missouri Pacific and Texas & Pacific extended towards the Southwest. Regional systems such as the Chicago & Northwestern, the Burlington, etc, all expanded to prevent being outflanked by competitors.

But the problem was that Wall Street's financing capacity had completely outstripped economic reality, and the system could raise more capital than the economy could productively absorb. Five transcontinental railroads existed when demand could barely sustain two, and many railroads built parallel routes competing for the same shipments. The railroads massively overbuilt for a future that would never arrive. Enter the rate wars.

The Rate War Spiral

It’s simple, if there is too much supply chasing too little demand, prices collapse. And this is where the rate wars began. Rate cuts were vicious and destructive cycles of competitive price-cutting that drove rates often below operating costs. The Chicago to New York freight rate fell from 35 cents per hundred pounds in 1873 to under 20 cents by 1886. Passenger rates fell from over $100 a ticket to as low as $1 during the most intense fare wars of 1886-1887.

Railroads formed cartels to fix prices on routes. But the cartels collapsed, the temptation to cheat was irresistible when you had high fixed costs and empty variable pricing (cars) to fill. Railroads even lobbied for regulation, and the Interstate Commerce Act of 1887 proved toothless and not enough to turn around the price war.

Nothing could stem the overcapacity. No financial arrangements could fix the reality that there were too many railroads chasing too few customers.

Jay Gould really embodied this era. After his Erie saga, he moved west and assembled 16,000 miles of railroad. Similar to the Erie, he specialized in speculation and financial manipulation. He acquired a struggling railroad, issued bonds to raise cash, triggered rate wars to damage competitors, and often profited more by shorting competitors’ securities simultaneously. Jay Gould died in 1892, one year before the Panic of 1893.

The Panic of 1893

The 1893 collapse destabilized the system and was among the first global crises of its time. It’s hard to start in one place, but let’s begin with Argentine railroads.

In 1890, Argentina had the highest per-capita foreign debt in the world, and when a failed coup rattled Buenos Aires that summer, creditors panicked. Baring Brothers, one of Britain’s oldest banks, found itself holding the bag. In November of 1890, it nearly collapsed, but survived after the Bank of England organized a rescue.

British investors started the risk-off trade, and Australia was next. Melbourne had a speculative railroad mania in the 1880s, and during its collapse in 1893, real GDP fell 17%. This contagion spread worldwide: British investors burned in Argentina, British investors in Australia began to sell American securities, and gold began flowing out of the United States. Enter McKinley and the Silver Purchase Act.

The McKinley Tariff and the Silver Problem

In 1890, Congress passed two pieces of legislation that, in combination, were a catastrophe. First McKinley Tariffs raised average duties to nearly 50% to protect the domestic industry. This reduced imports, thereby limiting foreigners' ability to purchase American products and, significantly, to own American securities. International trade contracted, and gold continued flowing out of America.

Second, the Sherman Silver Purchase Act created a second type of gold drain. Western farmers (and populists) demanded “free silver” or the ability to swap silver for gold at a fixed ratio, creating inflation that would help ease debts and inflate crop prices. Now, the government didn’t love this, so the Sherman Act was a compromise; it required the Treasury to purchase 4.5 million ounces of silver monthly, paid with notes redeemable in either gold or silver.

The problem is that silver is easier to obtain than gold, and silver was overvalued relative to the gold peg. Holders of Treasury notes redeemed them exclusively in gold, not silver. By issuing more notes that would be withdrawn this accelerated the gold withdrawals. Reserves at the United States Treasury fell from $190 million to the $100 million minimum required by law. Foreign observers doubted whether the gold standard would hold in the United States, which in turn precipitated a further run on Treasury securities and increased pressure on gold and the gold standard.

By early 1893, the treasury’s gold reserve was dwindling. This is all to set the stage for the eventual panic.

Bank Failures and Dominoes Fall

The Philadelphia & Reading Railroad declared bankruptcy on February 20, 1893, 10 days before Grover Cleveland’s inauguration. The Reading tried to corner coal, and while the company failed, the market mostly absorbed the shock.

In May, the National Cordage Company (a rope trust) collapsed, triggering a broader stock market fall. Credit markets froze, banks called loans, and then railroads, with massive debt balances backed by minimal earnings, began to fail. This was a violent and aggressive move, much like 1873, but on a larger scale.

By the end of 1893, 74 railroads entered receivership, representing 30,000 miles of track and $1.8 billion in capital. By 1897, 192 railroads failed, representing 1/4th of all railroad track in the country. This wasn’t just small speculative companies. The Union Pacific failed in October 1893 (again lol), the Northern Pacific failed the same month. The Santa Fe entered receviership in December, and the Erie failed again (for the fourth time)—the Reading Failed, etc, etc. You get the point, but this contraction wasn’t quite as bad as 1873, just in absolute dollars, much larger.

Only Hill’s Great Northern (with minimal leverage) survived intact.

President Cleveland, convinced that the Sherman Silver Purchase Act was killing confidence in the dollar, called a special session and repealed the act in late 1893. Gold continued to flood out, and in 1895, Cleveland had to turn to J.P. Morgan to organize a private syndicate to lend the Treasury $65 million in gold to prevent the US from leaving the gold standard entirely. The federal government needed a bailout from a private banker.

Silver Radicalized, and the Cross of Gold

Ironically, the opposition (farmers) were enraged even further. The deflation crushed miners and farmers, and Cleveland’s actions meant that they were left out in the cold. By 1896, silver supporters had seized control of the Democratic Party. And at the Chicago national convention, William Jennings Bryan’s “Cross of Gold” address electrified delegates. He won the democratic ticket.

“You shall not press down upon the brow of labor this crown of thorns. You shall not crucify mankind upon a cross of gold.”

Bryan's Democratic nomination led him to ally with the populist party, which consisted of western farmers, silver miners, and urban workers, against the powerful “money” interests of the East. The 1896 election was a referendum on the gold standard and, to some extent, on the Gilded Age itself. Shares were mixed in this entire period.

But Bryan lost, William McKinley carried the day, and was backed heavily by private interests. McKinley’s strategy of sound money and high tariffs prevailed decisively, and the Gold Standard Act of 1900 committed the United States to the gold standard. But let’s bring this all the way back to Railroads, because this crisis broke the financial system that supported railroads.

The wild speculative capitalism of the 1880s, with European bondholders financing American speculation with zero oversight, was done. Enter the era of concentrated banker control, also known as “Morganization”.

The Morgan Reorganizations

After saving the government, John Pierpont Morgan stepped into the wreckage with a checkbook in hand.

He had a standard template. Foreclose on the existing capital structure, create a new company with less debt, and raise capital for the new owners to reorganize the company. Morgan then installed Morgan-selected management with Morgan-selected partners on the board. These boards often had interlocking ownership structures, including among competitors.

The promise was stability. If you reorganized your railroad with Morgan, you wouldn’t have a rate war with other Morgan railroads. The banker became a soft-power regulator of an industry that couldn’t control itself through price cuts.

Between 1893 and 1898, Morgan reorganized the Southern Railway, the Erie, the Reading, and the Northern Pacific. Other banks followed suit, with Kuhn Loeb reorganizing the Union Specific, Speyer, and the B&O. By the end of the decade, Wall Street had restructured and controlled most major American railroads.

The reorganizations worked, and with improved capital structures and diminished pricing pressure, railroads emerged stronger from the wreckage. They did, however, also leave much more concentrated. The same industry that had fueled the excess also reigned it in, and now Morgan owned the railroads.

The Northern Pacific Corner (1901)

Now for the final story before we wrap up. I would like to discuss the Northern Pacific Corner. Harriman emerged during this period by purchasing the bankrupt Union Pacific and quickly became one of the wealthiest men alive. After reinvesting in the railroad and then acquiring other competitors, he created the first conglomerate system.

Meanwhile, James Hill had been building the entire time prudently. Hill went after the Burlington to connect Chicago and Seattle, and at the time Harriman watched in alarm. This was a real competition for Harriman’s system. Harriman sought a stake in the Burlington, but Hill refused; instead, Harriman pursued the parent company.

Harriman quietly began buying Northern Pacific stock. By May 6th he thought he had enough shares to buy the entire company, and lastly wanted to buy preferred shares to wrestle control. Harriman’s banker (Jacob Schiff), however, didn’t execute the order (talk about all-time poor order execution), and by Monday, the corner was public news. Hill had learned of the news and phoned Morgan, who was in Europe, and cabled back to buy 150,000 shares of Northern Pacific common at any price.

This was a time before retained earnings, so most common shares traded at par, and appreciation was reflected in dividend payments rather than in earnings. Northern Pacific shares shot up from $100 in April to $147.50 on May 6th. On May 7th, $180; on May 8th, $280; and on May 9th, Northern Pacific reached $1,000 per share.

Short sellers bet against the price rise, creating the most epic short squeeze of the time. And since the titans of industry were buying at any price for control, they drove share prices higher. Harriman and Hill purchased more stock than actually outstanding, as shorts sold shares they didn’t own or could not deliver. Now the problem was to cover their positions shorts had to sell everything else. Stocks fell 20-50% in a single day as the liquidity black hole caused the Panic of 1901. All for control over a railroad and the resulting short squeeze.

Morgan and Schiff came to a truce. The shorts were permitted to settle their positions at $150 rather than face ruin, and the battle for control was inconclusive. So instead, a new holding company was created, the Northern Securities Company, which would own both the Northern Pacific and the Great Northern. Hill and Morgan were in control, but Harriman had a meaningful stake. The railroad wars (and the last big consolidation wave) were over.

Anti-Trust and the Long Decline

Now, given this is an economic history, I have to briefly touch on the decline. Roosevelt hated the railroad wars, and the attorney general filed suit under the Sherman Antitrust act. Railroads at the time were confident, as the administration before was extremely pro-business. But in 1904, the Supreme Court ruled 5-4 that Northern Securities violated the Sherman Act and must be dissolved.

Shares were distributed, and Harriman and Hill cooperated, but this would be near the zenith of railroads anyway. Harriman died in 1909, and Hill died in 1916; although they had incredibly organized the railroad markets, the end was beginning.

The peak in total miles of railroads was in 1910, with about 240,000 miles of track and 1.7 million workers. They were the largest enterprises the world had yet seen, consuming vast amounts of capital and employing ~2% of the United States population.

But at their zenith, their disruption was in the wings. Automobiles started to compete with passenger traffic, and trucks began stealing short-haul freight. The Panama Canal diverted transcontinental shipping, lowering prices, and the Interstate Commerce Commission began regulating rates in a meaningful way.

World War I was the catalyst that pushed the system over, and in December of 1917, as railroads failed to coordinate logistics, the federal government nationalized the whole industry. For 26 months, the railroads were operated by a single entity, and when they remerged, they would never be quite as dominant as they once were. Governments regulated rates, and a fair return on investment was established.

Total mileage peaked in 1916 and subsequently declined. The Penn Central failure in 1970 was the largest corporate bankruptcy in American history to that point. Today, railroads have roughly ~140,000 miles of track, down 40% from the peak. The whole cycle of innovation, speculation, overbuild, crash, consolidation, regulation, and decline had taken approximately 60 years.

The Capital Cycle Lessons, Then and Now

So now that we’ve done the whirlwind tour of railroads, what the hell does this have to do with semiconductors and AI? Well, of course, an infrastructure buildout! I think as time has gone on and as the benefits have become “clearer”, the infrastructure buildout that I am starting to resonate more with is the railroad bubble/capital cycle. Let’s discuss why, and then also some broader overlaps between the AI buildout today and railroads.

As a sidebar, I will not cover everything. I wrote some about this here, and will not try to duplicate work.

I will try to focus on how the railroad cycle is “different” more so than repeating previous lessons.

Government Involvement

First, I think we are approaching the aggregate ~1 trillion in investment that the telecom bubble had in real terms from 1996 to 2000. It seems likely that the aggregate capex will far outpace the telecom bubble in real terms, and that makes me look back further for analogies. I think another significant difference between the Telecom buildout and the Railroad buildout is that the government was highly supportive and integral to the whole buildout.

While you can argue that the telecom deregulation bill signed by Bill Clinton helped spur the buildout, I think changing the industry structure for favorability versus actively pouring fuel on the fire, à la land grants or launching the “Manhattan project for AI,” is fundamentally different. The administration and corporate interests are way more aligned in this buildout, and in my opinion, it is a closer overlap with the railroad cycle than the telecom cycle.

Demand Chases Supply, Price Cuts Follow

I think the other takeaway here is that price is going to be the ultimate driver of demand, and in this case, price cuts are the most important metric to follow. We should expect some price erosion for models, but massive price erosion for models will be a signal that supply is massively outpacing demand. Token price must go down, but if price wars break out, we are probably overbuilt.

I believe, at my core, that supply is being built for future demand, which exists, but in every capital cycle, supply comes first.

Financing was Integral to Any Capital Cycle

I am starting to view the reflexive interlock between new capital cycles and capital itself as a key emergent feature. It’s no surprise that a rate-hike cycle coincided with the end of the telecom bubble. Greenspan’s hawkishness really kicked off the end, and the money supply often seized up, as the knock-on effects of the industry slowdown set in.

I sought to highlight how this reflexive connection manifested repeatedly in the railroad cycle. The 1873 cycle ended in part because of capital constraints elsewhere, which necessitated that American markets shore up liquidity. The 1890-93 cycle was similar, with British capital seizing up via contagion, amplified by gold flowing out of America, leading to an eventual implosion. The Panic of 1901 literally was a short squeeze. As we are on the precipice of a very large wave of lending, I also have to ask myself, is capitalism itself ready for it? More thoughts behind a paywall.

Metrics to Track

Now this is something only SemiAnalysis does, but I believe the aggregate fleet is an amazing metric to track. I also think the aggregate amount of capital spent is another super important metric. But last and not least is capacity utilization. When the railroad cars are empty, it is over.

Additionally, I think that railroads made me appreciate how closely linked capital and technology are. How much aggregate debt is now the focus, and the ability to service it, are major true goals of this cycle. Sadly, the railroad bubble was so overbuilt that it doesn’t even make sense from a modern perspective. All the railroads were built on speculation alone!

Anyways, for some final and very premium thoughts, I thought I’d leave where exactly are we in the cycle for last, because this exercise was enjoyable for me in terms of aggregate analysis.

Railroads and Today

Read more

Doug: Thank you, Will, for being here. We’re going to have a conversation with Will Eatherton of Cisco. I’ll dive into your background, talk about Cisco the historical networking giant, and then Cisco’s AI opportunity today. First, Will — tell me about yourself.

Will Eatherton: Great, Doug. Thanks for having me on. I’ve enjoyed browsing the site and its articles, so I’m happy to be here. I run Cisco networking engineering, focused on hyperscale engagements, datacenters, and service providers, which overlap a lot. I’ve been in the networking industry for about 25 years and have been acquired into Cisco twice.

The first was around 2000, when I was at a startup working on multi-terabit ASICs — the first big wave when everyone thought they’d need multi-chassis switches to move terabits around. That work went into Cisco’s carrier switch. I was at Cisco for ten years and later was a VP at Juniper. The second startup was acquired by Cisco in 2018. The last seven years around AI have been an exciting ride. Most recently, I’m proud of the NVIDIA engagement; I’ve been the point person on the negotiations and integration, which hasn’t always been easy between Cisco and NVIDIA.

Doug: Yeah, especially considering they have Mellanox. Why work together? Cisco was the 800-pound gorilla in the 2000s, and over the last decade-plus merchant silicon and other providers entered the space while Cisco leaned on legacy assets. Now with hyperscalers and AI, Cisco’s opportunity is open again. How is Cisco entering the AI datacenter space?

Will Eatherton: To touch on history, there were glory days in the 2000s and 2010s — the telco and enterprise build-outs were largely Cisco. I left in 2010. The early cloud era was rough for Cisco; we acknowledged some arrogance and we missed the early cloud phase. I returned in 2018, and over the last several years we’ve gone all-in on hyperscale engagements. That dovetails with work on SONiC, optics, and the Acacia acquisition — a corrective pivot for a large company. We may not be as visible as we want yet, but we’re positioned to have impact and we’re taking moves — like the NVIDIA partnership — to address a broader customer base.

Doug: You have real revenue — a billion in AI revenue and guidance that order volume could double next year. Walk me through the prongs: hyperscaler work, NVIDIA, Cisco One. What are the opportunities today?

Will Eatherton: We set a $1B target for FY25; Chuck announced we exceeded it, and we’re guiding higher. FY25 (ended in August) was driven largely by hyperscale AI infrastructure. We explicitly separated traditional data-center revenue from that number.

The key components were GPU-to-GPU interconnects where our boxes were deployed. Fixed boxes are the primary focus across the industry, but hyperscalers also want variation — modular chassis for spines that increase port counts and can reduce network levels.

High radix can be achieved in two ways: at the chip level (higher-rate ASICs) and at the system level with modular boxes. We early on offered a high-rate product; today vendors (NVIDIA, Broadcom, Cisco) are converging around similar radices — roughly 512 up toward 1,000. We and our competitors also support modular systems whose network OS lets you treat all line cards as a single system rather than many fixed boxes. The critical part is the software that presents that single-system abstraction.

Building these modular systems is getting harder as certified speeds increase and internal interconnect complexity grows — similar to server scale-up challenges. But comparable port counts across vendors allow customers to simplify their networks by gaining large radix in modular boxes.

Doug: Cisco is historically known for its network OS. Hyperscalers moved off vendor OSes in the late 2010s (whitebox/bluebox + SONiC). As new silicon like Tomahawk-6 and drivers arrive, vendor OS layers persist. How do you compete long-term when hyperscalers think they can build their own software?

Will Eatherton: The network OS builds on three blocks: the silicon in the box, the optics, and the system design. On the OS side:

We have multiple mature offerings. NX-OS (Nexus) is our traditional data-center OS with rich L2 features and deployment at scale. IOS-XR is our carrier/telco OS; when you move beyond leaf/spine into DCI and WAN that connect users to GPUs across sites, you need richer routing and features — that’s where IOS-XR is used.

Wide-area DCI architectures — enabling workload migration and GPU connectivity across datacenters — are increasingly important. Cisco participated in an OCP presentation on multi-data-center designs, internet pairing, and routing. Technologies like segment routing and IPv6 help manage WAN scale and complexity.

On the front end, some customers with very large GPU fleets (100k+ GPUs) want NX-style features at the front while keeping the same silicon (e.g., 150T silicon) for backend roles — that creates heterogeneous, proprietary mixes.

For whitebox/bluebox approaches — SONiC, FBOSS, etc. — we’ve taken them seriously. We invested in interoperability and support rather than ignoring the trend. After Microsoft and NVIDIA, Cisco is one of the largest contributors to SONiC — we contribute a significant portion of the code. We’re working on adding modular support to SONiC so those capabilities can be generally available.

Doug: That’s basically the opposite of what Cisco historically stood for — cutting off your arms, right?

Will Eatherton: Historically, yes. But infrastructures are upgrading every 12–18 months and hyperscalers move fast. We support SONiC and have gone all-in. For example, with our Silicon One family — the P200 we recently announced — we use an internal “NAS kit” approach: on day one we bring silicon up across IOS-XR (telco/WAN), NX-OS (datacenter), FV-OS (partner environments), and SONiC (customers like Microsoft and Google). Engineering supports all those stacks.

That approach helps us win hyperscale business and lets us add differentiating technology into SONiC. With 25 years of modular experience, we’re applying that know-how to SONiC. The value proposition has shifted — customers expect silicon, optics, software integration, and manageability. The CLI is no longer the primary control point; customers want richer software integration and manageability options.

Doug: For listeners who don’t know: what’s the difference between modular and fixed?

Will Eatherton: A fixed box is a single switch ASIC where all interfaces go to external ports. A modular system uses fabric ASICs and separates line cards from fabric — that’s been around for decades. You have line cards with local CPUs and software for management, plus route processors that coordinate across line cards, forming a hierarchical software system.

Some vendors implement modular designs by running independent network OS instances per chip, which forces you to manage many independent switches — that’s ugly. The modular approach lets you present the entire box as one large switch from software, routing, manageability, and telemetry perspectives. That’s far more scalable for large customers.

Doug: Perfect. Let’s circle back to DCI/WAN and the P200. Telecom still dominates DCI; how does Cisco plan to play there versus Acacia/Sienna in long-haul optics?

Will Eatherton: Let’s start with optics. We acquired Acacia in 2019. That foresight is similar in spirit (if not magnitude) to NVIDIA’s Mellanox purchase. The deal took time and involved financial renegotiation; inside Cisco it quickly became clear the IP was worth it.

Since then, pluggables and routed optical networking have grown. Our routed-optical approach minimizes transponders and OTN, and is Acacia-centric. We’ve integrated that into IOS-XR for better OS support and manageability. Year-over-year growth in optics has been strong.

A non-trivial fraction of AI budgets goes into optics. Hyperscalers ask about supply capacity and quarterly maximums. We’ve focused on 400G and 800G pluggables and are looking ahead to CPO. Optics is a base layer of spend.

On DCI beyond GPU-to-GPU interconnect inside the datacenter, we’ve worked for years with large networks (Google, etc.) using IOS-XR. Cisco’s strategy is to combine optics, silicon (like the P200), and OS capabilities to address DCI/WAN via end-to-end integration and manageability rather than only discrete boxes.

Doug: You mentioned XGS — is that compensating for not having deep buffers? Are there partnership opportunities?

Will Eatherton: XGS can use small buffer chips and NIX-to-NIX algorithms to help compensate for shallower buffers. We’re exploring partnership opportunities to incorporate XGS algorithms. But beyond XGS, the problem has multiple parts: core bandwidth interconnect, flexible load balancing and failure management, and larger routing/control tables. As systems scale, larger routing tables and control features become important.

We’ve been pushing segment routing and IPv6 — technologies Cisco pioneered. For stacks like SONiC, which have a minimal routing protocol set, segment routing offers traffic engineering without requiring a full set of complex routing protocols. The P200 is designed for deep buffers and the ability to use them, plus more features and bigger tables to support WAN/DCI use cases — that will be a key component.

Doug: Thank you.

Will Eatherton: WAN is an engagement point. It’s a lot of work — often five times the effort for a smaller revenue share — but it’s sticky. Deploying WAN/DCI builds trust with large customers. Over the last five to seven years we’ve focused on moving fast, supporting custom work, and shedding the “arrogant Cisco” image.

Diversity of options is another point. Silicon One gives a common architecture we can leverage: G-series for spines, other variants for storage and L2 roles, and the P200. That family of chips offers customers choice — important when Broadcom has been the hyperscaler play for years.

We signed an extended multi-year agreement with NVIDIA so our silicon can interoperate in the NVIDIA ecosystem, enabling customers not to require completely different architectures for different vendors. SpectrumX licensing lets our silicon interoperate with NVIDIA NIX and support adaptive routing with fabric telemetry — whether the fabric uses Broadcom, Cisco silicon, or NVIDIA — enabling end-to-end architectures and supply-chain flexibility.

Doug: Why did NVIDIA partner with Cisco?

Will Eatherton: Initially, enterprise was the focus — bringing joint architectures into enterprise deployments. Over time it extended to NeoClouds. NVIDIA’s Cloud Partner Program is prescriptive and regiments configuration, and some NeoClouds want our network OS on top. We’ve had customers ask whether Cumulus could run on Cisco silicon for optionality; I’m exploring that. It gets messy because of cross-product integrations, but the underlying theme is choice and a common architecture across vendors.

Doug: Ethernet protocol wars — UEC and UA-Link for scale-up. Where does Cisco stand?

Will Eatherton: Much of NVIDIA’s stack is proprietary and runs parallel to standards. We fully support UEC and were early to do so, but I have reservations. There have been false starts and shifting definitions — for example, how big in-network compute should be, and whether features are scale-up versus scale-out. We back standards work and participate in OCP, UA-Link, and UEC, but adoption and real impact will likely take longer than many expect.

History shows these transitions take years. There’s time for standard maturation; we’ll participate, but broad impact is a multiyear story.

Doug: Anything people should pay more attention to in networking?

Will Eatherton: Manageability. Hyperscalers do their own thing; sovereign clouds follow prescriptive architectures and fear deviation; high-end enterprises want manageable systems without endless bespoke engineering. Manageability in AI infrastructure is underappreciated but will become crucial.

We’ve invested heavily in controllers and management consoles: Nexus Dashboard and a new product launching soon target cluster-level management — GPU-to-GPU meshes, latency visibility, debugging, and wiring. A 100-GPU cluster can have a thousand fiber connections. We’re building “hyper fabric AI,” a cluster management view for front-end, back-end, management, and storage networks — a single console that pulls together networking and cluster telemetry. That’s a route into sovereign clouds and high-end enterprise.

Doug: So the entry point is manageability — not just being the No. 3 supplier, but using enterprise relationships to win medium-sized GPU clusters. That’s the strategy and the hope?

Will Eatherton: Yes. We add value there. Ideally GPU capacity would be more distributed, not concentrated. I just finished a customer advisory board with about 100 top enterprise customers; they don’t want to manage high-performance compute themselves. They want simplified management. We’re focused on making the experience better and simpler for customers running clusters from ~100 GPUs up to thousands. We see hundreds of millions of spend across compute and networking in the high-end enterprise market, and we believe we can lead there.

Doug: Anything else? Thanks for joining — I learned a lot. I pushed on Cisco, but I want to see more competition.

Will Eatherton: That’s our goal. We want Cisco to be strong again. For people who worked with Cisco 15–20 years ago, it’s a different company now; we’re moving faster.

Will Eatherton (cont’d): This space keeps changing every few months. Scale-out networking and scale-up approaches will evolve. We’re working with partners on transitions — fighting today’s battles while planning the next phase, which will look different from the last 20–30 years. It’s an exciting time.

Doug: Networking may feel like it’s becoming more integrated — more like PCB traces or an integrated rack — which will make serviceability harder. What about CPO? Peers are pushing first-gen CPO chips; that will change serviceability and be a big shift. How is Cisco positioning for CPO?

Will Eatherton: We believe CPO (co-packaged optics) will drive a major shift. We have announcements coming, and we see this as a move from a chip play to a systems play — that’s Cisco’s wheelhouse. Serviceability, software integration, and failure handling are things we can integrate across our network OSes and system architecture. We’ll continue to evaluate the NVIDIA partnership and decide how to engage; we’ll have multiple approaches. For example, a Spectrum silicon switch might start with an NVIDIA CPU on the switch and later move to a Cisco CPU — and we’ll continue to offer our own silicon. The goal is to provide options and support customers where they want us.

Doug: Silicon photonics has always been “five years away.” Is it nearer now?

Will Eatherton: I wouldn’t be surprised by six- to twelve-month hiccups, but I think we’re under five years now. It’s finally getting close.

Doug: Agreed. Thanks so much for your time, Will. Any final parting words?

Will Eatherton: No, that’s good. Thanks for having me.

Doug: Great. Thanks, Will. Take care.

Will Eatherton: Bye.

Read more

This was the largest one-day move in any stock’s history by absolute value. Larry Ellison also became the wealthiest person alive in the span of a trading day. The opening line of Oracle’s guidance explains why:

• Q1 Remaining Performance Obligations reached $455 billion, up 359% in both USD and constant currency.

Pause and digest that number. Denmark’s GDP is about $450 billion. It’s a number so big I first thought it was a typo. It is almost unbelievable for a company to disclose hundreds of billions in future revenue; it is even rarer to offer four-year guidance. But Oracle just did:

We now expect Oracle Cloud Infrastructure will grow 77% to $18 billion this fiscal year and then increase to $32 billion, $73 billion, $114 billion and $144 billion over the following 4 years. Much of this revenue is already booked in our $455 billion RPO number, and we are off to a fantastic start this year.

For context, Microsoft’s Azure is forecast to generate approximately $75 billion this year, while AWS is expected to generate around $126 billion, and Google Cloud is projected to generate about $56 billion. Oracle was not even considered a top-tier option until recently. If its trajectory holds, it could surpass Google and perhaps even Azure within a few years. Look what it did to our year-end estimates. The street consensus shifted from approximately $95 billion to approximately $118 billion after the guide.

This is what disruption looks like. I have argued before that the hyperscalers’ old moats, balance sheets, and CPU-centric distribution were less impregnable than they appeared. SemiAnalysis’s ClusterMAX benchmarks confirmed this, and it is now a genuine contender from Oracle. Capital as a moat is not a lasting one.

History suggests that incumbents rarely dominate tangential markets. A focused challenger emerges, captures share, and reshapes the landscape. AWS played that role a decade ago; Oracle may be playing a similar role now. Amazon’s in-house Trainium effort increasingly resembles an “Amazon Basics” strategy, viable but not leading, while Oracle doubles down on the largest GPU-first cloud in the world.

The driver of this sudden leap is clear: OpenAI. Microsoft has slowed its spending; Oracle has filled the gap, becoming OpenAI’s primary infrastructure partner. That single customer dominates Oracle’s RPO. And if Oracle leverages its entrenched database franchise, it could become enterprises’ default inference partner as well. In one fell swoop, they are now the addressable inference partner for the majority of enterprises globally.

It is worth recalling that Oracle’s database remains the most widely used globally—still the gold standard for ERP, CRM, and data warehousing. PostgreSQL and MongoDB may be fashionable, but feature parity remains elusive. On DB-rankings.com, Oracle remains in first place. A reminder is that this chart is logarithmic, so the gap between competitors appears wider than it actually is. Oracle actually is the most dominant database in the world.

Now Oracle is pushing vectorized databases, paired with inference capacity, directly into enterprises. It is a compelling combination: entrenched incumbency in data, plus the most significant player in GPU infrastructure. To Larry Ellison, this is how you win the future's biggest enterprise in the world (OpenAI) while servicing decades of data from your historical dominance. However, all this spending is going to be quite costly. Which brings me to my next point: The Capital Cycle. Because this must be funded with something, and Oracle cannot pay for this with cash flow alone.

Capital Cycles and Debt

There is no way for Oracle to pay for this with cash flow. They must raise equity or debt to fund their ambitions. Until now, the AI infrastructure boom has been almost entirely self-funded by the cash flows of a select few hyperscalers. Oracle has broken the pattern. It is willing to leverage up to hundreds of billions to seize a share. The stable oligopoly is cracking. I wrote about this earlier, and I think that we are starting to see the markers of a classical bubble.

The implications are profound. Amazon, Microsoft, and Google can no longer treat AI infrastructure as a discretionary investment. They must defend their turf. What had been a disciplined, cash-flow-funded race may now turn into a debt-fuelled arms race. That’s a vibe-shift if I’ve ever seen one.

Welcome to the next and newest phase of the capital cycle. I believe the Oracle quarter achieved something more elusive than just revenue. I believe Oracle has just sparked the elusive animal spirit to life.

For more thoughts, please subscribe.

Read more

Today’s post mixes free and paid content. First up: Texas Instruments. TI’s quarter was rough—optimism that the analog cycle had finally turned fizzled when management said it would cut inventory into the supposed “upturn.”

As I alluded to earlier, given the inventory position that we have right now and depending on what happens on revenue next year, we may need to adjust factory loading down in the coming quarters for a few quarters until we rectify the inventory situation a bit and just stop it from growing and maybe drain it -- and not maybe likely drain it for a few quarters. So that would put a headwind on gross margins, but it should be temporary for a few quarters until we rectify the inventory situation.

They’re now guiding to the low end of capex, signaling the COVID-era grand capex plan may never be needed. Texas Instruments also guided another disappointing quarter; extending the lagging-edge slow-motion train wreck.

This is the tail of the long auto/industrial cycle: a huge upswing followed by a slow bleed. The glut has run roughly twice as long as a typical downturn, and even the “green shoots” were tariff-distorted.

April’s pop was mostly pull-forward; that tariff-driven pull-forward was taken as fake demand.

Those were pretty strong January through April. And April, in particular, were really strong. And we think some of that was due to the Liberation Day and some of the dynamics that happened there. But then things did slow down after April, or at least didn't grow as they normally would have a month on month on month. And again, some of that was the Liberation Day potential pull-ins. And we talked about that at some length at our July earnings release call.

And since then, it’s been slowly bleeding out in terms of the pull-ins. The run rate of demand is actually lower.

My sense is not just with TI, but the industry as a whole, the tone was a little different in April and May than it was in July and August. And maybe nothing quantified, but the tone. And I think that was a function of those orders the order trends that I referred to earlier that April was pretty strong. And -- but it appears that some of that was pulling Again, it's hard to tell with so many permutations on orders and customers, but it looks like some of that was pull-ins.

There is no lagging-edge upcycle. Global capacity looks bloated, especially in categories competing with China’s lower-end offerings, where TI sits. Supply and demand here are fragile to small changes; the swing factor is incremental Chinese demand.

Even a 5% supply uptick in an “upswing” would sap incumbents’ pricing power and impair the value of their capacity. With utilization still lackluster globally, the writing is on the wall.

The Auto and Industrial inventory cycle is as bad as it has ever been. Stocks are waking up to a structurally inflated low end driven by China. I wrote about the edge price war in 2023, and it has been a one-way slog since.

Lagging-edge capacity now behaves differently. Another upturn may come, but it should be muted and prolonged. Something is broken in auto and industrial. TI is right to run inventories down; they likely will not need as much.

Lagging Edge Perma Glut

What no one wants to admit: lagging-edge capacity may be structurally different now. Earlier, I argued prices had to rise because a new fab cannot profit from selling a lagging-edge chip at prices set by a fully depreciated fab.

There are obvious pricing problems in this scenario. With a new fab, a company can’t turn a profit selling a lagging-edge chip at the price that was previously dictated by a fully depreciated fab. Prices have to go up.

Reality: the lagging edge took price in a temporary market upturn; now we are reverting to the mean. The old trend line is back. There is no meaningful new demand for old nodes, and what exists is made and consumed in China. The COVID super-cycle, plus a new China supply line that does not care about capacity economics, points to a forever glut.

China has a long record of flooding the low end. State-directed capex boosts output at the expense of efficiency. I wrote about this “involution” dynamic recently.

Products at 90 nm and above should internalize a permanent China glut. Companies resist that conclusion, but the case studies are familiar: solar modules, LCDs, LEDs, steel, shipbuilding, batteries.

Companies can still win share, but through performance and packaging rather than by producing older chips at lower cost. ADI is a master class in adding capabilities; MPWR has focused on higher-end power.

The long tail will face a flood of Chinese supply. Good luck to the older fabs.

I cannot resist talking about Marvell behind the paywall.

Read more

If you missed the news, there is a rumor that the Trump administration could be taking a stake in Intel. This feels like a long time coming, but the incompetent board is still fighting. For more work on the board, read both mine and SemiAnalysis’s pieces.

The final stakeholder is here, and Intel needs to do a better job of capturing its attention and help. Let’s walk through a brief recap of how we got here and where Intel could go from here.

Intel is a Walking Death

Intel was once the greatest semiconductor that ever was. And I cannot express how true this was. Nvidia is the shining star today, but it didn’t invent as many foundational technologies as Intel did. Intel’s story is THE story of Semiconductors. From the Traitorous Eight leaving Shockley’s lab, to Bob Noyce leaving Fairchild to then found Intel, the history of Semiconductors IS the history of Intel. Intel invented the concept of the Microprocessor, a CPU itself, and made the world’s first commercial RAM, among other innovations such as strained silicon, high-k metal, and FinFETs.

And it’s no surprise that the future of American semiconductors has Intel written all over it. But there’s no other way than forward, and I think it’s time to consider what needs to happen realistically, and that’s the death of the Intel we once knew to make room for what’s next. The key is that while CPUs don’t matter, the only American leading-edge foundry left making them is critical.

The problem is that the company that funds it might run out of money, and that’s why they need to publicly threaten to stop financing the future of the foundry, because it’s a problem they can’t do alone. That is why I believe they so publicly announced the ending of future nodes past 14A. If you want us to continue, we need help.

It’s time to threaten the future of America to the one stakeholder that matters: The Government.

Intel’s Foundry Must Live, at Any Cost

The calculus for America is pretty simple. In my view, there is very little strategic importance to the Intel CPU business. The x86 ecosystem was once the most incredible compute ecosystem, but AMD designs better chips than Intel could; Intel has the one thing that AMD does not, a Fab. The fabless business at Intel has a real issue in that making a CPU is becoming a relatively commoditized business. ARM has made it possible for almost any hyperscaler to have its ARM-based CPU, while AMD continues to outdesign Intel at its core job, and that’s not even discussing the longer-term RISC-V ecosystem.

Adding up the CPU side, I see a business with massive competition and Intel not at the top of the stack. Intel has to deal with increasing competition in’s core profit center while at the same time covering the increasingly heavy burden of a leading-edge fab. There is only one leading-edge foundry (TSMC), and a second American option is the single highest value-added project of all time.

If the DOD can become the largest shareholder in MP, a potential hedge for rare earth minerals, the government should take a stake in Intel. Unlike rare earth, which is an exercise of moving cost curves down in a very heavy-duty industry, this is technology that requires real IP and not moving more earth out of the ground. A compelling framing is replacement cost, and the US would have to pay like 200 billion or more to “replace” the Intel we have today. Throwing money to sustain it makes sense for the US.

What Would Intel’s Foundry Look Like as a Standalone

Here’s the core take. We cannot rely on Taiwan for the future of semiconductors. The more capacity we get from TSMC, the more we remain reliant on R&D in Taiwan rather than the US. Intel must be standalone and must have the capabilities to do the two things the US critically needs. High-end logic and military capabilities. I’d argue the second is met chiefly, but the first Intel is hopelessly behind.

What’s worse is that Intel has a bad customer, itself. Intel needs a good customer to be the anchor, and sadly, the core customer is a CPU company that is struggling to find its way in an accelerated compute world. So the core problem is that the funder of the large domestic capacity is a horrible customer, and the bill itself is extremely steep to pay. A stake by the government doesn’t sound like such a bad thing if it is a public asset. The problem once again is incentives, and how the hell does that even fix Foundry, anyway?

Trump’s Superpower: Bullying Partnerships

The solution almost seems obvious. Intel Foundry has many critical issues with competitive products, technology that customers want to use, and a customer experience that makes it possible. Nvidia, Apple, Broadcom, AMD, and Qualcomm, the Fabless 5 of America, all struggle with this chicken-and-egg problem. Will they be around?

I think that LBT knows it’s critical, Trump knows it’s critical, and Trump does have quite a superpower. If it means you do not pay tariffs, why not throw your orders towards a large, partially government-owned entity called IFS? If there is a huge stake, this could be considered compliant with investing in America, and the top 5 have hundreds of billions of dollars of orders. Seriously considering the process in exchange for paying less tariffs is a straightforward deal, and one that Trump is amazing at making.

What’s more, the rumors of a hyperscaler investment to me seem like another “investment in America” opportunity. That’s been the clear pattern: commit to invest more in America, and we will let you keep overseas revenue streams. Trump should and could do it - and Intel could have a path out for now.

It’s time to invest in the American (gold-plated) foundry of the future. I believe that large foundry partnerships will be announced as part of ongoing investments in America. In theory, Foundry needs orders to exist, and this is how they will get them.

Trump can bully Broadcom, Nvidia, Qualcomm, Apple, and AMD to put orders towards Intel, while possibly forcing Amazon, Microsoft, Google, and others to make a large investment in the fab itself (or push orders). Additionally, forcing semicap companies like KLAC, Applied Materials, and Lam Research to invest and give resources in exchange for approved licenses is another example of a carrot and a stick. I think Trump could forge the giant partnership to happen, but then execution is all up to Intel. And LBT is still once again qualified for the job.

Now, how does that impact the equity? Yeah, we will see about that.

Final (Paywalled) Thoughts

Read more

That is an obvious statement. But so what? I think today, I want to propose my theory on how I see the broader software ecosystem will be impacted by the influx of new supply from coding agents. Welcome to the Software disruption flood. We have a very recent case study at hand to consider how software will be impacted. Today, I want to compare the rise of coding agents to the rise of YouTube and the destruction of linear media. I think it is the single best frame to view how software will change from here. So let’s set the scene.

The Internet, YouTube, and the Bundle

Once upon a time, there was a paradigm of content in which for a single subscription, you could consume 1000s of channels of content in a bundle via TV. The anchor of this bundle was Live Sports (still is), and this was the last meta of Media consumption. This lasted for decades as TV was the killer application (over radio!) from the 1980s to the mid-2010s.

Now Media itself is just content, and it can be viewed on TV in the Bundle, over-the-top (like Netflix), or even now via short-form content on TikTok, Reels, and YouTube Shorts. The key inflection point that screwed the Bundle from it’s peak of ~87% penetration to ~40s penetration today.

This is the ugly peak of TV, and compared to the very glorious rise in penetration, it’s a painful sight to watch in real time.

Meanwhile, YouTube (my proxy of choice this time) blew up in conjunction. The timing is quite accurate, and you can see the beginning of the S-curve of YouTube almost perfectly align with the top of the cable bundle. In the 2010s, as YouTube experienced explosive growth, the cable bundle reached its peak and then began to decline.

This is what happens when your platform peaks, and I think that the Web 2.0 SaaS era bears a striking resemblance to this analogy. But before we get there, let’s talk about what killed the bundle. In hindsight, it’s pretty straightforward to attribute the issue broadly to the internet. However, I think that it wasn’t just the distribution (via the Internet) but also the media itself that changed, shifting from being concentrated in relatively few media makers (thousands of channels) to millions of individual creators. There are ~113.9 million channels on YouTube, with 32,300 with over 1 million subscribers.

For example, Mr Beast now has 400 million subscribers. That’s larger than the entire United States and much larger than Cable TV ever was. And it’s not like Mr Beast entered the ring with massive funding and a distribution channel. He raced to the top because the cost to enter the Media creation race dropped. This, to me, is the key part of the two-pronged success story of the millions of smaller creators that applies so directly to software today.

The Cost to Write Software is Fraction of the Previous Cost

I believe this is the critical insight here and why the Media disruption case makes a lot of sense compared to Software today. In 2000, it cost actual money to start a TV show. My extremely imprecise estimate is that it costs approximately $250,000 to launch a TV show in 2000, and now it costs roughly $3,000 to launch a YouTube channel.

This is likely similar to the cost-down trend that’s happening in coding assistants today. It used to cost $100s of dollars in man-hours to write ~100s of lines of code. Now it costs 100s of dollars a day to write millions of lines of code. The cost is now plummeting to enter, so I expect massive entrants everywhere.

Now, skeptics of this argument will contend that service, product market fit, and distribution (i.e., sales) are the significant differentiators. And unlike the two-pronged change in distribution and content costs for traditional media, it’s only the cost of creation that's decreasing, not the distribution costs that are changing. And frankly, I find this to be a very mediocre argument. Systems of Record are unlikely to be changed, but new business opportunities are available. And if context windows become infinite, what is the point of 1 specific solution over another if there is endless recallability and manipulation of the data and information within an LLM?

Software has always been a high-margin product, with gross margins of 90% or higher. Generative AI costs a lot, but for the creation of a product, this inherently lowers the net cost of a traditional software solution. So, if you make a lot of money, don’t have a clear differentiation from the 10s of other solutions in the market, what helps you grow? Why sales and marketing? B2B SaaS sales is a meme because it is the key differentiator between two similar products. And assuming that the bar is rising, the S&M spend “moat” feels like a race to the bottom. Now, everyone has a point solution that is ready to be sold, and this supply of software solutions will only ever go one way for the rest of our lives: Up massively.

Peak Linear TV will be an Accelerated Peak Software.

The era of making software for massive profits is over. The game used to be focused on hoarding a relatively small number of people who were able to generate the best code, thereby increasing your internal velocity and denying this supply to your competitors. Coding agents mean that the supply of software is going up exponentially, and I think that isn’t good for the entire industry.

Software feels a lot like traditional media, but will probably experience the rush of disruption much faster. Supply will flood traditional software makers, and niche solutions will eventually overwhelm the incumbents. Folks you’re looking at peak SaaS.

I think the timeline matches quite well. I believe that there is money to be made in the stocks for traditional TV. Still, it will come via a wave of consolidation as Microsoft, Salesforce, ServiceNow, and Adobe are the Disney’s and Foxes of tomorrow. The long tail is probably dead, and Cursor and other coding agents right now are adding as much revenue as the entire software space is this year. That’s your YouTube vs Bundle analogy, and I think it’s 2010. It worked out for a minute, but they have underperformed broad indexes since 2010.

Software was a Local Minimum Anyway

Now, this is something I wanted to discuss anyway. If you go back in the history of computing, most software was sold as a bundle to the user anyway. Initially, software was included with the machine you purchased. IBM’s original product came free with the hardware. This changed in 1969 when IBM unbundled its software and hardware solutions.

The concept of writing software and charging money for the IP is novel, but it was only valuable when there were barriers to entry to creating that software. This was the law of the land when hardware (PCs with Intel CPUs) were ubiquitous and commoditized, and now that era of hardware is over.

In a world of infinite software generation, what is the point of software anyway? It just is an output of the hardware. In a world of hardware generating software, I don’t believe there is much of a moat to be had. The fact that original devices, such as Teradyne’s ATEs, early CNCs, Switches, and Cray computers, were custom software solutions for their intended tasks, bundled into the devices, makes me think that we could return to that direction.

Software was just a local minimum on the actual march of progress, and that was hardware all along. I want to reiterate this: one of the most magical things we have ever done is to impose information using physics onto silicon chips. Everything else is downstream, and now we have created a generalizable model that can create, organize, and generate information from a sufficiently large number of hardware transistors. Software is likely to resemble a local minimum in the long term as the value shifts back to the hardware. And as Claude code and cursor increase the supply of software, we go back to the real scarcity, hardware.

Some further thoughts behind the paywall.

Read more

Doug: Welcome to Fabricated Knowledge. This is a podcast edition today, and I have the special honor of having Val Bercovici from Weka. And today, we wanted to specifically highlight an important trend I think everyone should be paying attention to. Disaggregated PD. I also wanted Val the opportunity to tell the story of Weka and AI-enabled storage today anyway. So take it away Val.

Val Bercovici: Thanks, Doug. So, Val Berkevich, Valentin, for those who did sometimes see my name online, a long-time infrastructure guy focused on storage for most of my career. I ended up being the CTO at NetApp about 10 years ago, following a cloud acquisition they made called SolidFire.

At that time, I was also actively involved with Google in open-sourcing their Kubernetes project. That was cool; it helped create and co-create the Cloud Native Compute Foundation (CNCF) under the Linux Foundation. I enjoyed category creation, and I was relatively early in the cloud, early in machine learning, and not so much in general AI, but I'm catching up on general AI right now. And yes, I'm at Weka right now, focusing as Chief AI Officer on our AI product strategy and marketing and education. Because, as we're going to talk about Doug, there are a lot of very new and different things happening about infrastructure when you look at AI workloads. It bears almost no resemblance to traditional CPU-based data centers, and that's what we're going to dive into right now.

Doug: Yeah, so maybe to start, I do think that the important thing is context, right? Things are changing quite rapidly all the time. Let's walk through how inference in a single GPU node was done and what has changed recently to shake up the entire ecosystem with this disaggregated pre-fill / decode. I think a listener needs to understand what pre-fill decode is on the traditional node, and then we can discuss what's changing by splitting the two.

Val Bercovici: Sure, and let's maybe introduce some overall financial context here and Semi-Analysis is leading this. But estimates are now that about four out of every five dollars of AI is gonna go to inference. So inference is not a trivial thing here; it's a very, very material thing in the future. One of the reasons why inference matters so much is that LLMs, in general, and large reasoning models, in particular, are concerned with this aspect.

LRMs fundamentally use inference-like techniques as part of their test-time computation, which is scaling now in the inference dimension far more than in the legacy pre-training dimension. So, all of these kinds of workloads matter a lot in terms of how infrastructure is allocated and spent, as well as the revenues collected from it. So specifically on Disag Prefill, which is a relatively new phenomenon. It's mostly a 2025 thing.

Some research papers obviously were written well before last year, and last year on it. And it's a way to scale the entire process of inference. And this is specifically to be technical, not for image models, which are classically or video models, known as diffusion generative AI models, but the more popular text-based, if you will, and video-based as well, transformer models.

So this is a crucial qualifier. But having said that, transformer models are really where a lot of the innovation is happening and new features are happening right now, and a lot of the focus is on engineering and efficiency.

So, Disagg Pre-Fill / Decode, to open up the topic, one of the simpler ways to start is to consider the following: if you're not necessarily a layperson, but you're not particularly AI savvy, and you've just been in tech for a while, think of zip files. Please consider the process of archiving data, then compressing and decompressing it. So a lot of AI discussions focus on encoding.

That is optimizing the training of models and getting that loss function down and getting this fit right between underfit and overfit. That's basically what it takes to compress the internet, as we like to say in the industry semantically, right? That's fundamentally what a lot of AI models are, compressing some fraction, or if you're OpenAI, most of the world's knowledge online into a multi-gigabyte file that gets loaded into these GPUs.

So the encoding part is the compressing part, and inference is all about the decoding. It's all about how you decompress that model and how you do that by serving the model hundreds and thousands of times across hundreds and thousands of GPUs and how you decompress that model for every user. So every time you and I sit down at a ChatGPT session, or more often now, how an agent consumes these AI models on your behalf if you're running Cursor or other popular AI apps, the Manus AI agent, for example, how that decoding is happening.

And that decoding now is really very different from the unzipping or unarchiving of old days, fundamentally because, if we were to use a zip example, that was a CPU and was traditionally a single core. So it was a single-threaded or single-instance process focused on compression. Therefore, it's a very serial process for compressing the data, and a similarly serial process for decompressing the data.

Lots of those bottlenecks are with regard to just a CPU serially waiting on memory and storage. GPUs, again, maybe the main takeaway for everyone here is that nothing happens in serial on GPUs. The whole point of GPUs, the reason we spend an obscene amount of money on GPUs in AI data centers today, is that everything wonderful happens in parallel. That's the only reason, now, Gen.AI in particular exists is that it's a very parallel process run in these GPU kernels.

And so the process of Encoding the data into a model and particularly as we're talking today the process of decoding the data from these models as we serve them at inference time is a very very parallel process and asking the way these models work asking the GPUs to do different things in parallel turns out to be incredibly inefficient.

So, let me give you a tangible example of that, and we'll dive into the details of why the reality of the market is the way it is today. When you go to OpenAI's pricing page, Google's pricing page, Anthropic, Together AI, and other popular model providers, you'll find that they have three classes of pricing.

One is the price per million input tokens. One is the price per million cached input tokens. And one is the price per output token. This caching tier, or rather, this caching class of pricing, also known as context caching or prefix caching, goes by multiple names and was introduced about six to nine months ago because people were discovering that it was very inefficient to reprocess the same data repeatedly.

If you can process cache data, it is much more efficient. And we've seen OpenAI now join the pricing wars, along with Google and DeepSeek, setting the stage for a race to the bottom. Some great reports that SemiAnalysis wrote, you know, going back even two years ago. There's kind of an industry standard of about a 75 % discount to the user or to the API caller, whether it's a developer or an agent, if you use a cached input pricing option. But there's a giant caveat.

That is, all of these providers, as they're inferring, will only give you about five to 15 minutes. They can't even specify beyond that, five to 15 minutes of prefix or content caching before you lose the context, you lose the cache. And so why is that?

Let's dive a little bit deeper into how this decoding and inferencing work. The way it works is that you build a working memory. So as you're inferencing a model, as you're decoding the model, you actually can't jump into the decoding part, right? That's the valuable part. That's where those reasoning tokens pop up.

Doug: I'll slightly interject here. I want to make sure that we have this, just because Val, you're going pretty quickly. It's a hard space. But specifically, let's discuss the prefill decode, right? Because we're talking about it, we're throwing these words around. On the prefill side, I think it's helpful to understand that that's compute-bound, and the decode side is memory-bound. And so traditionally, how this was done was all on one node, GPU was essentially, you know, this is a virtualization problem essentially. What we're talking about now is the shift to the understanding that the decode to in the, you know, the pre-fill, which is, I believe the input side of it versus the output tokens, the ratio is exploding so that you're talking 10 to 1, 20 to 1 of these tokens that are moving. And so what people are finding out is that decoding is becoming a big issue in inference scaling. And so, maybe that context is helpful, because I feel like we're diving in very quickly, and I wanted to make sure we...

Val Bercovici: I was jumping ahead to illustrate the fact that most people skip the prefill, right? They skip this fundamental critical path item in parallel with how inferencing works overall because we're used to paying for input and output. So what is this prefill thing? Prefill is literally how we prepare to do the decode. Decode and transform models can't happen without pre-fills. So, what is prefill?

It's a bit challenging to fathom if you're not in the machine learning or AI space, but it's utilizing a significant portion of the data that we have at inference time. So the data we want to process, which is supposed to represent that, but as we're doing inference, we ask, know.

Upload a complex legal PDF or upload terms of service for the stuff we sign up for every day, or upload some complex documentation, and start to ask questions about it, right? Specifically. So we're not asking about general world knowledge. We're not asking about mathematical or scientific knowledge. I want to know if I can get my insurance carrier actually to cover this particular claim. And it's a very long and complicated document. So that's a very good example of what it takes to do inference.

So, Prefill is taking your question about that document as well as the document itself. And after it's tokenized, after we convert the words, charts, and graphs into these AI tokens that we all use, we have to put context around these tokens. And we have to vectorize these tokens. And again, hard to fathom, but we add anywhere from 10 to 20,000 dimensions to each of these tokens. Because every word will have a different meaning based on its context, right? The C word here is everything context. So, Prefill is all about adding all these dimensions of context and creating these query key value KV matrices, which take kilobytes of these tokens and convert them to six orders of magnitude to gigabytes, so way past megabytes. We're approaching the gigabytes of working memory for this key-value cache, also known as a KV cache. has to be stored in memory.

It can't be stored because we're accessing it all the time. The only way that...

Doug: We also explained KV cache, right? Because I know we often use these phrases, I think it might be helpful to discuss them. KV being key value cache, right?

Val Bercovici: Yeah, these are multi-dimensional structures. So, the keys and the values themselves are multi-dimensional matrices, and they each consume, again, gigabytes of data as they're transformed from words and tokens, graphs and tokens, to ultimately tokens, the keys and values. And it is how the algorithms of inference work. These algorithms are often referred to as flash attention or simply attention algorithms. One of the most popular ones for Nvidia processors is Flash Attention.

And all of this ultimately involves matrix multiplication. And so that's why we've gone from these very, I think you know, fortunately timed graphics processors, which were always the matrix multiplications to render pixels on screen. We had these arithmetic logic units, ALUs, and these graphics processors, and we had thousands upon thousands of them. And NVIDIA, wisely recognizing that machine learning and Gen.AI utilize very large matrix multiplications. Now, they are adding more and more of these tensor cores, not just regular GPU cores, to their GPUs. These tensor cores specialize in performing large matrix multiplications at a more AI-optimized precision.

Ironically, and I'm not sure if we want to go down this rabbit hole, but AI doesn't require as much precision as traditional graphics or high-performance computing. So, lower precision, same results, basically, same general level of accuracy, but way faster processing. So the flash attention algorithms essentially are what makes AI work on ⁓ GPUs.

Doug: So let's specifically get back to the pre-fill deep, and like maybe I want to do some paraphrasing here. So, essentially, when you run an inference workload on a GPU, what happens is you load all the key values into the KV cache into memory. This memory is often HBM. The reason why HBM isn't so high in demand is that you want this big, as large as you can, essentially. Then pre-fill the weights. And then that's the compute-bound part. But then after that, you're just running the inference over and over and over. You submit a query, and then the model generates the results. So that's the decode portion. And that is much more memory bandwidth limited. I wanted to confirm that's the high-level summary. Is that correct?

Val Bercovici: Exactly. And this is very much a kid buying our laptops today, right? We always want to buy as much DRAM if we wish to or memory on our laptops as possible. The same rules always apply to servers for making databases run faster. Really critical to this process of inference with GPUs. And there's three key tiers of memory we want to discuss here, Doug. There's the SRAM, which is really where the algorithms do the actual work with these tensor cores. These precious tensor cores on the video GPUs do the matrix multiplication.

Then there's a high-bandwidth memory reference, which is essentially the working memory, where data is stored and retrieved from SRAM to facilitate the actual matrix multiplications. However, the challenge remains that we can never afford as much memory as we would like on our laptops or servers. It's the same challenge we see on GPUs. On the GPU package itself.

There are all the GPU cores and the tensor cores, and we package this high-bandwidth memory; the real estate is just very finite. We can only get, you know, one, maybe 200 gigabytes now per GPU of high bandwidth memory. And because the working memory of these models has to also contend with the weights of these models, by the time you load the model weights for DeepSeek, by the time you begin to create this key value cache as working memory for the very first user you're almost out of that high bandwidth memory, right?

Memory is just so precious right now, these models and the working memory are so large that there's this third tier of memory, which is the DRAM, regular dynamic RAM, DDR class five, you know, DDR five class memory right now. And that's a memory that's shared on the motherboard across all the GPUs, often eight GPUs on a single motherboard. That's the shared memory. And ⁓ that typically when you...

Doug: Also, and then for the next generation, because you know the listeners are often semiconductor investors, so they are clued in, right? In the next generation, the grace portion of the Grace Blackwell CPU controller is often connected to banks and banks of DRAM, so that DRAM essentially is the third tier of memory. And so that's where the third tier kind of rolls in

Val Bercovici: Grace Blackwell, yeah. That is certainly the case for GB 200s and even GHs, as well as Grace Hoppers, which have become a very standard computing unit in the world of AI training and AI inference.

And the software, which we haven't mentioned, the software, which is key here in terms of how inference happens, is now very aware of that. So, the software consists of these inference servers. Historically, it would be NVIDIA's Triton TRT-LLM. For the past year or so, we've seen overwhelming market momentum and interest in the open-source VLLM inference server, as well as its thriving community. And we also see SG-LANG and other inference server models.

We combine models and these inference servers into large kernels that we load into the GPUs and let them run. ⁓ so the software is now aware of these three tiers of memory. And there are these things called KV, Key Value Cache Managers, which manage the three tiers of working memory on behalf of these inference servers so that when you run out of high bandwidth memory, which is always, that you don't have to evict from KVCache.

You don't have to say after five to 15 minutes, I'm just out of cache. I have to go back and re-prefill all that data, give you that 30-second delay, consume hundreds and hundreds of thousands of watts, and start the KV cache process all over again.

Everyone's trying to minimize and avoid KV cache eviction as much as possible. We frequently observe KV transfers between GPUs and GPU servers to move memory around until it's exhausted, and then we have to re-prefill. One of the fundamental aspects here is that we want to minimize the time spent pre-filling by keeping everything in cache across an entire cluster of hundreds or thousands of GPU servers.

Doug: So this is a, think, okay, let me try to summarize this back, because this is dense stuff. We're talking about how inference is done at a core level, right? We're discussing all the aspects that go into loading the KV cache into memory, including having banks of it, trying not to get evicted, and the software having an understanding and being able to KB manage so it can dynamically address where the memory weights are being stored or held. And that's kind of, I think, the state of the art till today.

However, the reason I have Val on is not to explain it, but rather to explain how we infer at all on a GPU. It's the next step. The next step that I think is starting to become increasingly apparent. For instance, the foundational change is something called disaggregated, pre-fill, and decode.

So, we've been discussing pre-fill as the loading of the KV cache onto the GPU, and then decode as the serving of the model, essentially running a query and then activating all the model weights, which then tells you the result. But importantly, there has been a significant shift, where we're starting to disaggregate it, meaning that it no longer has to be done on a single GPU.

Because we have this KV manager, we can pretty much handle routing, and we could create a pooled access of resources to achieve better utilization of a GPU cluster. And that's disaggregated pre-filled decode at the highest level, but I wanted to give the mic over to Val to explain what, maybe in a little bit more depth, what exactly it is, why is it such a big deal, can you kind of even talk about the amortization of the KVCache, like how many users are being able to, the difference of being able to do one GPU on the pre-fill versus many on the decode, just kind of walk through the whole and what it will hopefully be able to enable.

Val Bercovici: Absolutely. So it was DeepSeek, at least publicly introduced a new tier of pricing, that new class of pricing, cached input pricing, last year. They wrote a paper about it and disclosed how they do it, a few months ago, during their infamous six days of open-source disclosures. But it takes the concept, and we can go back to the early days of cloud and Snowflake. One of the reasons Snowflake became so popular is that they said, you know what, you don't have to have the same Amazon instance, cloud instance, to do both your data processing and your data storage.

You could decouple, which was a term they would use back then, your storage from your processing, scale those differently, pay for those differently, consume those differently, as you have different ratios of, you know, storage work and processing analytics work. The same thing is now happening in the AI inference space, with disaggregated, rather than decoupled, pre-fill and decode.

And so what that means now is the process of pre-filling, preparing the data to decode, is very GPU-intensive. That's where all those tensor cores kick in and operate full-time. And that is where it makes sense to have your best accelerator, your highest-performance GPU, focus on pre-filling your data and creating the KV working cache for five to 15 minutes or so, so that you can then begin to decode it. And that still is a serial process. You really can't decode until you create the KV cache, and creating the KV cache is very compute-bound.

Now, once you've created that KV cache, the process of, again, getting to your reasoning and finally outputting tokens that you care about is decoding. It's very memory-intensive. It is going back because of the way these autoregressive transformer models work, which is a very Bayesian approach. All of the next token prediction is best performed by looking at all the prior tokens just up until the point in time you're creating that next token. So, you've to look back into your memory, into the context, over and over again in parallel millions of times, so that you can make that high-quality next token prediction.

This is very memory-intensive, the decode. And you're stressing very different parts of a GPU server at that point, right? At decode, you're stressing those three tiers, particularly, you know, the high bandwidth memory and the DRAM portion of KV cache, because you're trying to pull during decode as much memory into the GPU, into the tensor core as fast as possible.

Any delay there again just keeps that costly asset idle. Therefore, decoding again is a significantly different workload profile in the server compared to pre-fill. And what's making sense right now is dedicating at scale, particularly banks and banks of GPUs, just for pre-fill. That first part of the disaggregation, and then banks and banks of GPUs just for decode, and you can process optimize the GPUs for pre-fill, you should be processing optimizing differently than the GPUs for decode, because those are memory-centric and memory-focused. And what that means is you can bring new life into prior generations of accelerators by mixing Blackwell's, for example, for pre-fill, and hoppers for decode and each are performing optimally, each are certainly giving you a certain depreciated value, know, and current rate of return and you're not creating any artificial delays, you're keeping everything humming as efficiently as possible by optimizing what each class of processor and its memory ecosystem is best suited for.

Doug: Yeah, so I want to reiterate this because I think it's really important to understand why this is such a big deal. Nvidia talked about it at GTC. Dynamo was essentially the implementation of some of the work that DeepSeek had already did. This is going to be the key story for inference serving for the rest of this year. I think, and we've got to think about the bigger picture here, I think it's really important because what this will enable, like we're talking about the inference and the token throughput per a single node, this will hopefully be able to add a lot more scaling out and meaning that you know it's the same GPU bottleneck we had will probably because of the increase. I don't know, and this is something that I think a lot of people are working on in terms of quantifying. think we are at SemiAnalysis as well, right?

However, understanding what this unlocks is a really big deal. And essentially, I mean, it's kind of interesting. I don't think it will happen, but it would be effective within inference, right? Within this test time, compute scaling, we're almost unlocking two different types of compute. That's being done just for inference, if you remember roll back a year ago. Everyone's talking about doing or two years ago.

Everyone's talking about accelerators focused on inference versus training, right? This is just within inference, two different types of computation being done here, can head, and then the point is that the heterogeneous Output is going to be a lot bigger than what one single node can do. I'm very excited because I think it's going to increase token throughput massively, I think it's going to improve memory utilization massively.

Val Bercovici: Exactly.

Doug: And then obviously that brings the cost of tokens down. Is there any other kind of things that I should be aware of that are like logical takeaways from that?

Val Bercovici: So, you know, one of the first takeaways here is that ⁓ disaggregated prefill, disag prefill is introducing the notion of assembly lines to AI factories for the first time. because it's kind hard to imagine a factory without an assembly line today.

But back in the olden days, when factories were first established, they had clusters of workers coming in and out of a particular work area, such as a car assembly line. The different specialists would go and work on the car, then leave, but the car never moved, and it was very inefficient to have to move the workers in and out. That's exactly how inference happens today before Disagg Prefill.

Assembly lines, as the name implies, essentially mean that we keep the data flowing and keep them flowing through whatever specialty is necessary for processing. So prefill is a very different set of work than decode. And that's why moving data using an assembly line process, the way DeepSeek innovated, at least publicly, is very, very important right now. So that's one of the first takeaways: we're finally entering the era of assembly lines for AI factories on the inference side. Next, of course, is just the nature of the workloads themselves.

Specifically, 2025 and a subsequent year, as an excellent proxy for this, have seen workloads shift from being individual chat session-based to agent-based, which means much more context.

We're talking about processing large volumes of documents, extensive code bases, extensive transcripts, or large videos themselves. And more importantly, we're not just stopping at the first question and answer. We're asking a lot of repeated questions and receiving the same answers. It's called a multi-turn prompt.

This notion of high-context, multi-turn communication is becoming the dominant workload of 2025. And that really presses, know, or stresses inference far, far more and creates this need to scale inference and decouple the two different natures of the workloads, the pre-fill and the decode, to get to the maximum efficiencies, tokens per second, tokens per watt, and ultimately be able to support more users simultaneously, larger batch sizes on the same GPU asset investment, the same memory deployment, and certainly just the same power budget that every AI data center operates under.

Doug: So, I'm going to transition this to Weka and your potential solution. You know, we're working on some of the verification for all this stuff, but talking about how we scale the decode out to become a lot bigger, because in this process, we're talking about the assembly line and being able to split one portion of inference into one part and another portion of inference into the other.

The thing that's specifically on these agentic really large multi-turn processes is that the KB cache just explodes because you know the entire context of your Claude code or your cursor session becomes it hits the context window every single time, so the decode and kind of disaggregating the decode can make hopefully I believe a much, much larger context windows.

Can you help me discuss what is currently being done, which, to my understanding, involves DRAM caching —essentially, giant pools of DRAM cache? For example, if you have many users at OpenAI, what Weka is trying to do is address and scale out the decode context window.

Val Bercovici: Absolutely. So, you know, let's take one quick step back, one short step back in an ideal world. Since we're essentially creating this working memory in prefill before we can actually use it for the outputs on the decode side, we don't want to have to recreate it repeatedly. A best-case scenario is that we create working memory for every model instance session, and then we just decode forever.

So DeepSeek created a simple formula for this, xpyd, and the open source community in particular, Vllm, has adopted it as a discussed scaling inference. So, XPYD simply stands for X, how many times you have to pre-fill the data in a working session. So, if your working session extends for more than five or 15 minutes, and you have multiple simultaneous users on the same cluster, that five to 15 minutes gets compressed because you have to support more users on the same hardware and memory. So, you typically have to do many, many pre-fills, and that's where a lot of the waste, slowness, and inefficiency come in. However, you want to have a certain ratio between X pre-fill and Y number of decodes. However, the ideal scenario, of course, is one P and one pre-fill.

You pre-fill the data, and then you decode forever. How do you get there? Well, how you get there is by having more or less infinite memory. And we've heard this term from Google, for example, infinite context windows. So, Google has been able to approximate this by using both their TPU, or Tensor Processing Unit, architectures, as well as a ring retention algorithm particular to Google and Gemini.

And they've been able to take banks and banks of DRAM, network DRAM associated with their TPUs, and give you these million tokens, up to 10 million token context windows. They were first to market with that because they were able to optimize their infrastructure for that. However, the economics of doing so are very stressful, even for companies like Google. And not everyone is Google. Until recently, no one else had million-token context windows. We're just on the cusp of seeing that go mainstream with Facebook, with Llama 4, with Minimax, which was just released the other week, and various models that will soon support millions in token context windows, which everybody wants, by the way.

So, the pressure is on. How do we take these three tiers of memory —SRAM, which is super expensive, super finite, and high-bandwidth memory —as your readers know all too well, also relatively expensive and completely finite —and DRAM, which is also theoretically expensive and finite? Well, when you look at the math, DRAM is the one now, and its cousin, NVMe, non-volatile memory extension, and flash devices, are now within striking distance of each other. know, DRAM in isolation is nanosecond class latency, NVMe in isolation is microsecond class latency, but you brought up the grace chip earlier on, and even, you know, the Hopper class, whether it's GH or Blackwell, actually doing the transfer between DRAM and HBM is on the order of microseconds, whereas in doing the transfer from HBM to SRAM is still on the order of nanoseconds on these GPU servers.

So because now we're looking at a microsecond-level transfer between HBM and DRAM, optimally managing these NVMe NAND flash device as well is the key to making this all work. And I always like to joke there's no S in NV’ in NVMe, right? It's a non-volatile memory extension. So that's one of the things Weka was optimized for doing from day one is taking full advantage of all of these new converging standards, NVMe, the fact that a lot of people still haven't internalized the notion that the high performance compute networks in AI factories that GPUs are attached to are faster than the motherboards themselves.

This is very counterintuitive to most people. So there are more PCI lanes accessible on the network than on any individual server motherboard, or GPU server motherboard. And so that means that if you can aggregate all that amazing network bandwidth to the GPUs across your high-performance computing networks and attach NVMe devices on one end and get the consumption by each GPU across that high-performance computing on the other end, you end up with more memory to the GPU, more DRAM particularly, to the GPU from the network than from the motherboard. And that's what WECA has released with augmented memory technology.

We've been publishing our own benchmarks on this topic in our blogs since February. We recently had our first cloud partner, OSCI, and Oracle Cloud publicly publish their own benchmarks and validate these results, showing that you can extend the DRAM class of memory from the motherboard down to the compute network on these GPU clusters. And by leveraging that compute network, we'll dive into what that means in contrast to the storage network. You're able to have effectively limitless DRAM, which now means limitless KV cache, and ultimately, infinite context windows for everybody else outside of Google.

Doug: So yeah, that's the engineering that will make the infinite context window. And to be clear, context windows are exploding already. And leading-edge models, for example, are often larger than the stated numbers available in the API, which is necessary for some fine-tuning, as well as for some of the system prompts that are incorporated into the models. think, yeah, that's kind of like the Weka solution here.

I would actually like to use the rest of this time to transition to a brief history of storage, as Val has been around the block, and it would be a real shame for me not to discuss the other companies.

One of the ones that everyone is probably aware of from here is like Pure Storage. They're publicly traded. They're, you know, the kings of network-attached storage. I would love to discuss this transition, possibly moving away from the disaggregated, pre-filled decode, which, to be clear, is a significant change. This is how we're going to scale to much bigger.

Now I want to talk about how the network attack storage space itself has kind of transitioned from you know, kind of the history of the past to now, because I think that that would be a very interesting arc to sort of go through with my listeners.

Val Bercovici: Absolutely. So let's go all the way back, even though storage predates network attached storage, but clearly network attached storage is a big deal. NetApp was born in the network-attached storage era.

They kind of optimized storage not for the storage media, which back then was only what we call spinning rust, now hard drives, which have heads that platter that spin and heads that actually go back and forth across the platters to extract the data. NetApp was born in that era, optimized for that environment, and, you know, became the brand that it is today.

Pure that you brought up earlier on was born of the flash era, of the NAND flash era. And so Pure realized, you one fundamental thing, which was, hey, all of a sudden this NAND flash media that we're having terabytes and terabytes of in our storage arrays is worth more than this CPU based controller that controls all of it and presents, you know, these protocols, these SAN and NAS and other protocols to the servers and other end users.

So by just realizing that, you know, there was an advantage to engineering the NAND flash and the NAND flash shelves and then creating this really brilliant program, the Evergreen program, just to let people upgrade controllers because they were the lower-priced item compared to the media.

Pure, you know, became the brand that it is today in the sand space and in the flash space, flash storage space, because they were born in this era, understood the supply chain, understood the value, the ultimate customer value that is fundamentally different than spinning media, the older hard drive generation of technology and emerged to be the leader they are today.

Doug: So, then I want to, so I know Pure Storage is the leader, like in the legacy compute, let's call it that. I'm sorry to call it legacy compute. I know the CPU guys probably hate it, but let's be real, it's legacy computing, right? Let's discuss what that looks like in the future. Because I think that there have been a few other, DDN, VAST, Weka, started, these newer challengers who are entering this space specifically with the focus on the opportunity from the newer compute strap ⁓ world, which is mostly ⁓ a GPU-driven architecture. So let's just kind of talk about how, know, what are, you know, you can talk about your competitors, whatever you want to do. I just think a market ecosystem overview of, and then obviously with the caveat that that works for Weka.

Val Bercovici: Yeah, you know, let's take a simple visual that I think a lot of us have seen. We've all encountered the concept of exponential data growth. And we've probably seen some of the crazy charts. I'm a visual guy, so looking at the hockey stick charts for exponential data growth, that's no longer a future state; that's a present state.

We're in the steep part of this hockey stick curve, and exponential data growth means just that. There's just an enormous amount of data being generated synthetically now, as well as organically by the world and by all of our systems. The vast majority, 80- 90 % of it, is unstructured data.

So the legacy market, the legacy compute and storage market that we talk about now jokingly, was not optimized for unstructured, explosive, exponential data growth. It was optimized for transactions, Oracle databases, DBT databases, SQL servers worldwide, structured data, and latency-sensitive data. And that was paired with more analytical systems, which provided more horsepower, more CPUs, and more memory, all brought to bear in the cloud. This was the dominant workload of the cloud, creating these analytical systems to extract meaning and value from all this transactional data.

Enter GPUs. And again, GPUs just are fundamentally different than CPUs. The stress and workload on GPUs bear no resemblance to what a database transaction looks like. To process all of this massive, exponentially growing data in parallel, we need fundamentally different storage systems and they're not storage arrays at all.

They have to scale out much, much more horizontally. And let me know if we're gonna have a new member in the group. That's all good. To address exponential data growth, you need all your storage systems to work in parallel. They have to be scaled out from the start. They have to be optimized for unstructured data.

And unstructured data doesn't mean just one workload, one storage workload. It means large file reads and small file reads. It means both random access and sequential access. It means ⁓ focusing on ⁓ billions and billions of directories and trillions and trillions of files in those directories. It means object interfaces, S3 protocol interfaces to millions and trillions of buckets. It means so many different things.

In parallel compared to what we've seen in the past for CPU-based storage. So the fundamental systems are just built differently. Weka, in particular, doesn't build storage arrays, never really has. Many people like to think that we build bigger storage arrays. That's not true. We've had this software-defined containerized system from day one. Again, we're born not just of the cloud era, but of the AI era, just as NetApp was for NAS and Flash was for Pure. We're AI-native; we're very GPU-native systems.

So we call this a mesh now, just a mesh of containers that fundamentally take advantage of one reality, which is that the network in GPU computing and AI factories is faster than a motherboard. Weka is the first storage system and the first storage cluster designed for this reality. No one else is. Everyone else optimizes around.

Either individual storage controllers or, like most of our competitors, clusters of storage controllers front-end the actual high-performance NAND flash and storage media. Weka doesn't work that way, right? One of the most radical examples of how Weka differs is something we call converged mode, where we deliver software-defined memory. That's an oxymoron for most people, but then again, software-defined networking was very heretical in the earliest days.

Software-defined storage became somewhat less heretical but still novel when it emerged. Software-defined memory today is very heretical, but people will realize soon that when you're buying these banks and banks of GPU servers, they come, of course, with GPUs, they come, of course, with all three classes of memory we talked about. Still, they come with X NVMe drives, often eight NVMe drives per server.

Since GPUs, particularly at inference time as well as training, often work in clusters themselves, when you have eight GPU servers, each with eight drives, or let's pick nine GPU servers to talk about NVL72s in particular, with eight GPUs per, often you get about eight drives, about 72 drives, NVMe devices, per NVL72.

Installing Weka software on that instantly creates software-defined memory because we take those 72 NVMe devices and convert them into a DRAM class of memory. And these inference servers we spoke of, VLLM, TRTLLM, SG-LANG, now understand how to recognize the thousand times more density, the terabytes per device of memory, now to complement the very limited number of terabytes.

Really the fractions of petabytes of NVMe capacity with the terabytes of DRAM capacity with the gigabytes of HBM capacity. And all of a sudden, we have a theory mechanism that gets you to this Nirvana ideal state of pre-filling that working memory that KV cache wants after you've loaded the model weights, never having to pre-fill again, never having to evict cache and decoding forever. And we can essentially fast-forward AI inference to just decoding.

Everybody wins, particularly for these cursor-like, agentic workloads where it's always high context, it's always multi-turn. If we never pause and slow down to re-prefill every 10 or 15 minutes, but we hit the throttle, you know, we hit the throttle, we put the pedal to the metal, so to speak, nonstop during inference, we get optimal inference. then what I love is the token economics, the unit economics of this, all of a sudden make a lot of sense right now, because we can do, create more tokens per second in aggregate, generate larger batch sizes, which means supporting more users, and every one of those users gets the lowest latency time to first and last token. So it's a win-win all around when you can unlock that last final bottleneck of AI inference.

Doug: Yeah, so yeah, I think that that's a really good ending on what weka does as well as like, know, Kind of wrapping it all together of the disaggregated PD. This is the example. Yeah, I'm pretty excited for what's to happen there. You know, there could be some, well, let's see, I'm not sure if everyone wins, right?

Well, I guess the reality is that if we purchase more GPUs, right or no, if we do more tokens, we create more things. They'll still purchase more GPUs, like me, something that I'm not familiar with, but it doesn't concern me. However, that does sound like GPU utilization is about to reach its limit, right? But that's an aside, right? But I do appreciate, yeah.

Val Bercovici: I would say just to wrap up technically, GPUs were underutilized before. They weren't utilized during the majority of inference, which was the decoding part. Now we truly drive up GPU utilization to its maximum potential.

All of the NVMe, ⁓ sorry, all of the HBM capacity being manufactured is already allocated and being purchased. All the DRAM capacity, particularly the AI-friendly DRAM, is already being purchased. So this is very bullish for NVMe, the Flash SSD ecosystem, particularly the high-performance TLC class of NAND Flash. It's very bullish for that because it represents a new life on top of a pretty valuable market opportunity that NVMe has always had.

Doug: To be clear, that's not the official opinion of fabricated knowledge of SemiAnalysis, but it's the official opinion of Val. Anyways, I just wanted to wrap it up here. Is there anything else you'd like to say to the listeners, or should we probably leave it at that?

Val Bercovici: It's my opinion that I expect one fundamental change in the AI business, which is again funded by inference. And that is, we're going to see the three classes of pricing, which is input cashed input and output, collapse very soon to only two classes of pricing because people will realize when you don't have to only have five to 15 minutes of life for your input token pricing, it could be weeks or months of life or effectively infinite. You don't need to have a distinct tier of pricing.

So this is gonna fundamentally shift. The business of AI and the unit economics of AI, simplify the pricing and give more value to users. And it's a question of which provider will lead the charge and define that new low class of pricing, and who will be the followers.

Doug: I'm excited to see how it shakes out. Thanks for the time, Val. Appreciate having you on.

Val Bercovici: Likewise, love the conversation.

That’s it for this week! Thanks for reading!