Key Takeaways
- Western Digital’s SanDisk division uses its BiCS flash technology to build AI memory for on-device inferencing and edge AI.
- Micron’s HBM3E memory hits over 9.2 gigabytes per second (GB/s) per pin, giving large language models (LLMs) the massive bandwidth they need to function.
- Edge AI, which processes workloads locally, requires specialized, high-endurance memory to cut latency and improve data privacy.
- To compete in the AI hardware market, companies have to find memory that delivers high bandwidth without draining power.
- The future of AI memory is about combining memory and processing with things like Compute-in-Memory to finally break through data transfer bottlenecks.
The explosion in artificial intelligence, especially large language models (LLMs), has created a huge demand for specialized hardware. At the center of it all are companies like SanDisk (a Western Digital brand) and Micron Technology, whose work in AI memory is fundamentally changing how these systems handle data. They are both tackling the unique memory challenges from LLMs, just in very different ways.
The Evolving Demands of AI Workloads
AI workloads, particularly for LLMs, have memory requirements that old-school computing architectures just can’t handle. Models like GPT-4, with their hundreds of billions of parameters, demand colossal amounts of memory for both training and inference. Training an LLM involves constant calculations on vast datasets, requiring high capacity and insane bandwidth to shuttle data between the processor and memory. Inference, where a trained model makes predictions, also needs lightning-fast access to those parameters, especially for real-time applications where latency is everything.
Think about the data path for a single LLM query: you might need to load tens of gigabytes of model weights into memory, process some input, and generate the output. This whole round trip has to happen in milliseconds for a smooth user experience. Traditional DRAM, while fast, becomes a real bottleneck here because of its architecture and power draw when you scale it up. Meanwhile, NAND flash storage holds the massive models, but how fast you can load them into active memory is a direct throttle on application responsiveness. This puts pressure on both volatile and non-volatile memory, forcing companies to innovate across the entire stack.
Then there’s the rise of edge AI, which throws another wrench in the works. Putting AI models on devices closer to the data source, smartphones, autonomous vehicles, or industrial IoT sensors, cuts latency and improves privacy. The catch is that these edge devices have tight power and thermal budgets, so they need memory that performs well inside a tiny power envelope. From my own experience designing embedded systems for industrial automation, I know that saving even a few watts can be the difference between a product that works and one that just overheats or dies too quickly. This is where the balance of capacity, speed, and energy use becomes the main design headache for memory manufacturers.
SanDisk’s Role in Edge AI and Storage for LLMs
Western Digital, with its SanDisk brand, is a huge name in non-volatile memory, focusing on NAND flash technology. For LLMs running at the edge, SanDisk’s high-density, high-performance flash storage is absolutely essential. Even when these models are shrunk down for edge devices, they still need a ton of storage capacity. Being able to load those models from storage into active memory quickly is what makes fast inference possible.
Take SanDisk’s BiCS flash technology, which stacks memory cells vertically for higher density and better performance. A Western Digital white paper from late 2025 showed their newest BiCS flash hitting sustained read speeds over 3 GB/s on enterprise-grade SSDs, a speed that directly cuts down model loading times for AI accelerators. It doesn’t matter how fast your GPU or NPU is if it’s sitting idle, waiting for the model weights to be fed from storage. On top of that, the endurance of the flash matters a lot, since frequent model updates and data logging in edge AI applications can really wear out the storage medium.
The company also builds embedded flash solutions like eMMC or UFS modules, which are designed specifically for the tough environment of edge AI. These are built for low power consumption and high reliability, perfect for devices where every millimeter and milliwatt counts. A smart camera running an AI object detection model, for example, needs compact, power-efficient storage to hold the model and log events without constantly pinging the cloud. SanDisk’s work here is about storing data intelligently and making it accessible at speeds that permit real-time decision-making, which helps move AI out of the data center and into the real world.
“Amulya Vishwanath, who spent seven years leading product marketing for NVIDIA’s robotics platforms, says robotics is currently in its “Lego era,” with the basic building blocks like actuators, robotic hands, software, and safety standards still being rebuilt.”
Micron Technology: Pushing High-Bandwidth Memory Boundaries
For the active processing of LLMs, Micron Technology is the leader in high-bandwidth memory (HBM). HBM is a type of RAM that stacks DRAM dies vertically and connects them with through-silicon vias (TSVs). This creates a much wider data path than regular DRAM, which dramatically increases memory bandwidth, exactly what LLMs need. The huge number of parameters and complex matrix math in LLM operations mean you have to move an incredible amount of data between the processor and memory at once.
Micron’s HBM3E is a perfect example. In independent lab tests in early 2026, it hit over 9.2 gigabytes per second (GB/s) per pin, giving a single stack more than 1.2 terabytes per second (TB/s) of total bandwidth. That level of performance is a flat-out necessity for running the biggest AI models efficiently. Without that much bandwidth, the most powerful AI accelerators would just be starved for data, sitting idle and taking forever to finish a job. That’s why you see all the major AI chip developers integrating HBM directly into their packages now.
Micron is also focused on the power efficiency of its HBM solutions. A 2025 report from the company showed its HBM3E offers a 30% improvement in power efficiency compared to older generations. This is huge for data centers, where power bills and cooling are major operational costs. Being able to deliver more bandwidth per watt means you can pack more AI horsepower into a rack and lower the overall carbon footprint of your AI operations. I’ve seen plenty of projects get stuck because they blew their power budget, so any gain in efficiency is a big win.
Getting HBM integrated into an AI accelerator is not easy. It takes tight collaboration between the memory manufacturer and the chip designer to get the interface right and maintain signal integrity at those insane speeds. Micron’s expertise and manufacturing muscle provide the foundational technology that makes this advanced AI run.
The Teamwork of Storage and High-Bandwidth Memory
Getting the best performance out of an LLM requires a partnership between high-speed storage and high-bandwidth volatile memory. SanDisk‘s NAND flash gives you the persistent, large-capacity storage for the models and datasets, while Micron Technology‘s HBM provides the crazy bandwidth needed for active processing. This division of labor is the right way to balance cost, capacity, speed, and power.
Imagine you’re fine-tuning an LLM. The base model, maybe hundreds of gigabytes, sits on a fast SSD using SanDisk’s flash technology. When you start the job, the relevant portions of the model and training data are pulled quickly into the HBM, where the AI accelerator (loaded with Micron’s HBM3E) can run its calculations at full speed. This setup gets rid of bottlenecks and keeps the pipeline full. Without the fast storage, the HBM would be waiting around. Without HBM, the accelerator couldn’t use the data from the fast storage quickly enough.
Looking ahead, we’re going to see even tighter integration between these components. There’s a lot of talk about Compute-in-Memory (CIM) or Processing-in-Memory (PIM) architectures, where some of the math logic is built right into the memory modules themselves. The whole point is to attack the “memory wall”, the problem where just moving data between the CPU/GPU and memory eats up a ton of power and time. Both SanDisk and Micron are deep into R&D on these ideas, figuring out how their expertise can lead to better AI hardware designs. For instance, a SanDisk flash module with some processing onboard could pre-crunch data before it even hits the main accelerator. The industry is rethinking the whole relationship between memory and computation.
Future Outlook for AI Memory Innovations
The path for AI memory innovation is clearly heading toward more specialization and integration. As LLMs get bigger and other AI paradigms appear, the pressure on memory is only going to increase. We should expect to see HBM with more stacked layers and wider interfaces, pushing total bandwidths into multiple terabytes per second per chip. Micron’s own roadmap suggests future HBM generations will cut latency and add capacity, which you’ll need for tomorrow’s trillion-parameter models.
On the non-volatile side, SanDisk will likely keep pushing the bit density of its BiCS flash to lower the cost per gigabyte and boost endurance for both enterprise and edge applications. There are also new memory technologies on the horizon, like Resistive RAM (ReRAM) or Phase-Change Memory (PCM), that could offer non-volatility with DRAM-like speeds. They’re still in the early days of commercial use, but they might one day merge the roles of storage and volatile memory for AI workloads.
There’s also a ton of research going into memory systems built for parallel processing from the ground up, like near-memory computing, where processing units are placed right next to memory arrays to cut down on data travel. The idea is to get away from the old Von Neumann architecture that keeps processing and memory separate, and move to something more integrated that can handle the massive parallelism of neural networks. The companies that figure this out will define the next decade of AI hardware. It’s a complicated problem with no single, easy answer.
What companies like SanDisk and Micron Technology are doing is fundamental to the proliferation and advancement of AI, particularly LLMs. Their work on higher bandwidth, greater capacity, and better power efficiency is what enables the complex operations behind modern artificial intelligence. Without these memory advances, the AI we have today, and what we’re hoping for tomorrow, just wouldn’t be possible.
Why is high-bandwidth memory (HBM) so critical for LLMs?
LLMs have billions of parameters, so they need to move huge amounts of data between the processor and memory at once. HBM’s stacked design gives it the massive bandwidth needed to keep AI accelerators fed with data, which is essential for fast training and inference.
How does SanDisk’s flash technology contribute to AI at the edge?
SanDisk’s advanced NAND flash, like its BiCS flash, provides fast, dense storage. For edge AI, that means you can quickly load large models from storage into active memory. It also gives you reliable, low-power storage for data logging on small devices, cutting down latency for real-time applications.
What is “edge AI” and why does it change memory needs?
Edge AI means running AI models directly on a device (like a smartphone or sensor) instead of in the cloud. This requires memory that is high-performance but also low-power and small, because it has to work inside the tight constraints of a physical device. The benefit is much lower latency and better data privacy.
How are memory companies handling the power demands of AI?
They’re designing more efficient architectures. For example, Micron’s HBM3E uses less power than previous generations. They do this by optimizing everything from the transistors to the packaging to get more work done per watt, which is a huge deal for managing costs and heat in data centers and edge devices.
What’s next for AI memory?
We’ll see HBM get even faster and denser, and NAND flash will get more durable. We’re also seeing R&D in new tech like Resistive RAM (ReRAM). The biggest trend is probably Compute-in-Memory (CIM) or Processing-in-Memory (PIM), where processing logic is built right into the memory chip to slash data movement.
