The friendship that built Google

When you think about Google , whose name pops up in your mind? Larry Paige? Serge Brian? Maybe Sundar Pichai? What if I told you, there is a duo of engineers without whom Google would've been a victim of its own growth. Meet the heroes of our story, Jeff Dean and Sanjay Ghemawat the only 2 engineers at Google to reach the legendary title of L11 - Senior Google Fellow and the creators of many modern systems that are industry standards today.

Our story begins in March 2000, back when Google was an up and coming startup, far from the tech giant it is today. To save the money, they decided to buy cheap used computers that ran Linux instead of spending 100s of 1000 dollars to buy an IBM supercomputer for their day-to-day ops, it did save them sizeable money but it was going to cause some major issues.

The compartmentalized hardware that they bought, wasn't built to scale at a pace that Google was growing. Evidently, the computers started bricking on regular basis. To counter this, Larry and Serge wrote software to re-route and distribute the load to other hard drives, problem solved, right? Yes and no.

Although it was good for sometime, but on one particular friday. The heart of Googles Engine broke, the software responsible for crawling and indexing the web suddenly started to slow down dramatically, and even when the results were shown, they were outdated, nobody knew what was happening and it was all happening when Google was about to close a deal with Yahoo to power their search engine, what were their conditions? Fast results and minimal latency.

To tackle this, 6 of Googles top engineers were called for a crisis meeting. "After brainstroming for hours, we couldn't figure it out" recalls Craig Silverstein (Google Employee #1), silently sitting in the other corner was Sanjay Ghemawat, one of our two heroes. He started combing through the code, one line at a time. Craig saw him and instantly knew what he needed, so he paged for Jeff Dean to join Sanjay.

Sanjay and Jeff became buddies when they started working together, at Digital Equipment Corporation. They had a history of collaboration and would often share the same computer, when one brainstormed the idea, other would write it out and vice-versa. They worked in a perfect symphoney, like two bodies with one brain. When Jeff was offered to join Google, he said that he would only join if they brought his buddy Sanjay with him.

When Jeff came after receiving a call from Craig, he pulled up a chair and sat next to
Sanjay and started brainstorming as they always do. Intially it was thought to be a bug, a missing edge case, a faulty snippet or a dependency issue. But after days if intense sessions, there was nothing, everything seemed to be perfect. Tired of hitting the walls, they dove into Machine code(Binaries), checking every single bit, and there it was, one bit that was supposed to be 0 was 1 and it lead to other flipping, resulting in loops running over and over.

What caused it? Why nobody else caught it? Because it was never a software issue! You see, the cheap racks that Google had bought, weren't built to handle the volume of requests they getting. Thus it caused them to flip the bits manually once their threshold was crossed. This was a rare issue and observed when a impossibly large process requests hit a server, but due to the scale of Google, even the rare issues had to be accounted for.

Sanjay and Jeff knew that this problem was not the one that would be solved by a fix or workaround, so they spent the weekend building new infrastructure from ground-up to re-route and manage the requests when the system was failing. They had successfully managed to weather the storm to take the systems online before the demo to Yahoo team.

All though they walked with minimal loss, it marked a turning point in Google's software strategy, they will not rely on hardware to work but on software to make it work i.e. not relying on cutting-edge hardware but smart , fault tolerant software. Sanjay and Jeff were on the forefront of it. They pioneered some of the distributed systems that are industry standards like MapReduce, Big Table, Google File System, led the team that created Tensorflow and so much more.

Over the time, as Google grew Jeff and Snnjay were elevated to legendary status, not just because of the revolutionary pieces of Software they wrote, but also because of the synergy that they worked in, collaborating, accomdating and critcizing every idea that crossed their minds. They just didn't set an example, they laid the foundation for a collaborative, ego free culture. And it all began during the March 2000 indexing crisis.