Introduction

Imagine the internet before Google. It was a bit like a sprawling, uncharted territory. When Sir Tim Berners-Lee first created the World Wide Web, he actually kept a list in the back of his notebook, of which new websites that came online.

Then search engines emerged to make it easy to find relevant websites. Early search engines like Yahoo! relied on humans to categorize websites, resulting in often inaccurate or incomplete results.

Searching for information was a tedious and often frustrating process. It was clear that a better way was needed to navigate the ever-growing expanse of the web.

Enter Larry Page and Sergey Brin, two Stanford PhD students with a radical idea. They envisioned a search engine that would rank websites not just by keywords, but by their importance and relevance within the vast network of the web. This was the birth of PageRank, a trillion dollar algorithm that would propel Google to become the dominant force it is today.

Link to the paper: The PageRank Citation Ranking

Mapping the Web

Most webpages link to other relevant webpages. For example, this page links to the original paper. When the paper was written, the web had 150 million pages and 1.7 billion links. Each page can be seen as a node, and the links as edges connecting one node to another.

So, if webpages A and B links to webpage C it should look something like this:

Here A and B are called backlinks of C.

Why are these backlinks important? The idea is that "highly linked pages are more important than pages with few links." For example, many blogs and articles about PageRank will link to the original paper, supporting the hypothesis that the original paper is more important. This is similar to citations in academic research; usually, the more impactful a publication is, the more it gets cited.

Another observation the authors pointed out is that any outgoing links from these relatively important pages should also be given higher importance. PageRank tries to determine how well the importance of a page can be estimated from its link structure.

PageRank's brilliance lies in its smart use of a mathematical concept called the Markov Chain.

Mathematically, a Markov Chain is a model that describes a sequence of possible events where the probability of each event depends only on the state attained in the previous event. It's like a random walk where your next step is determined by a set of probabilities associated with your current location.

First, let's understand how the author of the papers defines PageRank.

Defining PageRank

If a webpage A has 4 outgoing links, the Rank(A) is divided equally among the 4 links. So, if webpage C has backlinks from 2 different webpages, then the Rank(C) will be the sum of the ranks from each link.

More formally:

The paper includes a constant factor but let's keep things simple for now.

But then how do we get the Ranks of a.com and b.com? Well, we do the same process, recursively.

What if there is an outgoing link from c.com to a.com? Then once we get the rank of c.com we have to recompute the rank of a.com? This is a stochastic process, and we can use the mathematical model of Markov Chains for that.

And the PageRank formula described above can be represented using matrices as follows. At any point we can define A as a square matrix with the rows and columns corresponding to the web pages.

if (page u and v are connected):
    A[u][v] = 1/N[u]; // N(u): is the number of outgoing links from u
else:
    A[u][v] = 0

If R is a vector over all the webpages then we can describe it as:

R = cAR

Here c is a normalization factor, to ensure that the total rank of all webpages is constant. And we can keep computing this until we don’t see any change in the value of the vector R. That final value is called the Stationary Distribution of a Markov Chain. But will that always exist? To know we first have to understand a few more terms:

Reducible Markov Chains

A Markov chain is considered reducible if its states can be divided into two or more disjoint sets, where it's impossible to move from one set to another. In other words, there are "barriers" within the chain that prevent transitions between certain groups of states.

Periodic Markov Chains

A Markov chain is considered periodic if there is a fixed pattern or cycle in the way it revisits certain states.

In both these cases, finding a stationary distribution becomes tricky.

Problem of Rank Sink

But we have another problem. Consider this case, b.com points to a.com. But a.com points only to c.com and c.com points only to a.com.

What will eventually happen is all the rank from the entire chain will get trapped in this cycle b/w a.com and b.com. The loop forms a trap which is called the rank sink.

The authors tackled these ingeniously by a method called...

Random Surfer Model

Sounds fancy, but it's simple. Just think from what you would do in such a scenario. If you visit a webpage that has just one link to another webpage, and that page then links to the original webpage. Will you keep clicking and going in loops? Obviously not, you will just go to a different page. This is what a random surfer model will do. Occasionally a random surfer will get bored and then jump to a random page chosen based on a distribution.

This super simple solution is enough to tackle all the problems we described above.

Mathematically, we can assume that there is an outgoing node from every node to every other node. And then to compensate for this extra link, we can tweak the factor c to again normalize the metrics.

Let's dive into the Implementation

So, there are two parts of this search engine problem. First we need to have the database of all the links to create the Markov Chain in the first place. This is done by a program called Crawler.

Then comes the actual PageRank implementation. And here the steps:

1. Convert each URL to an unique integer ID and store the mapping in a database.

2. Sort the link structure by Parent ID.

3. Remove all the dangling links. (Dangling links are those pages that have no outgoing link. Removing these links have no effect on the final distribution)

4. Have an initial assignment of ranks.

   • The initial assignment won’t really affect the final results as we saw that Ergodic Theorem guarantees an unique stationary distribution, but it will affect how quickly we can reach the final state i.e the rate of convergence.

5. The original implementation used single precision floating point values at 4B each. So, for 75 million URLs it amounted to 300 MB.

   • In terms of scalability, if a point comes where memory available is insufficient to hold the weights, the previous weights can be stored in disk and then accessed linearly.

6. Finally once convergence is reached, the dangling links are attached back.

When the paper was released this whole process took around 5 hrs.

On a large 322 million link database the convergence took around 52 iterations. The convergence on half the data took around 45 iterations, which shows that this PageRank algorithm can scale very well even for large collections as the scaling factor is roughly linear in logn. The authors admitted that the cost required to compute the PageRank is far insignificant compared to the cost required to build a full text index.

Early Results

To initially test the usefulness of PageRank the authors implemented a search engine which used just 16 million web pages and performed just title based search and then sorted the results by PageRank.

Here is an example of the search result for the term “University”. On the left we have the PageRank algorithm and on the right we have another popular search engine at that time, Altavista.

PageRank returns the home page of Stanford as the top result (since the initial PageRank was built from Stanford, it is understandable that most webpages had outgoing links to the Stanford homepage). You can also see the PageRank value normalized to 100% as the red progress bar next to each link. In contrast, Altavista returned random-looking webpages that matched the query "University."

The story behind the name

BackRub was the initial name given to the search engine that would later become Google. The name "BackRub" actually reflects the core mechanism behind the search engine's algorithm, can you guess?

Yes, "backlinks".

The name "BackRub" was eventually dropped in favor of "Google" in 1997. The new name was inspired by the mathematical term "googol," which represents the number 1 followed by 100 zeros, symbolizing the vast amount of information that the search engine aimed to organize and make accessible.

In one of the blogs published by Google in 2008, they mentioned that “by 2000 the Google index reached the one billion mark. Recently, even our search engineers stopped in awe about just how big the web is these days -- when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!”.

Conclusion

The tale of PageRank is a fascinating journey through the early days of the web, where two Stanford PhD students laid the foundation for what would become the world's most powerful search engine.

I'm still constantly amazed by the sheer scale at which this algorithm operates, processing trillions of webpages and billions of searches every day. PageRank's legacy continues to shape how we access information, discover new ideas, and connect with others across the globe.

I hope this exploration into the inner workings of PageRank has shed light on the brilliance behind this groundbreaking algorithm. It's a story of how a simple idea, combined with rigorous mathematical principles, revolutionized the way we navigate the vast landscape of the World Wide Web which we kind of take for granted.

If you like my blogs, do follow me and subscribe to my newsletter for more such informative bits.

A quick disclaimer: Don't worry if you're not a graph theory expert. You can still follow and discover the potential of this technology.

Traditional databases: a world of Records and Transactions

The early days of computing were dominated by structured data like inventory records and financial transactions. Relational databases, with their tables and rows, were the perfect tool for this kind of information. These data were mainly discrete and traditional relational databases would suffice such kind of data.

However, the internet revolutionized the data landscape. Today, major players like Google, Meta, and X deal with vast amounts of interconnected data.

Take Facebook, for example. The entire platform is built on a social graph, a web of relationships where Alice is friends with Bob, Bob is friends with Charlie, etc. Similarly, the "follows" relationship on Instagram/X forms another kind of graph. Google's search engine relies on a knowledge graph to understand the connections between websites and the information they contain.

While relational databases are still incredibly useful, they struggle when it comes to modeling and querying these intricate relationships. We’ll take a closer look at these cases.

Representing Relationships

Relationships within interconnected data are best described using graphs.

Let's illustrate this with a simple Twitter example:

• Alice follows Bob

• Alice follows Charlie

• Bob follows Charlie

• Charlie follows Bob

• Alice follows Dave

• Dave follows Charlie

• Bob follows Dave

If we were to ask, "How many people does Bob follow?", we'd have to scan through all the records to count them.

But if we visualize this data as a graph: The answer becomes immediately clear.

We can also see that Charlie is the most followed person in this dataset. Graphs provide a powerful way to visualize and analyze relationships, making them a natural fit for representing interconnected data.

While graph theory has existed since the 18th century (with contributions from mathematicians like Euler), it wasn't until the early 2000s that graph databases started gaining traction in the commercial world. Companies like Neo4j were among the first to offer graph database solutions, paving the way for wider adoption. Today, many popular graph databases are available, including Amazon Neptune and ArangoDB. Beyond social networks, graph databases are used in various applications like fraud detection, recommendation engines, and knowledge graphs.

Terminologies for Graph Database

Graph databases are based on a model called Labeled Property Graphs.

Let's explore the key components of this model

Nodes

Nodes represent individual entities in the data. They can be people, places, things, or any other object you want to model. Each node can have an optional set of properties stored as key-value pairs. For example, a node representing a user might have properties like:

{
    id: 1001
    name: Alice,
    handle: alice123
}

Labels

Nodes can also have one or more labels, which categorize them based on their characteristics. For example, in a social network graph, some nodes might be labeled as "user," while others might be labeled as "post." Labels are essential for querying and filtering data efficiently.

Relationships

Relationships define how nodes are connected. They are inherently directional, meaning that "Alice follows Bob" is not the same as "Bob follows Alice." Relationships can also have optional properties. In our Twitter example, a "follows" relationship might have a property like { created_at: 2024-05-01 } signifying that the person started following from 1st May 2024.

Where Graph Databases Shine

Imagine trying to answer the following query using our Twitter example in a relational database: "How many people does Alice follow who have at least 100 followers and have all posted at least 3 tweets with more than 10 likes?" The SQL query would be a nightmare of multiple joins and subqueries, likely taking a significant amount of time to execute.

In contrast, a graph database could answer this question with a simple traversal, following the relationships from Alice's node to her followers, then to their followers, and finally to their tweets. This traversal could be orders of magnitude faster than the equivalent relational query.

Graph databases also excel at uncovering unexpected insights. While relational databases are optimized for structured queries based on a predefined schema, graph databases allow for more exploratory analysis. They're not just about answering known questions; they're about discovering relationships and patterns you didn't even know existed.

Furthermore, graph databases offer flexibility. Making changes to the schema or adding new types of relationships is often much easier than in a relational database, where complex migrations might be required.

In essence, graph databases are the tool of choice when relationships between data points are as crucial as the data itself. If your data is highly interconnected and you need to perform complex relationship-centric queries or explore your data in new ways, graph databases might be the perfect fit. But if it’s not the case, then relational databases should be your goto choice.

Native Graph Processing

This way of modeling data is consistent across various graph database implementations, but there are different ways to encode and represent graphs in a database engine’s main memory. A graph database can have native processing capabilities if it exhibits a property called index-free adjacency.

How do we represent which nodes are adjacent to each other? In a graph, two nodes are adjacent if they are directly connected by a relationship. In non-native graph processing, an index might be used to look up adjacent nodes, similar to how a relational database uses indexes for faster query performance. However, this introduces overhead and can limit the speed of graph traversals.

So, even on the surface it is still a graph based model, it’s just that there exists a global index to fetch the adjacent nodes.

In contrast, native graph processing engines use index-free adjacency. Each node directly stores references (like pointers) to its neighboring nodes. Think of it like a linked list where each element points to the next. This allows for extremely fast traversal, as there's no need to consult a global index.

There are multiple advantages of having index-free adjacency

Faster Lookups

Depending on the implementation, index lookups could be O(log n) in algorithmic complexity versus O(1) for looking up immediate relationships. To traverse a network of m steps, the cost of the indexed approach, at O(m log n), dwarfs the cost of O(m) for an implementation that uses index-free adjacency.

Simpler Queries

Furthermore if we were to ask “how many followers does node B have?” In non-native graph processing we have to iterate through all the indices to get the count. To avoid that, we have to instead maintain a reverse-index.
Whereas with native graph processing we can just look into the incoming edges to node B and we will know the answer.

While index-free adjacency offers significant advantages, it's worth noting that it can lead to higher memory usage for dense graphs where nodes have many connections. However, for many real-world use cases, the performance benefits far outweigh this potential drawback.

Conclusion

In this introduction to graph databases, we've explored how they differ from traditional relational databases, their core terminology (nodes, relationships, properties), and the concept of native graph processing. In the upcoming posts, we'll delve deeper into the inner workings of graph databases, exploring how Neo4j and other systems store and manage graph data on disk.

If you found this post helpful, please like and follow for more insights into the exciting world of graph databases.

Here was my previous blog which builds the intuition behind compression:

A quick recap…

In the previous blog, we dove into the world of compression algorithms! We explored the fascinating idea that compressing data can, in some cases, actually make the file size bigger. We talked about how every compressed data set has a counterpart that expands when compressed. This is because compression relies on finding patterns within the data.

Idea behind a practical Compression Scheme

One way to deterministically achieve compression would be to have symbols for repeated chars or bits. Let’s take the example of the phrase to be or not to be. If we use a particular symbol X as a replacement for the to be part it will be encoded as: X or not X. This has reduced the number of symbols, hence we can say compression is achieved.

But wait, hold on a sec!! This raises a few questions:

1. What if we had the char X in the stream itself? Or a better question would be how to effectively differentiate between actual characters and symbols used as replacements?

2. How will the decompressor know that X refers to to be?

Solution #1: Let’s say our input is made up of 8-bit ASCII codes. Then in our compression scheme we can agree upon some kind of contract that after compression, the compressed data will be made up of 9-bit ASCII code instead. The decompressor will have to keep this information in mind. So, with 8 bit ASCII we have values from 0-255, with and additional bit we can have extra values from 256-511 that can be used for our symbols. This way we can very easily differentiate between symbols and input chars.

But wait, what kind of compression scheme is this, where we are taking the input is made up of 8-bit but the compressed data is made up of 9-bit? Won't that just increase the size of the data? Will compression be achieved at all?

That’s a really good question. Let’s tackle the 2nd question first and come back to this.

Solution #2: How will the decompressor know that X refers to to be? One approach would be to maintain a table/mapping for every symbol and what characters they represent. And the decompressor should also have this table/mapping information to refer to while performing the decompression.

Here let me callout an important piece of information to keep in mind while developing compression schemes. When we compare compression algorithms we just can’t compare the compressed text with the original text. We need to take into account all the data that the decompressor needs, to be able to accurately decompress the compressed data to our original text (mind you we are still talking about lossless compression as mentioned in my earlier blog). So, in our case we will have to consider the compressed text + metadata (which can be the table/map that were talking) while comparing the performance of compression algorithms.

Now, let's tackle that tricky question.

From my previous blog we got an intuition that compression algorithms should be able to compress effectively, those inputs that are commonly used, at the expense of maybe expanding inputs which resembles random noise. So, when we use 9-bits or more in our compression scheme, we are hoping that with the extra 256 symbols we will be able to reduce the size so much that even with 9-bits we are able to get an effective compression. Let’s do that math with an example string: to be or not to be.

The original string has 18 chars, each char is of 8-bits i.e 18*8 = 144 bits in total.

If we replace the to be with the symbol 256 (0b100000000) then the string itself gets compressed to just 10 symbols, with each symbol taking 9-bits according to our agreement i.e 10*9 = 90 bits.

But, also we need to take into account the additional mapping for the decompressor to know that symbol 256 refers to to be, that will take another 48 bits. So, effectively we have 90 + 48 bits = 138 bits and we achieve a compression ratio of 1.043. Although this compression is trivial, but this is a good example that supports our hypothesis.

Note: Trying to judge the compression schemes on smaller inputs is not ideal. Compression is all about saving space on large data. So, we should see if the compression scheme has the potential to scale with larger inputs. If the data to be compressed is itself very small, there is not much requirement to compress it anyways. But here, we will still stick to smaller inputs for easier understanding.

To draw some conclusion, to make a good compression algorithm, it needs to be able to find repeated patterns in the string and substitute them. But the tradeoff being, we have to somehow flow this information to the decompressor.

But is there any way we can make this even better? What if we can save the extra over-head to send the metadata to the decompressor?

Idea

One approach might be to traverse the input stream and build a dictionary simultaneously with all the information that we have so far. In other words, if the table/mapping can be built deterministically we don’t have to send that information to the decompressor and that will save us some extra overhead. The decompressor can use the same logic to build the table itself.

This is actually the hallmark of the LZ Scheme of Compression Algorithms. The LZ family of data compression algorithms trace back to the work of Abraham Lempel and Jacob Ziv, who published two influential papers in 1977 and 1978. These algorithms, LZ77 and LZ78, laid the groundwork for many variations that are still used today, including LZW, LZSS, and LZMA.

This article will focus on the LZW scheme. Terry Welch published it in 1984 (the W in the LZW) as a significant improvement to LZ78. The LZW algorithm quickly gained widespread use due to its efficiency and adaptability. It became the compression method of choice in the popular GIF image format. In 1985, Sperry Corporation (which had acquired Lempel and Ziv's employer) obtained a patent on LZW. Later, Unisys acquired Sperry and began enforcing licensing fees on the use of LZW in software. But developers weren’t happy with this but that hatred fueled the creation and adoption of other compression algorithms like DEFLATE (which is used in the gzip tool). But, the classic Unix command compress utilizes a modified version of the LZW algorithm.

It’s a little hard to explain the algorithm in a blog but I have given my best:

Getting hands dirty

Setup

The compressor and decompressor maintain a symbol table. The symbol table is initially populated with the first 256 ASCII characters. As we process the input string we will create new symbols and they will take the index 256 onwards. So, we can say symbol 256 is for to be and so on.

In the LZW compression algorithm, whenever we add something to the compressed data result, there must exist corresponding symbol in the symbol table.

We also have to maintain something called the Working String which stores a new symbol. At every step we either try to make the working string longer or add the current working string to the symbol table.

And, How do we build the working string? We use the Current Character while iterating the string. Once we add the current working string to the symbol table we reset the working string to just include the current character. So, effectively the working string is never empty except at the beginning of the compression.

Compression

Let’s consider an example string banana.

When we check if the working string is in the symbol table we append the current character at the end of it. Let’s define it as Augmented Working String (AWS) for easier understanding.

Let’s start with our first character: b.

Working String: "<empty>"
Current Character: "b"
Augmented Working String: “b”

Is AWS in the Symbol table? Yes, as we mentioned before, all the single character ASCII values are already in the table.

Now, the new working string becomes this augmented working string. And we don’t output anything in this step.

---

Let’s go to our next character: a.

Working String: “b”
Current Character: “a”
Augmented Working String: “ba”

Is AWS in the Symbol Table? No, currently we don’t have any extra symbols except the first 256 ASCII characters in the symbol table.

So, we will add this augmented working string in the symbol table with the next available index, which in this case is 256. So our symbol table now has this extra symbol:

symbol_table: {
    256: “ba”
}

At this point we will output the existing working string i.e “b”. So we got our first character in the compressed data.

---

Let’s go to our next character: n.

Working String: “a”
Current Character: “n”
Augmented Working String: “an”

Is AWS in the Symbol Table? No. So, we add it with the next available index i.e 257. Now our symbol table becomes:

symbol_table: {
    256: “ba”,
    257: “an”
}

And we output our current working string i.e “a”. So our compressed data is now ba.

---

Let’s go to our next character: a.

Working String: “n”
Current Character: “a”
Augmented Working String: “na”

Is AWS in the Symbol Table? No. So, we add it with the next available index i.e 257. Now our symbol table becomes:

symbol_table: {
    256: “ba”,
    257: “an”,
    258: “na”
}

And we output our current working string i.e “n”. So our compressed data is now ban.

---

Our next character: n.

Working String: “a”
Current Character: “n”
Augmented Working String: “an”

Is this in the Symbol Table? Well yes !! It’s index is 257. So, we move to our next step.

---

Our next character: a.

Working String: “an”
Current Character: “a”
Augmented Working String: “ana”

Is this in the Symbol Table? No. So, we add it with the next available index i.e 259. Now our symbol table becomes:

symbol_table: {
    256: “ba”,
    257: “an”,
    258: “na”,
    259: “ana”
}

And we output our current working string i.e “an” in fact the index for this symbol. So our compressed data is now ban<257>.

And our working string for the next iteration is “a”.

This is interesting to see that with this algorithm, we are even building up on the symbols that we already saw previously, in the hope that if it occurs again we can use this new symbol. This shows how compression schemes like this can benefit from repeated patterns in long input.

Now we are done with the input character, but we still have some data left in the working string. So, we need to flush them to the compressed data, so it becomes: ban<257>a.

As you can see we were building the symbol table as we were traversing the input. Another good observation is that we solved the problem of what a symbol represents. In the compressed string where we used 257 as a replacement for an; the substring an already exists in the string before the symbol has been used.

Decompression

For decompression the important objective is to build the symbol table by the time it reaches the symbol, so that it can use the substitution.

So, if the decompressor follows the same logic, by the time it reaches the symbol 257 it should be able to learn that 257 represents the character “an”. Let’s go through the decoding process too.

Now, we have the compressed data ban<257>a.

The logic to fill the symbols table remains the same.

Let’s go to the first symbol: b.

Working String: “<empty>”
Current character: “b”
Augmented working string: “b”

Since we already have the symbol in the table, we will continue and we can simply output the decompressed result b.

---

Next, our symbol is a.

Working String: “b”
Current Character: “a”
Augmented Working String: “ba”

We don’t have it in the symbol table so we will insert it:

symbol_table: {
    256: “ba”
}

And we will append “a” to the decompressed result: ba.

---

Our next symbol is n.

Working String: “a”
Current Character: “n”
Augmented Working String “an”

We don’t have it in the symbol table so we will insert it:

symbol_table: {
    256: “ba”,
    257: “an”
}

And we will append “n” to the decompressed result and it will be ban.

---

Next our symbol is 257 .

As soon as we come across a symbol we check if there is any substitution, and it does. We have “an” for 257. So, instead we will replace 257 with “an”. And treat each char individually like we were doing so far.

---

So, the next symbol is a (from the "an").

Working String: “n”
Current Character: “a”
Augmented Working String “na”

We don’t have it in the symbol table so we will insert it:

symbol_table: {
    256: “ba”,
    257: “an”,
    258: “na”
}

And we will append “a” to the decompressed result and it will be bana.

---

Then our next symbol will be n (We will still have a “n” from the substitution we performed earlier)

Working String: “a”
Current Character: “n”
Augmented Working String “an”

We have the symbol in the table, so we will continue with that augmented working string.

And we will append n to the decompressed result and it will be banan.

---

Now our final symbol is a.

Working String: “an”
Current Character: “a”
Augmented Working String “ana”

We don’t have it in the symbol table so we will insert it:

symbol_table: {
    256: “ba”,
    257: “an”,
    258: “na”,
    259: “ana”
}

Finally we will have our decompressed result: banana.

Also notice how we arrived at the same symbol table that we saw during the compression process. This is the beauty of the LZW compression algorithm.

Edge Case :)

But there is a small caveat !! Try to follow this same algorithm to compress the string another_banana and then decompress it. While compression you won’t find any issue and if you do it correctly you should get another b<256><265> as the compressed data. But there will be a small edge case when you perform the decompression.

But it can be resolved with a little trick.

Conclusion

"The art of compression is the art of finding patterns."

That was all for this blog. While LZW marked a significant milestone in compression algorithms, it's important to note that data compression continues to evolve. Modern techniques often combine LZW with other algorithms or use more sophisticated dictionary-building mechanisms to achieve even higher compression rates. Just like Google’s DeepMind AI AlphaDev found “faster” sorting algorithms, maybe someday they will come up with better and more efficient compression algorithms, until then let’s add more creativity to this field.

I will not cover any compression algorithm in this blog, will keep that for future blogs. But what I will do cover is if you want to create your own compression algorithm what should the starting point be like.

Let's step back a little

“What hath God wrought”

So our protagonist is Samuel F Morse, a successful portrait artist. But he got hooked on the idea of an electric telegraph way back in 1832. The problem at the time was that other telegraph systems needed multiple wires – a mess to manage over long distances. So Morse came up with the idea of sending electrical pulses through wire with an electromagnet attached at the end of it. A springy metal bar (the armature) is attached to the electromagnet. When the magnet pulls, the armature makes a satisfying "click" as it hits a contact.

After years of tinkering and getting partners on board (shoutout to Alfred Vail!), Morse strung a telegraph line between Washington D.C. and Baltimore. In 1844, he tapped out the now famous message, "What hath God wrought!" It wasn't just a test; imagine the hype, like sending the first ever tweet!

Morse's real genius wasn't just the tech; it was the code itself. Instead of a whole alphabet, he figured you could represent everything with just dots and dashes. He assigned the most common letters the shortest codes.

The most frequently occurring letter in the English alphabet is the letter "e," and its corresponding Morse code representation is a single dot. Similarly, the second most common letter, "t," is represented by a single dash in Morse code.

But letter’s like "z" are represented by two dashes followed by two dots. So if you had to say something like "Jazz" in Morse it would become

dot dash dash dash dot dash dash dash dot dot dash dash dot dot

But here is my question, can we call this a “compression” algorithm? Like in the last case is it performing any compression?

So, let’s understand what qualifies as a compression algorithm.

What is a compression algorithm?

Let’s say s is a string of bits of length n, and then C(s) is a function that returns a stream of bits which is a "compressed" version of s.

For a compression algorithm to work, we should be able to decompress it. It would be stupid to have an algorithm that just compresses stuff and there is no way to get back the original.

Hence, there should also exist an inverse of C(s), let’s call it D(s) such that D(C(s)) = s.

One small thing to note here is that after decompression if we get the exact uncompressed data it’s called loss-less compression.

There is another kind of compression where we expect that some information can be selectively discarded that is of less importance or less noticeable to the human eye or ear to achieve much smaller file sizes. They are called lossy compression (e.g. JPEG). Let’s just focus on loss-less compression for now.

What makes a compression algorithm good?

So, how do we know if it's a good compression algorithm? Let’s define a formula for that.

Let’s call it the compression ratio: the length of the original data divided by the length of the compressed data.

$$\text{Compression Ratio} = \frac{\text{Length of Original String}}{\text{Length of Compressed String}}$$

$$\text{i.e } \text{CR} = \frac{|s|}{|C(s)|}$$

So, if our original data is 100 bits long and after compression it becomes 50 bits long then the compression ratio is 100/50 i.e 2.

Case #1: When CR = 1

If the compression ratio is 1 then it means that the length of the data before compression is the same as the length of the data after compression. Well, then what’s the point of calling it a compression.

Case 2: When CR > 1

This is only possible when the compressed data has length less than original data. Which is supposed to be the case right? Well here is the catch.

Let’s say we have a compression algorithm C(s) which has a compression ratio greater than 1.

In the worst case:

If |s| = n, the |C(s)| will be n-1 i.e we are saying that the compression algorithm is only able to save 1 bit of space which sounds very inefficient but let’s hold on to that.

What if we apply the same compression algorithm again? Then

$$|C(C(s))| = n-2$$

If we continue doing that:

$$|C(C(C(s)))| = n-3$$

Now you see where we are leading to? Effectively after applying the same compression algorithm again and again we will be able to compress any data to 0 bits. Which is impossible, and here lies the fallacy.

Example:

Let’s take an example to understand this deeper. Let’s say we want to compress 2 bits of data. And we want to achieve at least some compression so the compressed data should be of 1 bit.

So, one possible input is 10 and we want to compress it to 1 and let’s say we want to compress 01to 0. But? What about 11or 00? We cannot compress it to 1 cause we are using 10 for that, nor can we use 0 cause we are using 01 for that.

We can actually prove this with a theorem which is one of my personal favorites: Pigeonhole Theorem.

So, for some integer n >= 1 and consider the set S of 2^n strings of length exactly n. Let's define

$$S_c = \{C(s):s \in S\}$$

$$\text{Note that, } |S_c| = |S| = 2^n$$

So, if every string in S_c has length at most n-1 then size of S_c can be at most:

$$|Sc| \leq 2^0 + 2^1 + ... + 2^{n - 1} = \sum_{i=0}^{n-1} 2^i = 2^n - 1 \text{ (contradiction)}$$

$$\text{Therefore, there exists } s \in S \text{ such that } |C(s)| > n - 1.$$

Conclusion

The conclusion from this proof is that "Compression is a Zero-Sum game". For some input to be compressed to something smaller there will be some input that will be compressed to something larger.

Don’t believe me? Try this online tool (link) to compress the following text using DEFLATE, the algorithm behind gzip which kind of runs the entire internet as of now.

ykjNyclXKMnILFYAosSivMQkheLUPIWMUXFyxAEAAAD

You will see the compressed string is larger than the original string.

Whereas if you give an input something like a text shown in the image below; we get a really good compressed result. Most of the compression algorithm used nowadays are very good at getting rid of redundancy which occurs a lot in our day to day lives. Which is why some characters in the English alphabet has higher frequency than others.

And this is the truth, most of the compression algorithms focusses on compressing inputs that are more generally available than any random stream of noise.

Therefore if you are planning to create a new compression algorithm you should ensure that your algorithm is able to achieve good compression of actual data at the expense of bad compression of random noisy data because we rarely care about them.

Getting back to where we started...

So, getting back to the question we started with, can something like Morse code be considered a compression algorithm? Well, in my opinion, it sure can. We can represent letters that show up a lot with less effort, but we have to make do with longer codes for the letters that don't show up as often, like Z, Q, and so on. And that's basically what compression is all about, as we just saw.

That spirit of squeezing the most meaning into the fewest bits? That's the heart of compression, even with today's fancy algorithms.

As software developers, our daily routine revolves around crafting programs in familiar languages such as JavaScript, C++, Python, Rust, Java, and more. We immerse ourselves in mastering the syntax of these languages and then put the keywords such that it achieves our business logic. But the question is: How does the computer comprehend this diverse array of languages?

Most of the languages that we usually come under the category of High-Level Language. They are high-level in the sense they provide a higher level of abstraction and allow us to write more "human-readable" code.

Let's take a look at this super fancy C Code, which most of you have written at some point in your career as a developer:

#include <stdio.h>

void main() {
    printf("hello world");
}

A computer doesn't understand these words like "printf", "void", "main". They are meant for us humans to write code faster and more efficiently. Once the compiler compiles the code into Machine Language it looks like this:

Note: This assembly code is for Linux using the x86 architecture. The actual machine code can vary based on the target system.

You can notice the words like push, mov, call, nop, pop etc. These are commands that the CPU can understand and they all belong to something called an instruction set.

It serves as an interface between the hardware and the software, allowing us to write software that can run on a specific architecture.

The instruction set of a processor typically includes a variety of instructions that can perform operations such as arithmetic, logic, data movement, control flow, and more. Here are some common types of instructions found in an instruction set:

1. Arithmetic Instructions:

   • Addition (ADD), subtraction (SUB), multiplication (MUL), division (DIV), etc.

2. Logic Instructions:

   • Logical AND (AND), logical OR (OR), logical XOR (XOR), bitwise operations, etc.

3. Data Movement Instructions:

   • Load (LOAD), store (STORE), move (MOV), etc.

4. Control Flow Instructions:

   • Jump (JMP), conditional jump (JZ for zero, JNZ for not zero, etc.), subroutine call and return, etc.

5. Comparison Instructions:

   • Compare (CMP), test for equality (EQ), test for greater than (GT), etc.

6. Input/Output Instructions:

   • Input (IN), output (OUT), system calls, etc.

The specific instructions available in an instruction set can vary between different CPU architectures. For example, x86 and ARM are two different instruction set architectures commonly used in today's processors, and they have distinct sets of instructions.

In today's world, we have it easy, we just have to write the high-level code and the compiler translates the high-level code into the machine code instructions that the processor understands, based on the instruction set architecture.

Although, understanding the instruction set is essential for low-level programming, optimization, and when working with assembly language or writing programs that directly interact with hardware.

In this blog, I will talk about one specific kind of Instruction Set which I found really fascinating.

OISC

OISC stands for One Instruction Set Compiler (OISC). It is also called Single Instruction Programming Language or Ultimate Reduced Instruction Set Computer (URISC).

As the name suggests, this instruction set has just one instruction.

Okay, but you might think, what's so special about that? Anyone can create an instruction set that has just one instruction and call it an OISC.

Well, not really. Most mainstream programming languages are able to solve such a wide variety of problems and computations because they are all Turing Complete. If a programming language is Turing complete, it means that it can theoretically compute anything that is computable, given enough time and resources. (I'll do a separate blog on Turing Completeness, but for the time being, you can understand it like a test for programming languages)

Here is a movie recommendation: "The Imitation Game" based on the life of Alan Turing.

The OISC that I am going to talk about is also Turing complete, but the best part is unlike the Turing machine it doesn't need to have an infinite memory model. Hence, it is equivalent to a real computer than a Turing machine. And the instruction is subleq (Subtract and branch if less than or equal to zero).

For any instruction to be Turing Complete (or in simple words be able to compute most complex computations) it needs to have some sort of conditional branching. You can imagine this to be something like

if <condition>
goto <branch>

Now, these machines can be categorized into 3 types:

1. Bit manipulating machines

2. Transport triggered machines

3. Arithmetic-based machines

mov is another such instruction. The mov instruction copies the data item referred to by its second operand (i.e. register contents, memory contents, or a constant value) into the location referred to by its first operand (i.e. a register or memory).

mov a, b

//it is equivalent to
*a = *b;

This mov instruction is Turing complete and comes under 2. Transport Triggered Machine. There is an amazing repo that showcases the fact that any C program can be compiled into a program written only using mov instructions. You can check it out here.

On the left, you have the code in Assembly, and on the right, you have the same code which does the exact same thing but it only contains the instruction mov.

Arithmetic-based Turing-complete machines use an arithmetic operation and a conditional jump. This is what subleq does.

Let's simplify the subleq operation. If we do subleq a, b, c it comes down to:

Mov[b] = Mov[b] - Mov[a];
if (Mov[a] <= 0) goto c;

So, you can see there is a basic arithmetic operation and one conditional branching, (we already mentioned that it is required to be Turing complete). There is a syntactic sugar for the same which goes like this

 subleq a, b   ; Mem[b] = Mem[b] - Mem[a]
               ; goto next instruction

Here we basically drop the c which means we aren't branching anywhere but rather moving on to the next instruction. Now the big question. Is this instruction enough? Well as it turns out. Yes, it is.

Let's look at some programs using this instruction:

Example 1

; initially *z = 0
subleq a, z
subleq z, b
subleq z, z

let's break this down

//initally
*z = 0;

//line 1
*z = *z - *a // since *z=0, this line boils down to
*z = -*a

//line 2
*b = *b - *z // *z = -*a, this line boils down to
*b = *b - (- *a)
*b = *b + *a

//line 3
*z = *z - *z
*z = 0 //we started with *z=0 and ended with *z=0

What we see is *b = *b + *a i.e. this 3-line instruction performs addition.

The reason we use (*) before a, b, z is because those represent memory addresses and *a represents the value in that particular memory block

The reason the last line is important is because we have to ensure we do not end up changing the values that we are not dealing with, here we were just dealing with a and b. z was initially 0 and finally should also be 0. In other words, there shouldn't be any side effects.

Not just addition other instructions can also be implemented with subleq instruction as well.

Example 2

; initially *z=0
subleq b, b
subleq a, z
subleq z, b
subleq z, z

You can work these out and you will come to the conclusion that it ends up assigning the value of a to b.

Or you can apply 200IQ and notice that this is exactly the same as the previous one, only a new instruction subleq b, b was added to the beginning, we know the other 3 lines implement this *b = *b + *a and the first line is basically *b=0 , hence what we finally get is *b= 0+*a i.e. *b=*a.

Example 3

subleq z, z, c

This is very basic, it will just jump to the branch c.

The interesting thing is this programming language with this one instruction is equally powerful to any other programming language like C++, Java .etc.

It may not be very efficient and easy to write but in the end, all of these come under the class of Turing Complete. Yet there are some interesting benefits to employing a one-instruction computer. For example, hardware-level functionality is simplified when implemented around a single instruction. This greatly simplifies the underlying implementation, as the same functional element is repeatedly used to form the processor core. Another advantage is that since all the instructions are the same, the instruction decoder circuitry and complexity can be eliminated.

Why bother about OISC?

OISC architectures provide an excellent paradigm for implementing traditional von Neumann computers using non-traditional materials. Simply put, a practical computer can be built by massive scaling of simple, single instruction elements. Embracing OISC challenges us to strip away the layers of abstraction, inviting a deeper exploration of the foundational principles that govern all programming.

I hope you found my blog informative. If you have any feedback share it in the comments. You can sign up for the Hashnode newsletter to get notified every time I post a blog. Learn more about me at arnabsen.dev/about. Have a nice day.

I have been recently diving into the world of Distributed systems and I came across a rather interesting paper: "In Search of an Understandable Consensus Algorithm". What caught my attention was the paper's primary objective of creating a consensus algorithm that is easy to understand. Typically, in the realm of new research, we tend to emphasize factors like efficiency, correctness, and conciseness while overlooking how accessible a concept is for others to grasp and apply in practical scenarios.

We often neglect how easy it for someone is to grasp the concept and use it to build practical stuff or solve real problems. Sometimes we do the opposite, if something works but is very hard to understand and is confusing we consider it to be very "clever" and end up rewarding complexity. That's why I appreciate the approach taken by the authors, Diego Ongaro and John Ousterhout, in prioritizing the algorithm's comprehensibility.

With many major corporations transitioning to distributed systems, having an algorithm that is both easy to understand and intuitive can empower developers to create systems more seamlessly and devise more efficient implementations."

So, what is Consensus

Let's start with a simple analogy:

You have been invited to a party and you have dressed up all nice and good. Then you ask your 4 friends if you are looking good (if you don't have so many friends just imagine you do, at this point, you must be good at it). Two of them say that you are looking good, and the rest say you are not. What are you supposed to do?

But if 3 of them said that you do look good (quite unusual right?) then you can go with the majority decision.

This is consensus !!

In distributed systems, we know that we have multiple nodes, and consensus refers to coming to an agreement.

Consensus simply means to get all the nodes in distributed computing to agree on a common value or decision, despite the presence of faults, delays, or unreliable communication. This agreement is crucial for ensuring that the distributed system functions correctly and consistently.

Even though it sounds simple, many broken systems have been built in the mistaken belief that this problem is easy to solve.

There is a pattern in most of the fault-tolerant systems. For example, in MapReduce, the computation is replicated but the whole thing is controlled by a single master. Similarly, we have GFS which replicates the data and has a single master to determine who the primary is for the piece of data. The benefit of having such a single master system is that there will be no disagreement. But at the same time, it makes it a single point of failure. So we need some sort of consensus to avoid such failures.

History of Consensus Protocols

Well, now that I think "history" is an overstatement cause there is just one very popular protocol called "Paxos" which has been the gold standard for a long long time. First submitted in 1989 by the Leslie Lamport, the Paxos protocol is named after a fictional legislative consensus system on the island of Paxos in Greece.

FYI Leslie Lamport was the creator of LaTeX. He also won the Turing Award which is like the Nobel Prize for Computer Science.

The problem with Paxos was it was really hard to wrap our minds around. It only solved a portion of the problem and hence made it difficult to build systems around Paxos.

Raft Algorithm

As mentioned before the authors' primary design goal when building Raft was "understandability".

Now this doesn't mean that Raft is very very easy to understand, there are still a lot of intricacies and edge cases to cover, but I will try my best to give a brief overview of the entire thing. So let's start.

What are we dealing with?

We have a client, which can be a user or a service etc. And it tries to fetch some data from the server we have which is nothing but a cluster of nodes.

Now, to ensure consensus we need the nodes to agree on a particular value and then send that to the client.

How do we ensure that all the nodes have the same value? Raft achieves that through something called a "Replicated Log".

It ensures that all the commands (or operations) that need to be performed are replicated across the nodes in the same order. So, let's say you have a variable X initially, the value is 0 and then you perform some operations like:

• add 5

• mul 10

• sub 15

If these commands were logged in the same order then the final value of X will be the same for all the nodes assuming the fact that the operations in the nodes (which are also referred to as state machines) are deterministic.

So, when a new command gets proposed, it first gets logged on all the nodes (at least most of the nodes) at the end of their logs. Only when the log gets properly replicated, does the node perform its computation and send the result to the original client.

The idea behind this is if the logs get replicated properly the cluster can continue functioning as long as the majority of the nodes are up (Yes, it's more like democracy, the majority wins).

Who will instruct the nodes on what to do?

Again similar to democracy, here also we have a leader node elected by other nodes. The leader node is responsible for talking to the client, giving commands to the other nodes, etc.

Before diving deep into the functioning of the leader let's see how the nodes become a leader.

Server States

The nodes/server have 3 states:

• Follower

• Candidate

• Leader

In normal service, there will be just one Leader and all the other servers will be Followers.

Followers are passive, they issue no requests and can only respond to requests from Leaders and Candidates.

The Leader handles all the client requests. It also sends regular heartbeats (empty requests) to the Followers to maintain authority. More like it's saying "Don't get too excited, I am still your leader".

If a Follower receives no communication over a time period (called election timeout) it assumes that no Leader is available and starts the process of Leader Election.

Leader Election

Let's talk about term before diving into Leader Election. To have consensus we need to have a mechanism to detect obsolete information. Like if a particular server was a leader and then it stopped responding, and a new leader was appointed. We need to ensure that the other servers don't take instructions from the old leader anymore. Raft achieves this by dividing the time into terms or arbitrary lengths called term.

Each term starts with an election of the leader and then if the election is successful the leader rules for some period of time which is represented in green in the above diagram.

You can see how this is similar to the administrative process of a country where each leader has a term like 5 years and then post that election happens to elect a new leader.

In Raft, now the interesting thing is there is no global concept of term among the servers. Each server starts with its own terms.

When the election process starts the follower increments its current term index and becomes a candidate. It votes itself and then sends specific requests to the other nodes requesting to be a leader in other words asking for their vote. This happens by a special kind of request called RequestVote RPCs.

Whenever the server communicates with the other it includes what it thinks the current term index is. Then the other server responds by including its term index.

Now if there is a mismatch the server with the lower-term starts having an identity crisis and falls back to being a follower. And the other server just ignores it.

A Server is entitled to vote only one server and that happens on a first come first serve basis. So by the end of the process, the Candidate server with the most votes becomes the leader and then sends heartbeat messages to all of the other servers to establish its authority and prevent new elections.

Here I had a doubt, what if all the server realizes that the leader no longer exists or the current term is over, votes themselves, and requests a vote at the same time? They will just end up waiting indefinitely to get majority votes. Or what if two candidates receive the same number of votes (this is called a split vote)?

Raft solves this by using randomized election timeouts, which means election timeouts are chosen randomly from a fixed interval (e.g., 150-300 ms). This way in most cases only one server will timeout at a time, become a candidate, and will request a vote before the other server times out. Even in case of a split vote, each candidate restarts its randomized election timeout at the start of an election and it waits for that timeout to elapse before starting the next election. This randomness drastically reduces the chances of another split vote.

How does the normal operation happen?

Let's go through how a normal operation will take place with consensus in Raft.

First, the client sends a command to the leader. Wait, how will the client know who the leader is? Well, the client can send it to any server and if the server is a follower it will simply redirect it to the leader.

The Leader appends the command to its log and sends another request (called AppendEntries RPC request to all its followers). The followers on receiving this request just append the new log to their existing logs record. Note, at this point they just append it and don't actually perform any computation. It's the leader who decides when it is safe to apply the log entry i.e. when the entry has to be committed. This is actually called 2PC (Two Phase Commit).

The append request contains two identifiers, one is the term index which I already mentioned before, and another is the log index which is the position of the new entry in the log. The leaders use these two index values to determine if the majority of the servers have the logs up to date and then issue a commit.

In any case, the logs become inconsistent, the leader forces the followers to duplicate the leader's log. To bring a follower’s log into consistency with its own, the leader finds the latest log entry where the two logs agree, deletes any entries in the follower’s log after that point, and sends the follower all of the leader’s entries after that point.

This is how the Leader ensures that every server (or at least the majority) has the same order of the logs i.e. consensus is reached, performs the computation, and then responds to the client.

This is how we can build a system with high data consistency with Raft and many distributed consistent database like CockroachDB uses Raft.

This was a very brief overview of Raft. There are many more aspects to this that couldn't be covered in a blog and I urge you to go through the original paper. I hope you found my blog useful. If you have any feedback share it in the comments. You can sign up for the Hashnode newsletter to get notified every time I post a blog. Learn more about me at arnabsen.dev/about. Have a nice day.

MapReduce is a programming paradigm that helps us perform large-scale computation across computing clusters.

DISCLAIMER: In this blog, I will take a very simple overview of this topic. If this blog actually spikes your interest and you want to delve into the technical nitty-gritty then I would urge you to read the original paper: "MapReduce: Simplified Data Processing on Large Clusters" (It was written by the Top-Gs in the programming world: Jeff Dean and Sanjay).

Background

Let's start with a little bit of background.

Currently at Google, I am working on a project that requires me to process a large amount of logs, perform some computations, and then write the computed results in a database. This has to happen on a daily basis and the output is then consumed by downstream teams for further operation.

One might think, what's the big deal in that? Just take the log file as input, implement the logic, and write the results to an output file. That's what we have always been doing.

Well, one thing we should take into consideration is the size of the file that we are dealing with. It is actually in the multiple of Terabytes. Taking our traditional path for problems like this would cause a lot of issues like:

1. Performance Issue: the entire thing would probably take a whole lot of time and it can have a lot of potential bottlenecks.

2. Memory Issue: How are we even going to load the entire TBs of data into memory? We have to implement some partitioning ourselves to do it effectively.

3. Scalability Issue: Very difficult to actually scale it to multiple machines.

4. Fault Tolerance: Let's say you somehow managed to overcome all the problems above, what if at the last moment, something crashes, how are we going to recover?

This is why Google came up with the MapReduce paradigm. There is also an open-source implementation of the same called "Apache Hadoop".

Link: https://github.com/apache/hadoop

What's MapReduce?

You must have already noticed that the name MapReduce is made up of two parts: Map and Reduce.

You might have often used these functions in Python, JavaScript, etc. Something like this:

Map Function

The Map function applies a given function to each element in a collection (such as a list or array) and produces a new collection of transformed values. It is often used to perform element-wise operations on data.

Reduce Function

The Reduce function takes a collection and combines its elements into a single result by repeatedly applying a binary operation. It reduces a collection to a single value.

Here is a visualization to help you understand it better:

These two methods are fundamental to the Functional Programming Paradigm. I came across these when I started learning functional programming languages like OCaml, Lisp, and Rescript. The main idea is to process the data in a functional, declarative manner.

Inside the map function, we provide the transformational logic and in the reduce function we provide the aggregator logic, and no need to specify any logic for the looping, etc.

A large number of problems can actually be split into two broad logic: Map phase and Reduce phase. Some examples mentioned in the paper are:

• Distributed Grep

• Counting URL access frequency

• Reverse Web-Link Graph

• Term-Vector per host

Let me give a simple example, let's say you are working on a Music Recommendation Algorithm. Before you can actually use the data for the recommendation algorithm you will have to do some pre-processing. Usually, logs contain everything, and we may not need all of them, so we have to modify the logs to a form that is relevant to us. To further simplify things, let's say we just need the Genre of the music from the log data. We can use a map function for that.

Then let's say our recommendation algorithm also needs the number of music the user has listened to for each genre. A reduce function can do this. Here is a sample Python code to help you understand better:

from functools import reduce

logs = [
  {
    'name': 'Shape of You',
    'singer': 'Ed Sheeran',
    'genre': 'pop'
  },
  {
    "name": "Uptown Funk",
    "singer": "Mark Ronson ft. Bruno Mars",
    "genre": "pop"
  },
  {
    "name": "Bohemian Rhapsody",
    "singer": "Queen",
    "genre": "rock"
  },
  {
    "name": "Friends in Low Places",
    "singer": "Garth Brooks",
    "genre": "country"
  }
]

def update_frequency_count(freq_dict, num):
    if num in freq_dict:
        freq_dict[num] += 1
    else:
        freq_dict[num] = 1
    return freq_dict

# Map operation
# {name, singer, genre} -> genre
genres = list(map(lambda log: log['genre'], logs))
# genres: ['pop', 'pop', 'rock', 'country']

# Reduce operation
frequency_count = reduce(update_frequency_count, genres, {})

print(frequency_count)
# {'pop': 2, 'rock': 1, 'country': 1}

Both of these functions are deterministic. Also, the map operation on one element doesn't depend on the result of another operation. In other words, they are independent of each other as well.

So we can parallelize the operations of Map and Reduce. We can further extend the same thought process by breaking the input data into several chunks of data and having one map function for each chunk of data. This can also solve our Fault Tolerance issue, if something breaks on operation, it won't affect the others cause now they are all running independently in a parallel fashion. We can just re-run the broken process with the small chunk of data. You can see the big picture now as to why the Map and Reduce functions were considered specifically to solve this kind of problem.

To quote the paper:

"Our use of functional model with user specified map and reduce operation allows us to paralelize large computations easily and to use re-execution as the primary mechanism for fault tolerance."

Let's dive into the programming model deeper and then we will take another very popular example to understand the programming model.

With MapReduce, we usually deal with key/value pairs cause it's pretty generic. The Map takes an input pair and produces a set of intermediate key/value pairs. The MapReduce library groups all the intermediate values associated with the intermediate key I and passes them to the Reduce function.

The Reduce function then accepts the intermediate key I and a set of values for that key and then merge together values to form a possibly smaller set of values (it can be one as well).

Stages of MapReduce

Now let's dive into the actual working of MapReduce.

Splitting the Input

The input data is actually partitioned into smaller chunks, typically 16 MB or 64 MB. Let's say the number of chunks/splits is M. Then there will be M map operations in total.

Why 64MB? That can be a whole separate topic of discussion but you can think of it like these large amounts of data are usually stored on local disks of the machine that make up the cluster. Google File System (GFS) divides each file into 64MB blocks, so having the input data of the same size will ensure that the map function just has to read data from one block thus reducing I/O latency.

The Reduce invocations are also distributed by partitioning the intermediate key space into R pieces using a partitioning function (we will discuss this later in this blog). The number R is also set by the user.

Building the army

When we start the MapReduce function, it first starts a special process called "master". The master as you might have already guessed does a lot of management-related work like assigning Map and Reduce operations, handling exceptions, etc.

The program also starts up many copies of the program on the other machines. These are done by the fork system call. All the other programs except the master are called "worker nodes" or simply "workers". Each of the workers will be assigned a task from the M map tasks and R reduce tasks which is determined by the master.

Map Stage

A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory. Since the map operation doesn't depend on any other worker, all the map workers can parallelly perform this, thus allowing us to linearly scale the performance of the task of extracting data.

Before the workers with Reduce task assigned can work, these buffered pairs are written to the local disk which is partitioned into R partitions by the partitioning function. Once the partitioning is completed, the master is informed, who then signals the location to the Reduce workers to get the rest of the job done.

Reduce Phase

The Reduce workers then read the intermediate data by using RPC calls. Once it has read all the intermediate data, it performs a sort based on the keys. Why?

So that all the occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task.

Let me be frank, it took me some time to wrap my head around this, and I am still not sure if I got it correctly. So if you think my understanding is wrong please don't hesitate to correct me.

Previously we saw that we have M map operations and R reduce operations. Sometimes the same worker is assigned multiple Reduce jobs. Sorting the data makes it easier for the worker to distinguish between the new intermediate data and the old intermediate data. Once the worker finds that the partition has some new data it starts the reduce job again.

In the reduce step it also performs a merge of all the values corresponding to a particular key. So, sorting the keys makes it easier to merge. This will get clear once we go through the steps again taking the example of a word counter.

Finally, the reduce worker goes through the sorted intermediate data and for each unique intermediate key, it passes the key and the set of intermediate values to the user's Reduce function. The output is then appended to the final output file.

Last stage

When all the Reduce workers have completed execution, the master passes control back to the user program. The output of MapReduce is stored in the R output files that the R reduce workers created.

Partitioning Function

Let's understand what the partitioning function is all about. As a user, you can define the number of output files you want your data to split between. That number is denoted as R and MapReduce will assign the same number of Reduce workers. So if R = 10, it means that MapReduce will partition the data from the map workers into 10 splits. Each split will be assigned to one Reduce worker which means that there will be 10 Reduce workers. And then each Reduce worker will write the output to a file so the user will have their processed data split into 10 files.

By default, the partitioning algorithm that is used is hash(key) mod R. This usually gives a fairly balanced partition. In some situations, you might want to use a custom function as well. Taking the example from the paper, if your keys are a bunch of URLs and you want them to be partitioned based on the hostname, then you can use a function that extracts the hostname and then applies the hash and mod. So something like: hash(getHostName(key)) mod R.

Example

Let's take the example of the Character Count. We have an input text file with a bunch of words and we have to calculate the frequency of all the characters appearing in the input.

So, first, we have the input data being split into M partitions. In our case M = 5. So the master worker will assign 5 map workers to map each of these input partitions using the user-defined map function which can be something like this.

map(String key, String value):
    // key: document name
    // value: document content
    for each char c in value:
        EmitIntermediate(c, "1");

So, for the text "apple" our map function will return:

a: 1
p: 2
l: 1
e: 1

Now, the intermediate output will be "shuffled" by using our partitioning algorithm. Since we are going to consider only the lowercase English alphabet we can have R = 26 for our partitioning algorithm. That way each output will contain the count of one letter. Simply put, our partitioning function will group all the a's and then pass it to our reducer function.

So, in our case, the reducer function will get the following input:

a: [1, 3, 1, 1]

Since we want the total count, we can have a reducer function like this:

reduce(String key, Iterator values):
    // key: a word
    // values: a list of counts
    int result = 0;
    for each v in values:
        result += v;
        Emit(AsString(result));

This will give us the final result: a: 6.

So this was a very basic overview of the MapReduce workflow. Keep in mind, that this is not a very generic parallel programming paradigm but only applies to problems that fit into these map-reduce paradigms. But from my work experience, I have seen that a lot of the data analysis work does actually fall into this category. Otherwise, why would Google come up with this in the first place xD.

There are many more technical concepts of MapReduce that I didn't cover in this blog. Do go through the research paper for an even more in-depth analysis. In case you have any doubts, feel free to ping me or put them in the comments.

I hope you found my blog useful. If you have any feedback share it in the comments. You can sign up for the Hashnode newsletter to get notified every time I post a blog. Learn more about me at arnabsen.dev/about. Have a nice day.

Heads up: I initially wrote this blog at the end of my 3rd year, now that I have successfully graduated I figured why not take a stroll down memory lane and spill the beans on my last year too. Unlike my usual techy blogs, this one's more like the adventure of a super curious nerd trying to do it all! 😅

Hi there 👋

Let me start with a brief introduction. I am Arnab Sen, currently, a final year undergraduate at IIEST Shibpur pursuing my graduate with B.Tech in Computer Science. In this article, I will share my 4 years of experience as an undergrad exploring various technologies. It includes:

1. What technologies have I learned? When? Why? How? resources that I referred to.

2. My journey from CP → CTF → Open Source → Google Intern → Multiple Startups → Google (presently).

3. The various communities I am currently a part of. And many more...

Before Joining College

As a kid, I have always been into computers. I would always tinker with the PC that I got in my 5th standard. And when I say "tinker", I mean breaking stuff by installing pirated games that had a lot of viruses and you know what I mean 🤷‍♂️.

So, every month or so my dad had to call this Tech guy who would then "format" my PC and install a pirated Windows 7 and charge my dad 500 Rs. At one point, I got pretty frustrated because my dad would restrict my PC usage and at the same time felt a bit sad that he had to pay 500 Rs every time for my misdeeds (to a kid 500 was a lot of money). So I started looking for a free alternative. That's when I came across Linux for the first time.

I remember installing Ubuntu 14 by myself and I felt like I am going to become the next Bill Gates. To be honest, on the first try, I ended up wiping the entire hard disk. So it was actually after a bunch of tries that I was able to successfully install it. This kind of sparked my interest in computers and I ended up taking CS as my stream in College.

I even wrote another blog sharing my very initial days of programming here (but don't jump there directly, first complete this one and maybe then you can decide if it's worth it).

1st Year

By this time I had very basic knowledge about C++ since we had it on our CBSE Boards. And the first thing that I was exposed to was Competitive Coding. I practised my Data Structures and Algorithms from:

• MIT 6.006 Introduction To Algorithms.

• CLRS (The Bible of Algorithms and Data Structures) and its solutions.

For problem-solving, I used to refer to:

• Hackerrank

• Codeforces

• Codechef

• Atcoder

• CSES

• Project Euler

My main motive at that time was to prepare for ICPC and improve my general coding skills and my knowledge of C++.

By the end of the 1st year, I became pretty good at CP and even reached Experts in Codeforces. But I started losing interest in this field, cause they were mainly about solving abstract problems. I wanted to tackle some real-world problems and came across Cybersecurity.

2nd Year

This is when COVID happened, I had a lot of free time to explore. With my interest in CP declining, I wanted to know more about computer internals and its fundamental workings. So I started solving CTF Challenges.

Since not many people may know about this, let me explain CTFs. CTF stands for Capture The Flag. You will be given a security challenge, it can be an encrypted text or a vulnerable website. You have to bypass the security or decrypt the ciphertext to get a token which is called the flag. Finding the flag wins some points and finally, the team/individual with the most points wins. This category of CTF is called Jeopardy style.

There is another style of CTF called Attack-N-Defense. Here we have two teams one Red team and another Blue team. The Red team's job is to break into a machine/system by finding loopholes and security flaws, while the Blue team's job is to fix those loopholes and patch those vulnerabilities before the Red team is able to enter the system.

The reason these CTF challenges intrigued me was that they are based on more real-life scenarios. To solve CTF challenges I had to learn:

• Assembly Language

• Reversing

• Binary Exploitation

• Web Vulnerabilities like CSRF, XSS, SQL Injection

• Cryptography like RSA, Symmetric and Asymmetric key encryption

• Computer Networking dealing with network packet capture

• Steganography, learning how PNGs work, how zip compression works, etc

• Writing scripts in bash, python, C .etc and many more

My knowledge of computer fundamentals grew and I learned how to use tools like gdb, tcpdump, burpsuite, etc.

With rigorous practice and participating in a bunch of CTF challenges my team (which included me and my friend) ranked 67th in CTFTime in India.

There are no one-place resources for CTFs cause it is a huge domain. To even understand the vulnerabilities of a particular technology you have to be proficient in that technology first. For example, to understand the weakness of RSA, one needs to know how RSA works and what is the significance of exponent (e) and the primes (p and q).

But these are some of the YouTube channels I followed:

• LiveOverflow

• John Hammond

• ippsec

• GynvaelEN

However the best way to learn is to participate in the CTF challenges and then read the writeups or have discussions with other teams after the CTF is over.

By the end of 2nd year, I got into a fellowship program called Pupilfirst Coronasafe Fellowship. I was in the top 24 out of 50k applicants. So, for the rest of the 2nd year, I was learning technologies like Ruby on Rails, and Rescript React and started my open source journey. As a Pupilfirst Coronasafe Fellow, I contributed to two open source health care projects called Arike and Life. Through this experience, I learned about:

• Building and designing software, the workflow of the applications, and understanding the user experience

• Pair programming, is a cool way to learn and work. Read more.

• The importance of networking and having 1:1s with industry experts

• Open-source best practices

3rd Year

This is when my internship prep started. I got rejected by Microsoft for both Engage program and the on-campus drive. It was a bit disheartening.

But then I came across Summer of Bitcoin. It was an open-source opportunity and since I knew open-source won't dishearten me I applied. And guess what? I got selected. I was among the 51 mentees for the first ever batch of Summer of Bitcoin. My project was to improve support for User Statically Defined Tracing (USDT) in Bitcoin core by adding new tracepoints and scripts, leveraging eBPF (extended Berkeley Packet Filter) technology. But what's more interesting is that during this mentorship, I had to work with:

• C++ (which I did in my 1st year for CP)

• gdb for debugging (which I did during my 2nd year for CTFs)

• Github and Open Source practices (which I gained from the coronasafe fellowship experience)

My point is you might not see a direct advantage of what you are learning right now, but knowledge never goes to waste.

There is this very popular story of Steve Jobs. Jobs took a calligraphy class at Reed College based on campus posters he saw after dropping out. He knew the class would earn him no credit towards a degree but he still did it anyway. This knowledge of calligraphy, later on, inspired Jobs during his Apple days to include multiple typefaces in Macintosh.

"If I had never dropped in on that single course in college, the Mac would have never had multiple typefaces or proportionally spaced fonts. And since Windows just copied the Mac, it’s likely that no personal computer would have them.” ~ Steve Jobs

The takeaway is to be curious and keep learning.

I have shared my Summer of Bitcoin journey in detail in my other blog.

A couple of days later, I saw there was an opening at Google for SWE Internship. I asked my senior to refer me and I applied. After a week I got a mail saying that I have 10 days before my interviews. So, I just got back into interview prep. I mostly referred to these resources:

• Leetcode and Leetcode discuss

• CLRS

• MIT Lectures

• Gave a bunch of mock interviews at pramp.

I am not a big fan of practicing from different resources and platforms. I like to keep things simple 😇

Both of my Google interviews went really well, and after a long wait of 2 months, I heard back that I got the internship. I was on cloud nine because at that point I was starting to get a bit depressed about having no internships even after working really hard.

My first corporate internship

Luckily, I got a really interesting and impactful project. It had to be built from scratch. So, a lot of time was spent writing design docs, making sequence diagrams, brainstorming the component workflows, and designing models. I worked for Google Ads.

The main codebase was in Java and at Google, we have a lot of private frameworks and technologies for internal usage. But some of my key learnings were:

• Communication is the key. It doesn't matter if you have a solution in your head if you can't explain it to the other person.

• Reach out for doubts. No one knows everything, so don't shy away from asking doubts. But while asking doubts make sure you have researched properly, provide enough information regarding the doubts, your thought process, your approaches, etc.

• Always keep your manager/host updated about your work.

• Meet new people, ask them about their project, their work what they like the most, etc. Getting insights from people who are more experienced than you helps a lot in gaining a new perspective which is really important as a fresher. The more you network the more you grow. At Google, we have this Lunch Ninja where we are paired with a random Googler to have a 1:1 discussion. I met so many amazing people that way, people from different roles like DevRel, SRE, UI/UX Designer, etc.

• Writing high-quality code. In most of our personal projects, we don't care much about the code quality, but in industry, it's the code that is readable and is working that get's merged. So writing code that follows the readability standards and the style guides was a big takeaway.

I also wanted to add, that after the successful completion of my 10-week internship, I even got a pre-placement offer from Google and will be joining them next year 😊.

4th Year

Having the PPO at hand gave me some confidence to try out even more stuff. Even during my internship my entire team at Google praised me for my blogs. So, my first thought was to take up blogging.

Codedamn

I got the opportunity to write technical content for Codedamn. I worked with them for like 2 months and got a lot of articles published. You can find them here: https://codedamn.com/news/author/arnabsen.

The issue I had with blogging for someone else was that I had very limited freedom to decide on the topics to research and blog. It went against the very essence of why I started blogging in the first place. So, even after leaving Codedamn, I continued to blog about random stuff on my personal website.

Cypherock

But I still felt the need to do something more challenging, so I started applying for various startups and interviewed a bunch of them. Finally, I decided to start another internship with Cypherock.

In the next 6 months, I learned a lot about firmware and embedded systems development, various cryptographic, hashing, and encryption algorithms and how fast-paced startups work.

XROS Fellowship

Again the curiosity in me sparked and I found another perfect opportunity. Meta, Digital India and FICCI came up with a fellowship opportunity to train young devs in the field of Augmented Reality/Virtual Reality. It was called XROS Fellowship and I applied and was among the 100 devs who were selected for this opportunity. I worked for an org called GMetriXR.

While doing this I was also working as a Teaching Assistant at Pupilfirst, reviewing the submissions of the students in the cohort, conducting sessions and solving doubts.

Outro

As you might have seen I have tried out different stuff throughout my college days and still doing. I have also participated in hackathons, kept sessions at our GDSC, and facilitated campaigns. I love sharing my knowledge through blogs.

My approach remains the same for everything, "no matter what you are doing, give your best and try to go the extra mile".

Feel free to connect with me on any of my social handles.

Ok, but what is XR? XR refers to Extended Reality.

Umm, and what's that?....

Understanding "Extended Reality"

Extended Reality (XR) is a term that encompasses the entire spectrum of computer-generated environments, including both virtual reality (VR) and augmented reality (AR), as well as mixed reality (MR) and other related technologies. XR combines real and virtual elements to create immersive experiences that go beyond the physical world.

Virtual Reality (VR) refers to a simulated environment that is completely computer-generated, allowing users to be fully immersed in a digital world. Users typically wear a VR headset that tracks their head movements and provides a visual and auditory experience that feels like being in a different reality.

Augmented Reality (AR) overlays virtual elements onto the real world. AR technology superimposes computer-generated images, videos, or information onto the user's view of the physical environment. This can be experienced through devices like smartphones, tablets, or smart glasses.

You might have heard about AR from the game that went viral (for a lot of reasons 🥲) in 2016. Remember the game? Yes, it is Pokemon Go.

Mixed Reality (MR) is just a blend of AR and VR. In other words in mixed reality, virtual objects, and the real world coexist and interact in real-time, enabling users to engage with both simultaneously.

Now that you know what XR means, let's get back to XROS Fellowship.

XROS Fellowship

Here's a snippet from their website, which pretty much covers everything about what XROS Fellowship is.

XROS Fellowship Program is a uniquely curated initiative aimed at supporting Indian developers working on XR technologies by providing fellowships which will include a stipend and mentoring by industry experts. The Program will support developers to make contributions to open-source projects related to XR technology.

The Program aims to give developers a platform to work with the best resources to create digital public goods and further support their careers by facilitating fellowships in organisations working in the domain of AR, VR, MR, 3D Modelling, etc.

XROS Fellowship will provide a learning cohort for 100 developers and selected developers will work on projects with partner organisations and industry mentors.

The initiative was taken by FICCI (Federation of Indian Chambers of Commerce & Industry) which is the largest and oldest apex business organization in India and was established in 1927. And the fellowship is further supported by Meta and implemented by Reskilll.

Perks of this fellowship

• 💼 3 months of practical experience on live industry projects.

• 🤝 Networking with industry experts.

• 💰 Fellowship stipends of ₹ 4 Lakhs.

Eligibility for this fellowship

1. The applicant should be 18+ years of age.

2. The applicant should be in the pre-final or final year of graduation.

3. The applicant should be a resident of India.

My Journey

Initial Application

The initial step of the fellowship involved completing a form that requested basic details. This stage aimed to assess whether applicants met the eligibility criteria for the fellowship.

Following this round, I received an email notifying me that my profile had been approved. Additionally, I was provided with a link to the Slack channel, where project mentors and other approved candidates can have a discussion.

Proposal Round

There were approximately 31 organizations listed. Each organization had its own projects, and candidates will need to submit a proposal for the project they wish to work on during their fellowship. A single candidate can only submit a maximum of 4 proposals.

I intended to submit 4 proposals as it would enhance my chances of being selected. Therefore, I would thoroughly review each organization, conduct some research on them, and then examine their respective projects.

My approach involved submitting proposals in the following categories:

1. One proposal would focus on a project that genuinely piques my interest, regardless of my familiarity with the associated technology stack.

2. Another proposal would target a project aligned with a technology stack in which I possess considerable proficiency.

3. An additional proposal would be directed towards a project that showed minimal candidate activity, indicating reduced competition.

4. Lastly, I would submit a proposal for a project considered highly challenging. This choice would ensure limited competition and thus increase my chances of success.

Subsequently, I prepared a Notion document accordingly and commenced working on my proposals.

Here is a glimpse of the doc:

The Reskilll team also organized AMA sessions with the project mentors, which I would attend. During these sessions, the mentors addressed all the questions and these sessions provided me with a more comprehensive understanding of the project and the organization.

So, I finally sat and wrote 4 proposals. We were also given a very basic proposal template. These were the different sections of my proposal template:

1. Basic Personal Details.

2. What excited me to work at <org-name>?

3. Understanding <project-name>. (I tried to deep dive as much as possible and covered possible approaches. This particular section will be the determining factor)

4. Proposed Timeline.

5. Past Work Experience.

6. Previous Open Source Contribution.

7. Accomplishments.

8. Projects.

The Interviews

After the shortlisting of proposals, we needed to choose a time slot for the interviews. I received interview invitations from 2 out of the 4 organizations I applied to. In preparation for the interviews, I thoroughly reviewed the research I conducted to understand the project. Additionally, I dedicated time to practicing technical questions related to the technology stack.

In my view, these preparations are sufficient for the interviews. Both of my interviews went exceptionally well, and I was really excited about the final results.

The Results

After a long wait (it wasn't very long, but I was so much excited it felt long) I finally got a mail saying that I got into the XROS Fellowship Program for the organization GMetriXR.

After the AMA, I found myself genuinely intrigued by the GMetri project and its vision. I devoted significantly more time to reviewing the GMetri proposal than to any other. It is often said that diligent efforts yield fruitful results, and in my situation, it came true.

I am really proud that I was one of the 100 developers selected out of 10,000 applications.

The work at GMetri

GMetri in simple words is a toolkit designed for no-code construction of the metaverse. In recent days, we've observed the remarkable influence that Artificial Intelligence (AI) - particularly large Language Models like GPT - has had on everyday life. Its capacity to compose poems, craft music, and generate images is just phenomenal. My project at GMetri was aimed at leveraging the prowess of AI to simplify the process of metaverse creation for users.

For an effective generation of Metaverse Scenes, it is crucial that the AI model possesses an adequate understanding of the context. We called this knowledge base the "AI Brain". My primary responsibility entailed designing the user frontend experience to facilitate the creation of these AI Brains. This involved allowing users to upload information in the form of URLs, documents, and even raw text, which would subsequently be uploaded to the AI Brain. And once the AI Brain is trained the users can query the brain and use those results in the metaverse scene creation.

Final Thoughts

This was my last stint during college, cause I graduated after the fellowship period. I learned a lot about AR, VR, and XR through this experience and seeing how much the big techs like Apple, Meta are invested in this field we can expect some amazing innovations happening soon.

Do let me know if this article helped you and follow me for more technical content. Have a nice day.

XROS Summit

Recently all the selected XROS devs were also invited to attend the XROS Summit at New Delhi. It was truly an amazing experience meeting the amazing folks I worked with and networking with others.

In today's digital landscape, where cyber threats lurk around every virtual corner, safeguarding sensitive information is of paramount importance. One crucial tool that plays a pivotal role in enhancing digital security is captchas. Captchas, short for "Completely Automated Public Turing tests to tell Computers and Humans Apart," act as a gatekeeper, effectively distinguishing human users from malicious bots. They were invented in 2000 by Luis von Ahn, Manuel Blum, Nicholas Hopper, and John Langford at Carnegie Mellon University. The original CAPTCHAs were text-based and required users to identify distorted words or letters.

These ingenious puzzles serve a lot of purposes. They help prevent a variety of attacks like:

• Spam: Spam is unsolicited electronic messages, typically sent in bulk. CAPTCHAs can be used to prevent spam bots from creating accounts or sending messages by requiring users to solve a CAPTCHA before they can create an account or send a message.

• DDoS attacks: A DDoS attack is a distributed denial-of-service attack. This type of attack is designed to overwhelm a website or online service with so much traffic that it becomes unavailable to legitimate users. CAPTCHAs can be used to prevent DDoS attacks by making it more difficult for attackers to generate large numbers of requests.

• Credential stuffing: Credential stuffing is a type of attack where attackers try to use stolen login credentials to access multiple accounts. CAPTCHAs can be used to prevent credential stuffing attacks by making it more difficult for attackers to try login credentials on multiple websites.

• Phishing: Phishing is a type of attack where attackers send fraudulent emails that appear to be from a legitimate source. The goal of phishing is to trick the recipient into clicking on a malicious link or providing personal information. CAPTCHAs can be used to prevent phishing attacks by making it more difficult for attackers to create realistic-looking phishing emails.

• Click fraud: Click fraud is a type of attack where attackers click on ads without human interaction. This type of attack is used to generate revenue for the attacker. CAPTCHAs can be used to prevent click fraud by making it more difficult for attackers to click on ads without human interaction.

By incorporating captchas, website owners can fortify their defense against various cyber threats, such as spam, fraud, and unauthorized access. The utilization of visual or audio-based challenges, often involving distorted characters or logical tasks, impedes automated programs' ability to infiltrate online platforms.

How did Captchas even come into the picture?

Well, this dates back to a paper titled "Computing Machinery and Intelligence" that was published in 1950 by the famous and legendary computer scientist, mathematician, logician, cryptanalyst, philosopher, and theoretical biologist 🤯.

Are you able to guess who I am talking about?

Well here is a hint 💡: He is famously regarded as the father of modern computer science.

Yes, none other than Alan Turing.

He asked a question "Can a computer talk like a human?". This question led to an idea for measuring artificial intelligence that would famously come to be known as the Turing test. Turing proposed the following game. A human judge has a text conversation with unseen players and has to evaluate their responses. To pass the test, a computer must be able to replace one of the players without substantially changing the results. In other words, a computer would be considered intelligent if its conversation couldn't be easily distinguished from a human's.

Believe it or not, this same principle is used today to differentiate between a real human and a bot. You can think of captchas as simplified and automated Turing Test and every time you are able to solve a captcha you are basically passing a mini Turing Test.

P.S: If you are also a movie buff I would highly encourage you to watch the movie The Imitation Game where Benedict Cumberbatch played the role of Alan Turing.

Now, that you know a brief about how captchas came into the picture, let's learn the different types of captchas.

What are the different types of captchas?

You might have across different types of captchas while surfing the internet, here are the broad categories of captchas:

1. Text CAPTCHA: The oldest and most common type of CAPTCHA, this involves inputting text from a distorted or obscured image. The logic here is that humans can interpret obscured text while bots cannot.

2. Image CAPTCHA: This type of CAPTCHA involves identifying specific types of objects or patterns in an image. For instance, a user might be asked to select all squares of a grid containing traffic lights or buses.

3. Audio CAPTCHA: For users with visual impairments, audio CAPTCHAs are used. In this case, a short audio clip is played and the user is asked to type the words or numbers that they hear.

There are a bunch of service providers available who allow you to use their captcha services on your websites. Some of the most popular ones include:

• Google reCAPTCHA

• hCaptcha

• GeeTest

• KeyCaptcha

• Lemin Captcha

Now, from a personal point of view I kind of hate solving these captchas, and finding the traffic lights.

So, I started looking if there were any ways to bypass these irritating captchas and I landed upon a goldmine 2captcha. Turns out, these captchas weren't as secure as I used to believe. Today we will look at a demo of how I ended up writing a script to bypass these captchas.

But, before that, let's learn about 2Captcha.

What is 2Captcha?

2Captcha operates as a real-time CAPTCHA decoding service, proficient in distinguishing and interpreting captchas with accuracy. It leverages human intellect for image recognition tasks, ensuring high levels of precision. Offering compatibility with a wide range of programming languages through its API, 2Captcha can identify and solve a diverse variety of CAPTCHA types. In fact, 2Captcha can actually solve all the captcha types I mentioned above.

So, without further ado, let's use the Golang module of 2Captcha and build ourselves a CLI tool. You will be amazed to see how easy it is to use this service.

Building the Golang CLI tool

So, the first step would be to create an account in 2Captcha and get our API key.

1. Creating our 2Captcha account

Visit their website https://2captcha.com/ and then sign in with your method of choice. After that, you will land on a page like this:

Select the left option of "I'm a customer" and then you will be redirected to the dashboard where you can see your API key.

Now, to further move ahead with our project make sure that you have Golang installed in your machine. To check that simply run go version in the terminal and it should show the version of Golang that is installed. If it gives an error, you will have to install Golang.

Once you are done with all these we can finally initialize our Golang Project.

2. Initialize our Golang Project

To get started, let's initialize a new Golang project. Open your terminal or command prompt and follow the steps below:

1. Create a new directory for your project:

    mkdir captcha-solver
    cd captcha-solver
   

2. Initialize the Go module:

    go mod init github.com/your-username/captcha-solver
   

   Replace your-username with your actual GitHub username or any other relevant identifier.

3. Create a new Go source file named main.go:

    touch main.go
   

You have now set up the basic structure for your Golang project.

3. Installing Required Modules

To interact with the 2Captcha service, we need to install the 2captcha-go library. Run the following command to install the required module:

go get github.com/2captcha/2captcha-go

The module will be downloaded and added to your project's go.mod file.

Finally, we will write the code for our captcha solver.

4. Building the Captcha Solver

Now let's proceed with building the captcha solver. Open the main.go file in a text editor and add the following code:

package main

import (
    "flag"
    "fmt"
    "io/ioutil"
    "log"
    "os"

    "github.com/2captcha/2captcha-go"
)

const apiKey = "<API-KEY>" // Replace with your actual API key

func main() {
    // Parse input arguments
    captchaImagePath := flag.String("image", "", "Path to the captcha image file")
    flag.Parse()

    // Verify that the captcha image path is provided
    if *captchaImagePath == "" {
        fmt.Println("Please provide the captcha image path.")
        flag.PrintDefaults()
        os.Exit(1)
    }

    // Load the captcha image
    _, err := ioutil.ReadFile(*captchaImagePath)
    if err != nil {
        fmt.Printf("Failed to read captcha image: %v\n", err)
        os.Exit(1)
    }

    // Initialize the 2Captcha client
    client := api2captcha.NewClient(apiKey)

    // Use the 2Captcha client to solve the captcha
    cap := api2captcha.Normal{
        File: *captchaImagePath,
    }

    captchaText, err := client.Solve(cap.ToRequest())
    if err != nil {
        if err == api2captcha.ErrTimeout {
            log.Fatal("Timeout")
        } else if err == api2captcha.ErrApi {
            log.Fatal("API error")
        } else if err == api2captcha.ErrNetwork {
            log.Fatal("Network error")
        } else {
            log.Fatal(err)
        }
    }

    // Output the solved captcha text
    fmt.Printf("Solved captcha: %s\n", captchaText)
}

Now, let's break down the code and understand each section's functionality.

Parsing Input Arguments

captchaImagePath := flag.String("image", "", "Path to the captcha image file")
flag.Parse()

This code uses the flag package to parse input arguments provided when running the program. We define a flag named image that represents the path to the captcha image file. The flag.Parse() function is then called to parse the input arguments.

Loading the Captcha Image

_, err := ioutil.ReadFile(*captchaImagePath)
if err != nil {
    fmt.Printf("Failed to read captcha image: %v\n", err)
    os.Exit(1)
}

In this section, we use the ioutil.ReadFile() function to load the contents of the captcha image file specified by the user. If there is an error while reading the file, an error message is printed, and the program exits.

Initializing the 2Captcha Client

client := api2captcha.NewClient(apiKey)

Here, we initialize the 2Captcha client by creating a new instance of api2captcha.Client with the provided API key. Remember to replace the API key with the key that you saw in your dashboard.

Solving the Captcha

cap := api2captcha.Normal{
    File: *captchaImagePath,
}

captchaText, err := client.Solve(cap.ToRequest())

To solve the captcha, we create a Normal struct instance from the api2captcha package and pass the captcha image file path to it. Then, we call the Solve() method of the 2Captcha client, passing the ToRequest() method's result from the cap struct as an argument. This method sends the captcha image to the 2Captcha service for solving and returns the solved captcha text.

Outputting the Solved Captcha Text

fmt.Printf("Solved captcha: %s\n", captchaText)

Finally, we output the solved captcha text to the console using fmt.Printf().

Conclusion

Congratulations! You have successfully built a captcha solver using the 2Captcha service in Golang. You can now run the program by executing the following command:

go run main.go --image /path/to/captcha.png

Replace /path/to/captcha.png with the actual path to your captcha image file. The program will send the image to the 2Captcha service, solve the captcha, and display the solved text.

Feel free to explore the 2Captcha Go client library documentation (https://pkg.go.dev/github.com/2captcha/2captcha-go) for more advanced usage and options.

That's it! You can now integrate this captcha solver into your own projects or applications to automate captcha solving using the 2Captcha service.

You might be familiar with these containers.

These are used for storing stuff and shipping from one place to another. In DevOps, containers have similar applications as well.

A Container is a way to package an application with all the necessary dependencies and configuration.

And that package can be portable easily, which makes the process of development and the job of a developer easier.

What happens in these containers? Nothing much, just a couple of processes run in isolation on a shared kernel.

How is it isolated? The isolation of containers is provided by a Linux feature called namespaces. Namespaces partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources. They give the group of running processes an isolated view of the kernel. For example,

You can use the command lsns to get lists of all the currently accessible namespaces.

$ lsns
        NS TYPE   NPROCS   PID USER    COMMAND
4026531835 cgroup     85  1571 seth /usr/lib/systemd/systemd --user
4026531836 pid        85  1571 seth /usr/lib/systemd/systemd --user
4026531837 user       80  1571 seth /usr/lib/systemd/systemd --user
4026532601 user        1  6266 seth /usr/lib64/firefox/firefox [...]
4026532928 net         1  7164 seth /usr/lib64/firefox/firefox [...]

Another feature of the kernel is control groups (a.k.a cgroups) which monitors, limits, accounts for, and isolates the resource usage of a collection of processes (also known as containers)

Virtual Machine VS Containers

An operating system has two main layers:

1. OS Kernel: It communicates with hardware like memory, CPU .etc.

2. Applications Layer: They run on the Kernel.

Linux is a kernel. There are many Linux distributions each looking different from one another because the applications are different. But under the hood, they use the same kernel i.e. Linux.

Coming back to the difference between VM and Containers:-

VM runs on something called Hypervisor. Each virtual machine includes a full-blown OS and its process, which are very heavy and slow to start.

Containers on the other hand don't include full-blown OS, only include a set of OS-specific files. They are just processes that share the same kernel with other containers and the isolation by the containers is provided by the Linux namespaces. So since they run on top of the kernel they are very fast and also light-weight. So, we are getting the benefits of isolation by VM without the heaviness that comes with VM

But remember:

Containers don't replace VMs. Both have their own purpose.

What is Docker?

An early implementation of container technology was added to FreeBSD in 2001. Whereas Docker debuted to the public in Santa Clara at PyCon in 2013 and was made open source in March 2013. The tooling for using Linux Containers was really lacking and that's where docker comes into play. Basically, docker is tooling to manage containers.

Docker allows developers to package their applications into containers and directly use them in their CI/CD pipeline.

It helps achieve:

Build once, run everywhere

Why should we at all bother?

There are a number of advantages of using Docker and Containerisation: (No wonder it is very popular)

1. You can't say "But... It works on my machine" anymore. Good Luck with that. Because we are packaging the application with all the dependencies and the configurations required, even if it is running on a different machine it will still behave the same way.

1. They are very lightweight and fast. Already discussed this, just adding one thing. The fact that dockers are lightweight is the main reason behind them being so portable. Obvious

2. Docker has its own ecosystem provided by the community and has many tools that come with it, which helps solve a lot of issues.

Let's get our hands dirty now.

To install docker you can look into the official docs. Else if you want to just try your hands first, then you can use this: https://labs.play-with-docker.com/

Practical 1

Let's go through some popular docker commands. The docker commands are the same for all environments so it won't be an issue.

Run a container

Run this command

docker container run -t ubuntu ls

docker container run will run the image that your provided, in this case, it is Ubuntu. Now if you are running this command for the first time high chances are that the image is not downloaded, so it will show something like this:

Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

If you notice something here we just mentioned ubuntu not ubuntu:latest. If we didn't specify any version it will take the latest version. But if you wanted to use ubuntu 18.04 you can specify it like this:

docker container run -t ubuntu:18.04 ls

It will again show:

Unable to find image 'ubuntu:18.04' locally
latest: Pulling from library/ubuntu

But this time notice the version. Also in both cases you see an output like this

bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

Basically, it is the output of the ls command.

Now let's run this command

docker container run -t ubuntu top

And open a new shell and continue. If you are using Play with Docker. Then create a new instance, and then ssh into the previous instance.

Now type this command

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
6d2a990df65d        ubuntu              "top"               3 seconds ago       Up 2 seconds                            blissful_austin

This command shows you the containers and all the necessary information about it.

Usually, containers are used for running a process, maybe a server or some application, and soon as the main process exits the container will stop too. So let's create a very simple process.

So run this command

docker run -d ubuntu sleep 300

Yeah, this is our process, we will ask the container to sleep for 300 secs.

Every container has a unique container id. So in most of the commands which deal with a particular container, we have to provide the container id. Now let's hop into the sleeping container.

docker exec -it 6d2a990df65d bash

Note: 6d2a990df65d is the docker container id in my case. Just do docker ps look at the container id and paste it there.

It will spawn a bash terminal inside the container. You can run all the basic commands of Ubuntu.

To get the list of all the containers (even the ones that exited) run

docker ps -a

Stop a container

To stop the container we need to use docker stop <container-id-1> <container-id-2>.

docker stop 6d2a990df65d d9da0526d987

To remove containers you can use docker system prune

There is a nice collection of important docker commands by garystafford here which I find very helpful.

Debugging container

Usually, for debugging, we need to look at the logs. For that use the command:

docker logs <container-id>

Also, docker exec sometimes helps in the debugging process.

Remember that containers use kernel-level features to achieve isolation and that containers run on top of the kernel. Your container is just a group of processes running in isolation on the same host, and you can use the command docker exec to enter that isolation with the bash process. After you run the command docker exec, the group of processes running in isolation (in other words, the container) includes sleep and bash.

Where are all these images stored?

Docker maintains a public repository of all the images called Dockerhub.

Also, you can run more than one container simultaneously.

While running containers you can do a lot more stuff, like

• --name tag will allow you to name the container.

• --detach or -d will allow you to run the docker in detached mode i.e. in the background.

• --publish or -p to publish the ports to the host (very important)

You can the list of all the options here

Containers are self-contained and isolated, which means you can avoid potential conflicts between containers with different systems or runtime dependencies. You can run multiple NGINX containers that all have port 80 as their default listening ports. If you're exposing the host by using the --publish flag, the ports selected for the host must be unique. Isolation benefits are possible because of Linux namespaces. Running multiple containers on the same host gives us the ability to use the resources (CPU, memory, and so on) available on a single host. This can result in huge cost savings for an enterprise.

Although running images directly from the Docker Store can be useful at times, it is more useful to create custom images and refer to official images as the starting point for these images.

Docker Images and Docker Containers

Now, if you are confused let's look into the differences between Docker Images and Docker containers. A docker image is a tar file or an archive of the filesystem or the container. It contains the metadata of the filesystem.

Whereas Container is the running environment of the image. It's the process (isolated process to be specific). The filesystem of the container is virtual, i.e. it has its own abstraction.

Images are used to create containers (more than one). You can consider it as a blueprint. We can share our Docker Image and then we can create containers using the image. We can also push images to the Docker Hub.

How to create Docker Image?

To create a Docker image, we use a special file called Dockerfile (no extension), which consists of a list of commands to build our image. After we have created our image we can pass it to docker build which will build the image.

docker build -f Dockerfile

A Docker Image is a set of layers where each layer represents an instruction from the Dockerfile. The layers are stacked on top of each other. Each new layer is only a set of differences from the previous one.

The best part of the image layer is that they are cached. If you change say the 5th line then the docker engine will reuse the first 4 layers and then start building from the 5th line. This improves the time during build and also in the context of CI/CD once the base is pushed, the subsequent pushes will be very fast. To optimize the caching, we need to organize the Dockerfile in a way that the line that will change the most is located at the bottom of the Dockerfile.

Let's dive into this.

Practical 2

Creating flask application

We are going to create a simple Flask application.

• Download this Python script https://gist.github.com/arnabsen1729/1fa19228e4451963bbb64563da98f880 and save it as app.py.

• Install the package flask by pip3 install flask and run the app python3 app.py.

• When the app is running you can visit your 0.0.0.0:5000, you will see hello world!.

Hence we have set up a basic Flask server. To close this process press Ctrl+C.

Now we will dockerize this flask app.

Building Dockerfile

So create a file Dockerfile and open your text editor and paste this

FROM python:3.6.1-alpine
RUN pip install flask
CMD ["python","app.py"]
COPY app.py /app.py

Let's go through this line by line

FROM python:3.6.1-alpine

This is the starting point for your Dockerfile. Every Dockerfile typically starts with a FROM line that is the starting image to build your layers on top of. In this case, you are selecting the python:3.6.1-alpine base layer because it already has the version of Python and pip that you need to run your application. The Alpine version means that it uses the Alpine distribution, which is significantly smaller than an alternative flavour of Linux. A smaller image means it will download (deploy) much faster, and it is also more secure because it has a smaller attack surface.

It is highly recommended to only use official images found in the Docker Hub, or noncommunity images found in the Docker Store.

RUN pip install flask

The RUN the command executes commands needed to set up your image for your application, such as installing packages, editing files, or changing file permissions. In this case, you are installing Flask. The RUN commands are executed at build time and are added to the layers of your image. Usually for node applications, this will involve installing all the node_modules, if it was a larger Python application, you will have to install those as well.

CMD ["python", "app.py"]

CMD is the command that is executed when you start a container. Here, you are using CMD to run your Python application. There can be only one CMD per Dockerfile.

If you specify more than one CMD, then the last CMD will take effect.

COPY app.py /app.py

This line copies the app.py file in the local directory (where you will run the docker image build) into a new layer of the image. This instruction is the last line in the Dockerfile. Layers that change frequently, such as copying source code into the image, should be placed near the bottom of the file to take full advantage of the Docker layer cache. This allows you to avoid rebuilding layers that could otherwise be cached. For instance, if there was a change in the FROM instruction, it will invalidate the cache for all subsequent layers of this image. You'll see this a little later in this lab

But, how can the CMD command run if we are copying the app.py later? CMD is the command that is executed when you start a container, until and unless you run the container it is not executed. Also since the command to run the application will not change, we have placed it higher.

Here is the list of all commands

Now to build the image with the Dockerfile run

Running the container

$ docker build -t flask-app . # if the Dockefile is in that directory
#or
$ docker build -t flask-app -f /path/to/Dockerfile

It will pull the Python:3.6.1-alpine, and go through the process. In the end it will output something like this

Successfully built 38f35dd1d2a4
Successfully tagged flask-app:latest

Now run the command docker images and you will see flask-app or whatever name you mentioned in the list. So now you have successfully created the image.

For the final part let's run the container with this image.

docker run -p5001:5000 -d --name flask-container flask-app

What are we doing here?

• docker run to run the container

• -p5001:5000 to map port 5000 of our container with 5001 of the host machine

• --name flask-container giving the name of my container

• flask-app the image we want to build

Now do docker ps you will see

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
940f215ceea0        flask-app           "python app.py"     2 seconds ago       Up 2 seconds        0.0.0.0:5000->5001/tcp   flask-container

Now go to localhost:5001, if everything was correct you will see hello world!

To check the logs we can run docker logs

$ docker logs flask-container
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [03/Jan/2021 14:55:58] "GET / HTTP/1.1" 200 -
172.17.0.1 - - [03/Jan/2021 14:55:59] "GET / HTTP/1.1" 200 -

Docker images contain all the dependencies that they need to run an application within the image. This is useful because you no longer need to worry about environment drift (version differences) when you rely on dependencies that are installed on every environment you deploy to. You also don't need to follow more steps to provide these environments. Just one step: install docker, and that's it.

Now if you change the app.py only the last step in Dockerfile needs to be updated the rest is already cached.

And you have successfully dockerized your application.

Docker Compose

Sometimes the docker run command becomes very long and very tedious like sometimes we even need to specify lots of environment variables and stuff. Running those long commands every single time becomes difficult. Also if you are working with let's say 3 or 4 containers then you have to write such long-run commands every single time you want to start the container. So there is a simple way to express the commands in a structured way, save it in a file and simply run that file.

That file is the docker-compose.yml. The file is a YAML file. (Fun fact: full form of YAML is 'YAML Ain't Markup Language' recursive huh!!)

Writing a docker-compose file is not very difficult, you just need to know how to structure it. This article written by Gabriel Tanner explains it nicely.

Link to the article: gabrieltanner.org/blog/docker-compose

If you have reached this far and you have understood Docker, it's also important that you keep the best practices in mind. Here is a good guide for that: spacelift.io/blog/dockerfile#dockerfile-best-practices

Conclusion

In this article, we learned about Docker and how to dockerize a simple application. We also learned about Docker Compose and how to use it.

A really nice collection of Docker study material is available here: Docker Handbook 2021 Edition

Hope you liked my article, do follow me on Hashnode and on Twitter (handle: @ArnabSen1729) for updates.

While using various apps on your smartphones, you might have noticed that the button where we usually have "enter", sometimes gets replaced with terms like "search", "next", etc. This small thing might be insignificant but still gives a lot of clarity about what's going to happen next. Ever wondered how that is implemented?

It's through the enterkeyhint attribute in HTML.

The enterkeyhint attribute provides a means for developers to communicate the expected action to the browser when the user presses the Enter key within a form. By utilizing this attribute effectively, developers can enhance both the UX and accessibility of web forms. By guiding the browser to understand the intended action, users can navigate forms more easily, without relying solely on mouse or touch input.

So, let's dive in and explore how this seemingly minor attribute can contribute to a more inclusive and user-friendly web!

What is the enterkeyhint attribute?

The enterkeyhint attribute is an HTML attribute that can be added to form input elements such as <input> and <textarea>. It is used to suggest to the browser the type of action that should be taken when the user presses the Enter key while the input element has focus.

How does enterkeyhint work?

The enterkeyhint attribute accepts a few predefined values that represent different actions. Here are the possible values:

• enter: Indicates that the default action for the Enter key should be performed. This is the default value if the attribute is not specified.

• done: Suggests that pressing Enter should submit the form or perform the action that signifies completion.

• go: Suggests that pressing Enter should initiate a "go" operation, such as navigating to a URL or starting a search.

• next: Indicates that pressing Enter should move the input focus to the next input field or control in the form.

• previous: Suggests that pressing Enter should move the input focus to the previous input field or control in the form.

• search: Indicates that pressing Enter should initiate a search operation.

The browser may use this hint to display an appropriate keyboard layout or provide other UI cues to the user.

Example usage

Let's consider a simple example of a login form to demonstrate the usage of the enterkeyhint attribute. We have two input fields: one for the username and another for the password. Here's how we can use the enterkeyhint attribute effectively:

<form>
  <label for="username">Username:</label>
  <input type="text" id="username" enterkeyhint="next">

  <label for="password">Password:</label>
  <input type="password" id="password" enterkeyhint="done">

  <input type="submit" value="Login">
</form>

In this example, we set the enterkeyhint attribute for the username input field to "next" and for the password input field to "done". This provides a hint to the browser about the expected action when the user presses the Enter key.

By setting the enterkeyhint to "next" for the username field, we suggest that pressing Enter should move the input focus to the password field. This helps users navigate through the form easily without having to use the mouse or touch input.

For the password field, we set the enterkeyhint to "done". This indicates that pressing Enter should submit the form or perform the action that signifies completion, which, in this case, is logging in.

Browser support

The enterkeyhint attribute is supported by all the browsers.

Source: developer.mozilla.org/enterkeyhint#browser_compatibility

Here is how all of them look:

Conclusion

Hope you learned something new through this blog. Do check out my other blogs and follow me on Twitter at ArnabSen1729 for more such interesting updates. You can also subscribe to my Hashnode newsletter to get updates every time I publish a new article.

Have a nice day 😄 👋.

I will mainly cover the new software developments in the field of AI and not the hardware and android updates.

TL;DR

Google I/O 2023 showcased AI advancements such as LaMDA, PaLM2, Imagen, and new AI features in Google Search. They also introduced AI integrations in products like Gmail, Google Maps, and Google Photos, along with the launch of Bard, an AI chatbot. Other announcements include AI tools for developers like Vertex AI and Project Tailwind, an AI-first notebook.

Let's learn what Google I/O is all about.

What is "Google I/O"?

Google I/O is an annual developer conference held by Google in Mountain View, California. The name "I/O" is taken from the number googol, with the "I" representing the 1 in googol and the "O" represents the first 0 in the number.

A googol is the large number 10¹⁰⁰. In decimal notation, it is written as the digit 1 followed by one hundred zeroes: 10,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000,​000.

The conference also features a number of educational sessions and workshops, where developers can learn about the latest technologies and best practices.

In addition to being a valuable resource for developers, Google I/O is also a great opportunity for Google to connect with its users and get feedback on its products. The conference is open to the public, and there are a number of activities and events that are available to all attendees.

Google I/O 2023 was a two-day event held on May 11-12, 2023, at the Shoreline Amphitheatre in Mountain View, California. The event was attended by over 50,000 developers from around the world.

A keynote was addressed by Sundar Pichai, CEO of Google, where he announced new products and features for Google's various platforms.

💡 What were my expectations from Google I/O 2023?

We all know that Google has been a leader in the field of artificial intelligence (AI) for decades. In 2017, they released the revolutionary transformer model, which is now widely used in natural language processing (NLP) tasks such as machine translation, text summarization, and question-answering.

You can checkout the amazing paper by folks at Google titled "Attention is All You Need" from here.

It was therefore no surprise that Google's annual developer conference, Google I/O 2023, was full of announcements about new AI advancements.

Some of the most notable announcements included:

• The release of PaLM2, a 540-billion parameter LLM that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.

• The release of Imagen, a new AI model that can generate realistic images from text descriptions.

• The announcement of new AI features for Google Search, including the ability to answer your questions in more natural language and to provide more relevant results.

• The launch of new AI tools for developers, including the Google AI Platform and the Google AI Test Kitchen.

These are just a few of the many AI advancements that were announced at Google I/O 2023. It is clear that Google is committed to leading the way in the field of AI, and these advancements are sure to have a significant impact on the way we live and work.

A significant part of Google I/O focused on integrating Generative AI into their products. So, what exactly is Generative AI?

🤖 Understanding Generative AI

Generative AI as the name suggests is a type of artificial intelligence (AI) model called Large Language Model (LLM) that can create new content, such as text, images, audio, and video. Generative AI models are trained on large datasets of existing content, and they use this data to learn the patterns and rules that govern how that content is created. Once a generative AI model has been trained, it can use this knowledge to create new content that is similar to the content it was trained on.

Here is a nice video on the recent updates in Generative AI by Google.

📲 AI integrations in Google products

📪 Gmail's new "Help me write"

This AI feature will generate the entire email for you. All you have to do is specify the details of the email in the prompt. It can further pull context from the previous email.

This is something I will personally use quite often. Currently, there are some excellent tools in the form of browser extensions built on OpenAI's GPT model that serve the same purpose. However, it appears that this new feature from Google could be detrimental to those tools.

🌍 Immersive View for routes in Maps

Google Maps' Immersive View is being expanded to allow users to see their entire route in advance, whether they're walking, cycling, or driving. This will allow users to get a better feel for their route and make informed decisions about where to go. Immersive View will begin to roll out over the summer and will be available in 15 cities by the end of the year.

✏️ Magic Eraser for Google Photos

It is an AI-powered tool that allows users to remove unwanted distractions from photos. Magic Editor is a new AI-powered tool that will allow users to do much more with their photos, such as repositioning objects, changing the sky, and more. Magic Editor is expected to roll out later this year.

✨ The second-gen PaLM model: PaLM 2

PaLM is a large language model (LLM) from Google AI, but it is not the only LLM that Google has developed. Other LLMs from Google include LaMDA, Meena, and T5. PaLM is the largest and most powerful LLM that Google has developed to date, but it is still under development.

And in this I/O, Google announced a more advanced version of that called PaLM2.
PaLM2 is a second-generation large language model (LLM) from Google AI. It is built on the same architecture as PaLM, but it is trained on a larger dataset of text and code.

PaLM2 is a family of models, each with its own strengths and weaknesses. Here are some of the different models in PaLM2:

• Gecko: Gecko is the smallest model in PaLM2. It is fast and efficient, but it is not as powerful as the other models.

• Otter: Otter is a medium-sized model in PaLM2. It is more powerful than Gecko, but it is not as powerful as the larger models.

• Bison: Bison is a large model in PaLM2. It is more powerful than Otter, but it is not as powerful as the largest model.

• Unicorn: Unicorn is the largest model in PaLM2. It is the most powerful model in PaLM2, but it is also the slowest and least efficient.

PaLM2 is particularly good at:

1. Math: It can solve math problems, such as algebra, calculus, and geometry. It can also generate mathematical proofs.

2. Coding: PaLM2 can generate code that is both correct and efficient. It can write code in a variety of programming languages, and translate between them.

3. Learning & Reasoning: PaLM2 is able to learn new tasks and concepts quickly. It can be trained on a new task with just a few examples.

4. Translating languages: PaLM2 can translate languages with high accuracy. It can translate between any pair of languages that it has been trained on.

PaLM2 models perform even better when fine-tuned for particular applications. Some of my favorite fine-tuned models are Sec-PaLM and Med-PaLM2. Sec-PaLM can explain the behavior of potentially malicious scripts, and better detect which scripts are actually threats to people and organizations. Med-PaML2 can answer questions and summarize insights from a variety of dense medical texts.

Google is also working on another next-generation foundation model called Gemini. Gemini is being designed to be multimodal, which means that it will be able to process and understand information from a variety of sources, such as text, code, images, and audio. This will make Gemini much more versatile than previous AI models, and it will allow it to be used in a wider range of applications.

💬 Bard

The part of Google I/O that I was most excited about was the launch of Bard. It was initially rolled out to only US and UK, but now it's publicly available and I am really loving it.

A very popular alternative that we all know is ChatGPT. Here are some of the features that Bard is providing which is not yet available in ChatGPT:

• Information Cutoff: The information cutoff for ChatGPT is September 2021. Whereas Bard can perform an internet search and give a response accordingly. This allows you to get the latest information and updates.

  

• Export options: Bard allows you to export the response as Docs or a Draft in Gmail. If it detects code then it also gives an option to "Export to Colab" but ChatGPT doesn't have any such functionality yet.

  

• Citations: Bard can also cite sources, something which ChatGPT still can't do.

  

Further, Google is promising that in the next few weeks, Bard will become more visual. Bard will also be integrated with Google Lens, where it can analyze the photos and then respond according to your prompt.

🎨 Bard x Adobe Firefly

This collaboration will allow Bard users to generate images directly from their text descriptions, using Firefly's state-of-the-art technology.

To use this feature, Bard users will simply need to type a description of the image they want to generate into the chatbot. Firefly will then use its AI to create an image that matches the description. Users will be able to edit the image as needed, and they can even share it directly on social media.

💼 Workspaces

Some of the popular tools available in Google Workspaces like Docs, and Sheets will have options for users to prompt specific requirements. From generating job descriptions, and creating tables with data inserted, to creating images with specific styles everything can be done with just a simple prompt directly from these tools. You can do these with a side panel called "sidekick" which will keep track of the context and then provide relevant information. This will make prompting very easy.

All of this will be generally available to business and consumer Workspace users later this year via a new service called Duet AI for Workspace.

🔍 Generative AI in Search

A typical Google Search can be quite challenging, to be honest. To obtain very specific information, you must break down the text you are searching for and then skim through the results to find the precise answer.

But with this new integration of Generative AI, you don't have to do so anymore. It will automatically take care of extracting the information, getting relevant results and then compiling the results to give you the perfect response.

✳️ Vertex AI

Vertex AI is a managed machine learning (ML) platform that helps you build, deploy, and scale ML models. It offers a unified experience for managing the entire ML lifecycle, from data preparation to model training and deployment. Vertex AI also provides a variety of tools and services to help you accelerate your ML projects.

Other big techs are already availing this functionality in their apps, like:

• replit

• Uber

• Canva

• character.ai

Vertex has the support of 3 new models, in addition to PaLM2.

1. Imagen: which powers image generation, editing, and customization from text inputs.

2. Codey: for code completion and generation, which can be trained on code base to help build applications faster.

3. Chirp: universal speech model which brings speech-to-text accuracy for over 300 languages.

All of these features are already in Preview and can be used.

📔 Project Tailwind

Project Tailwind is an experimental AI-first notebook that learns from your documents and helps you learn faster. You can upload the docs using Google Drive and that creates a custom fine-tuned model for you to interact with.

It can be used:

• to create study guides

• find sources

• generate quizzes

• do a quick Google search to get more information.

Conclusion

Google I/O 2023 unveiled groundbreaking AI advancements and integrations, showcasing the power of AI to revolutionize the way we live, work, and interact with technology. These innovations promise to make our lives more efficient, connected, and inspired as we move toward a brighter future.

As we embrace the future of AI, let's harness its power to innovate, improve our lives, and create a world where technology is a catalyst for positive change, empowering us to reach new heights and achieve our greatest potential.

References

• Google I/O 2023: Keynote

• Introducing PaLM 2

If you found this blog useful, do leave a comment and like this blog, it really motivates me to put forward more such content. You can follow me on Twitter for further updates. My Twitter handle: @ArnabSen1729.

You can find the rest of my social links at arnabsen.dev/links. Thank you and have a wonderful day.

Google Cloud is a comprehensive suite of cloud computing services that can assist you in easily building, running, and scaling your applications. Google Cloud provides a wide range of solutions to help you achieve your goals, from data storage and analysis to machine learning and artificial intelligence.

In this blog, we’ll go over the fundamentals of Google Cloud, such as what it is, how it works, and why it’s such a valuable tool for both businesses and developers. We’ll go over the various Google Cloud services and provide real-world examples of how businesses are using Google Cloud to achieve their goals.

A little introduction about me

Hi, I am Arnab Sen. A Software Developer from India. I was an SWE Intern at Google and will be joining them as a Full Time Engineer. I was also a Google Cloud Facilitator where I taught 150+ students about Google Cloud and its tools. Feel free to reach out to me on my social handles at arnabsen.dev/links

Getting a brief idea

Before starting out, I urge you to watch this video by Google Cloud Tech once. It will really inspire you to learn about this new domain of technology.

Wow, after watching that video, it’s clear that Google Cloud is the bomb! It’s like a Swiss Army knife for building innovative solutions that can do some seriously cool stuff. Whether you’re trying to build a chatbot that can talk to customers or a machine learning model that can predict the future, Google Cloud has got your back.

One of the things that really stood out to me was how easy it is to use Google Cloud. The video showed how you can set up a machine-learning model with just a few clicks, and I was blown away by how simple it was. With Google Cloud, you don’t have to worry about all the nitty-gritty details of setting up servers and managing infrastructure. You can focus on building your application and let Google Cloud handle the rest.

What is Cloud Technology after all?

Cloud computing is the delivery of different services through the Internet, including data storage, servers, databases, networking, and software.

Ehh !! That sounds complicated.

So, let’s imagine you have a lot of toys but your room is small and you can’t keep them all. So you approach a friend and ask if you can keep some of your toys in their room. You can still play with your toys whenever you want, but they won’t take up as much space in your room.

Where do Cloud Platforms come into the picture?

Cloud computing is similar in this regard. You have a lot of computer stuff instead of toys, such as pictures, videos, and documents, but your computer doesn’t have enough storage space to keep everything. So you ask the cloud (which is similar to your friend’s room) if you can store your belongings there.

Let’s rewind a little bit

How do you think Cloud became mainstream? There were 3 stages to this. Let’s go through them one by one.

Stage 1: On-Premise

Back in the day, let’s say you created an e-commerce platform. It had 5 users and you had just one server on your desk and you are managing everything. Everything is doing great. All of a sudden someone tweets about your platform and now you have 1000s of users. But your poor server can’t handle those many users so it starts crashing. So you had to do one thing called “Scaling”. Basically, you had to buy a bigger, more powerful computer to handle all the users. Can you imagine how tedious it was back in the day?

Stage 2: Time Sharing

That led to the 2nd stage. Now, big companies like IBM would buy out a million computers cause someday they would definitely need them. But sometimes there might be some computers which are sitting idle and not being used. So what did they do? They started to rent those out, to other companies and small startups. That way it will be cheaper and more affordable for them and companies like IBM started making passive income, which is awesome.

Stage 3: Cloud

This became more and more popular and it got more mainstream for startups to rent out servers. Then bigger companies like Google, and Amazon saw an opportunity to make more active money by providing additional services along with resources. And this is what is called Cloud.

Career Prospects in Cloud

Cloud is a really booming technology, with the big and small tech companies now shifting to Cloud-based architecture. And this shift has generated the requirement for skilled Cloud engineers, who are proficient with cloud technologies and platforms like GCP.

In fact, the Cloud Computing Market is estimated to hit USD 791.48 Billion by 2028. So, if you’re looking to build a career in cloud computing, there’s never been a better time to do so.

Salary? Let’s see how much a Cloud architect makes in Bangalore India:

Yes, 21L per annum 🤑.

Google’s philosophy

Google has this very interesting philosophy:

“Every Company is a Data Company”

In the future, every company regardless of its size and technology will differentiate itself from its competitors through technology. And that technology will be in a form of software and great software always revolves around data. So, every company in some way will be a data company. And cloud gives you the services to manage large volumes, velocity, and variations of data.

With platforms like GCP, you can further analyze those data and gain further insights. By leveraging the power of data analytics and machine learning, businesses can gain valuable insights into customer behavior, market trends, and internal operations. This data-driven approach can help companies identify opportunities for growth, improve efficiency, and ultimately drive business success.

How do people use the cloud?

Cloud is usually broken down into 3 different umbrellas.

Infrastructure as a Service (IaaS)

This is basically Time-Sharing. In simple terms, it allows users to access computing infrastructure, such as servers, storage, and networking, without having to purchase and maintain physical hardware themselves.

Platform as a Service (PaaS)

This takes it one step further. Previously if you had to add 10 more servers, you could reach out to those IaaS providers and ask for more 10 servers. After that, you will have to manually handle the deployment and setup. But with this PaaS, all they need is the code of the application. The providers will take care of handling the load, setting up the OS, and everything and you only have to focus on the business logic.

Software as a Service (SaaS)

For SaaS, you can think of simple software that is available for users to use like this blog hosting tool which is providing a service for users to publish and read blogs.

In the SaaS model, the software provider is responsible for maintaining the application, including security, availability, and performance. Customers typically pay for SaaS on a subscription basis, with fees based on usage or the number of users.

Where does GCP fall in these categories? Well with GCP you can get your own servers, storage disks, and everything. But at the same time, it has features that handle them for you. So it covers both IaaS and PaaS domains.

Let’s dive into some of the GCP tools

Compute Engine

Compute Engine is a service that allows users to run virtual machines (VMs) on Google’s infrastructure. In simple words, it lets you run any code on Google servers. If let’s say you have a giant script that takes a couple of hours or days to run. Instead of doing it on your laptop and worrying about your battery dying, etc, you can create a VM instance for yourself and run the code over there.

Cloud Storage

A scalable and highly available object storage service for storing and accessing data on GCP. In other words, Google Cloud Storage is like your very own storage unit in the cloud, where you can keep all your digital “stuff” — like files, photos, videos, and more — safe and sound. But here’s the really cool part — Google Cloud Storage is designed to be super reliable and secure. It’s backed up across multiple servers and data centers, so even if one server goes down, your files will still be safe and accessible. Plus, you can control who has access to your files, so you can make sure only the right people can see and edit them.

Cloud Load Balancing

Cloud Load Balancing in GCP is a tool that will help you distribute the incoming network traffic to multiple virtual machines (VMs) or zonal or regional managed instance groups across different regions, zones, or across multiple regions. It balances the network traffic in real-time so that no single instance or zone gets overwhelmed, resulting in faster response times and better reliability.

In simple terms, let’s say you have an online store that is getting a lot of traffic. Without load balancing, all the traffic would hit a single server, potentially causing it to slow down or crash. But with Cloud Load Balancing, the traffic is distributed evenly across multiple servers, ensuring that the website stays up and running even during peak traffic times.

BigQuery

Big Query allows you to analyze massive amounts of data quickly and efficiently using a fully-managed, serverless cloud data warehouse. With BigQuery, you can easily load and query data sets, perform data transformations, and visualize your results using Google Data Studio or other third-party tools. It is highly scalable, secure, and easy to use, making it a popular choice for businesses of all sizes that need to analyze large volumes of data.

IAM and Admin Tools

IAM (Identity and Access Management) enables you to manage access control by defining who (identity) has what type of access (role) to which resource. It provides a centralized view of permissions and access for all cloud resources, including Google Cloud Storage, BigQuery, and Compute Engine. IAM allows you to create and manage service accounts, grant permissions to users, and set up custom roles to meet specific business needs.

And there are many more such tools. These tools have been used by companies to drive their software and manage the data that they receive.

Take Spotify for example. Spotify launched a music-streaming service in late 2008, surpassed 1 million customers in early 2011, and today offers 248 million monthly active users in 79 markets access to more than 50 million songs and podcasts. How are they handling this massive user base? Yes, with Google Cloud.

“Google Cloud removes a lot of the operational complexity from our ecosystem. That frees up time,” said Tyson Singer, vice president of technology and platform at Spotify.

“We can iterate quicker on key needs, like data insights and machine learning. Having infrastructure managed for us, with the lower-value details taken away, streamlines our ability to concentrate on what’s important to our users and give them the experiences they know and love about Spotify.”

Here is a great video on the Spotify case study:

The video teaches us a valuable lesson: “Why waste time building something that is already done?” Google is the world’s largest search engine, and it also happens to build the best data centers. And it makes sense; why would a company waste time building a data center when they can simply use a platform like GCP and focus on the business logic and idea they are developing?

Take a quick glance at the scale that Google Cloud works at:

How can you as a student use GCP?

As we say in the first video, you as a student can fully leverage the various tools that GCP provides for a variety of applications like training ML models, starting a server, and many more. Let’s get our hands dirty and learn how to use the Google Cloud Console.

Let’s go through it step by step:

1. Visit the Google Cloud Console and log in and check the terms and conditions.

2. Once you are in, you will see a dashboard something like this:

Currently, it’s pretty empty but once you start working on your project you will be able to see some stats.

Let’s dig into the Project Details section. Here we have:

• Project Name: This is to make it easy for you to differentiate between the projects that you have. It is not globally unique.

• Project ID and Project Number: These are in fact globally unique and different for different projects.

Now, let’s build a resource. Let’s create our own Virtual Machine. If you don’t know what a VM is, it’s like a computer inside a computer. It’s a software-based representation of a computer that can run its own operating system and applications just like a physical computer.

So, now click on the hamburger menu at the top-left. Look for “Compute Engine” and then click on “VM Instances”.

This will open up another dashboard that will show us all the instances that we have created. In my case, I don't have any and will create one so click on “Create Instance”.

That will open up a form for us where we will have to provide details regarding our VM instance.

First, we have to choose the name of our instance. You can name it anything you want, I will be leaving it as “instance-1”.

Then we have Region and Zone. This means where in the world our instance will be located. We already talked about how Google has its data centers spread across the world. We can leverage that scale and set up our VM anywhere we want. This way you can set up your VM closer to your user base which will reduce the latency. I will choose “asia-south1” cause that is closer to where I am right now.

Scrolling down you can select the Machine Type that would suffice your requirements.

I will just go with the e2-micro cause this is just for demo purposes. Next, in the Boot Disk section, you can choose the type of OS you want your VM to have.

You can skip the rest of it and then go to the bottom and click on “Create”. That will take you back to your dashboard and you can see the status of your VM instance.

Once the setup is complete you will see a ✅ in the status. And then click on SSH.

Basically, it will now open up a console and you can literally use the VM like your own machine, set up a server, run some scripts, etc. I created a Python Hello World program and executed it as you can see:

Once you are done with the work, it’s always a good practice to clean up the resources. So, let’s delete our VM instance. Head back to the VM instances dashboard. Click on the 3 dots and choose Delete.

It will ask for confirmation and then after some time, the VM instance that was assigned to you will be wiped out.

What’s next?

Now you must be very interested in learning more about Google Cloud. Well, Google has tonnes of great resources for you. Here are some of them:

1. https://www.cloudskillsboost.google/journeys This is a great place to learn Google Cloud. You will have step-by-step milestones to cover and learn about a particular tool.

2. Take part in events like 30 days of Google Cloud where you also earn cool swags along with learning about Google Cloud.

3. https://youtu.be/kzKFuHk8ovk?list=PLIivdWyY5sqKh1gDR0WpP9iIOY00IE0xL A playlist of Google Cloud Essentials.

Conclusion

So, now you have learned what cloud is, what GCP is, and how to use the platform to build resources. In conclusion, the Google Cloud Platform has revolutionized the way we approach computing and data management. With its vast array of tools and services, businesses of all sizes can harness the power of the cloud to scale their operations, improve efficiency, and drive innovation. From startups to global enterprises, GCP offers a world of possibilities, and with the right knowledge and skills, anyone can leverage it to unlock their potential.

So, don’t be afraid to explore, experiment, and learn. Who knows, maybe the next big thing will be built on the cloud, and it could be yours!

Why did I use the term "magical"? ✨

To me, coding is like casting a magical spell, through every line of code that brings a digital creation to life.

Believe it or not, a few lines of code propelled humanity to the moon, an unimaginable feat. The Apollo 11 mission that landed humans on the moon in 1969 was powered by a computer with a processor speed of just 1.024 MHz, which is slower than most modern calculators. The computer had only 4 KB of memory, which is less than what we use to store a single photo today. But the code that powered this computer was nothing short of miraculous.

Let me know in the comments if you know who this lady in the picture above is? Hint: she played a very crucial role in the Apollo 11 mission.

👋 Let's introduce ourselves first

I am Arnab Sen, a Software Developer from India. Currently while writing this, I have just 1 more month left for my college. In these 4 years, I have learned a lot and grown as a software developer. There are some achievements of mine that I am really proud of:

• Was one of the 24 people selected for the Coronasafe Fellowship by AICTE out of 50,000+ applicants and built an open-source healthcare platform.

• Selected for the Summer of Bitcoin project and spend the summer of 2021 contributing to Bitcoin-Core.

• Selected for SWE Internship at GOOGLE, and on top of that ended up with a PPO opportunity.

• Was selected for the XROS Fellowship which is powered by FICCI and supported by Meta.

• Interned at multiple remote startups: Dynopii, Cypherock, GMetriXR.

🏗️ Where it all started

The first time I did something close to programming was LOGO in class 4. For those who aren't aware, LOGO is an educational programming language designed to teach the basic concepts of programming. There is a turtle and we can give directions to the Turtle with keywords like "FORWARD", "RIGHT", etc and it will draw the path.

As a kid, I always loved creating different kinds of patterns in LOGO. Then in class 5, we had QBASIC in our syllabus. That's it, that kind of sparked a little fire in me, that maybe I should pursue Software Development in the future, cause I uncovered the magic beneath.

Luckily, I had internet access (the max speed was 500 kbps, but that was still a lot during those days). So I started learning a few other programming languages. My dad's colleague used to give computer science tuition to 11-12th standard students. He asked me if I was interested in joining his classes and I started doing so in class 6.

By the time I was in class 8, I already had the fundamentals of C and C++ strong. But after class 8 I joined an IITJEE coaching center, and just like everyone, I too had to leave behind my creative knacks to run in this rat race. With the JEE preparation pressure, I lost all interest in programming 🥺. That should have been the end, but...

In 11 standard I had computer science again in my course, and that rekindled the fire in me. During that time I also watched movies like, The Social Network, Jobs, Snowden, and many more, and I became passionate about coding more than ever.

I came across an olympiad competition called IOI (Internation Olympiad In Informatics), it's like IMO (International Mathematical Olympiad) but for Programming.

To be eligible for IOI, one has to qualify for INOI (Indian Olympiad of Informatics), and to get into that one has to qualify for either ZOI (Zonal Olympiad of Informatics) or ZCO (Zonal Computing Olympiad). But when I was in class 11, I already missed the ZCO or ZOI registration date. So I prepared for class 12th ZIO and ZCO. Qualified them and then went to INOI, but failed miserably. I left the exam center 1 hour before because I was unable to solve any questions. But that didn't stop me, cause I knew it was just the beginning.

So, far I did programming as a hobby or whenever I had a break. But after JEE I opted for CS in IIEST Shibpur, and now I was able to give my full attention to programming.

🎓 College Life

In the initial few days, I was really interested in building small games. Here are some of them.

Built a small TicTacToe game built with Python3 with one of my hostel mates.

Here is a fun fact: the logic behind Computer Moves is just a bunch of if-else statements 🤣🤣🤣🤣

Here is another beautiful yet very simple project that I built with simple HTML, CSS, and JS. It's based on Chaos Theory, it shows how pure randomness generates such beautiful patterns.

But as time passed, I started to focus on building stuff that could be useful to some extent. Like, a Discord bot for IPL Scores, a simple website to pdf converter, markdown to pdf generator.

For learning, I never really paid for any courses. There are so many great folks online who share their knowledge and are such amazing teachers. For learning web development especially I followed these channels and I feel they helped me strengthen my fundamentals:

• https://www.youtube.com/@freecodecamp

• https://www.youtube.com/@TraversyMedia

• https://www.youtube.com/@akshaymarch7

• https://www.youtube.com/@WesBos

• https://www.youtube.com/@academind

For theoretical topics related to computer science like Data Structures and Algorithms, Operating Systems, and Database Management Systems, I always preferred the MIT Open Courseware Courses. I think they are just goldmines for anyone looking to dive deep into these topics.

🤍 Met Open Source

When I was in my 2nd year of college, I came across open-source software and realized that through OSS I could make more impact with my knowledge. I was a part of the Coronasafe Fellowship where I got to work on multiple open-source healthcare projects.

One of the projects is Arike which is a software system for palliative care. The other project is called Life which acted as verified crowd-sourced emergency services during the peak of covid second wave. I am grateful to be a part of such a great cause and take my development skills to help people.

My other contributions can be found on my GitHub profile. Through contributing to open source, I have been able to gain valuable experience in real-world software development problems, collaborate with other talented developers, and learn new programming languages, tools, and frameworks. Contributing to open source has also allowed me to receive feedback from other developers and improve my ability to communicate my ideas effectively.

Open Source went on to change my career and brought me even better opportunities in the future.

Even though everything seems smooth, like every other thing in life there are always ups and downs.

If you are also interested to pursue a career in this domain of Software Development. Here are a few things to keep in mind, that I learned from my personal experience:

📚 Start with the basics

Before you jump into coding and building cool stuff, it's important to have a solid foundation of the basics. So, what are the basics? Well, at a minimum, you should have a solid understanding of at least one programming language. There are many languages to choose from, but some of the most popular ones include Python, Java, C++, and JavaScript. Each language has its own strengths and weaknesses, and choosing the right one for you will depend on your personal preferences and the type of projects you want to work on.

Also, it's worth noting that learning the basics isn't a one-and-done task. As you progress in your career, you'll continue to build on your knowledge of programming languages, data structures, and algorithms. The key is to start with a strong foundation and continue to learn and grow from there.

🧠 Keep Learning

The field is constantly evolving, and there are always new technologies and tools being developed. As a result, it's crucial to stay up-to-date with the latest trends and developments to remain competitive and relevant.

Reading industry blogs and publications is a great way to stay informed about the latest trends and developments. Many tech publications offer daily or weekly newsletters that summarize the latest news and events in the industry. Following these publications can help you stay informed about the latest advancements in the field, and help you identify emerging trends that may be relevant to your work. If you prefer learning from videos more then my suggestion would be to start watching conference talks.

I also love to watch documentaries like the ones by Honeypot.io. The amount of insights that are shared in these 30 mins Netflix-styled documentaries is just crazy, and trust me you will never get bored.

🏔️ Embrace Challenges

Challenges are a natural part of any career, but in the tech industry, they can be particularly prevalent due to the constantly evolving nature of technology. However, it's important to embrace these challenges as opportunities for growth and development.

One way to embrace challenges is to take on projects that push you outside of your comfort zone. By taking on projects that are unfamiliar or challenging, you'll be forced to learn new skills and adapt to new situations, which can help you grow both personally and professionally.

🎭 Dealing w/ Imposter Syndrome

This is a very very prevalent topic, especially in the software development domain.

Imposter syndrome is a nasty little bugger that can creep up on anyone in the tech industry. It's that feeling that you're not qualified or good enough for your job, even when all evidence points to the contrary. But here's the thing: you're not alone. Lots of successful people in the tech industry have felt imposter syndrome at some point in their careers. It's a pretty common experience.

So, the next time you start feeling like a fraud, remember that you're not alone, and remind yourself of all the awesome things you've accomplished. Seek out a mentor or coach for guidance, and connect with others who have gone through the same experience. With these strategies, you can kick imposter syndrome to the curb and achieve success in the exciting and dynamic world of tech!

Here is a great video by Ted-Ed on Imposter Syndrome:

🤝 Share your learnings

Sharing knowledge as a software developer is one of the most rewarding things you can do! Not only does it help you solidify your own understanding of a topic, but it also benefits the wider community and can even lead to new career opportunities.

Blogging is a great way to share your knowledge and experiences with others. By writing about your insights, tips, tricks, or lessons learned on a particular topic, you can help others who are facing similar challenges or looking to learn new skills.
One of my blogs where I shared my resources and how I prepared to land a job at Google got featured on Hashnode and has around 1k views (even Hashnode's cofounder commented on the blog).

Link to the blog 👇

Plus, blogging can help you establish yourself as an authority in your field and even lead to opportunities for speaking engagements or consulting gigs. My blogs on Bitcoin helped me get a web3-based internship.

Speaking at conferences or community events is another fantastic way to share your knowledge and connect with others in the industry. Not only do you get the chance to present your ideas and insights to a live audience, but you also get to network with other professionals and learn from their experiences as well. Plus, speaking engagements can be a great addition to your resume and help you stand out when you're looking for new job opportunities.

🎬 The End

As I reflect on my tech journey, I realize that it's been a rollercoaster ride with its fair share of ups and downs. But it's also been a journey filled with wonder, innovation, and creativity. I've learned that being a software developer isn't just about writing code. It's about problem-solving, collaboration, and being passionate about making a difference in the world.

If you're just starting your own tech journey, remember that it's never too late to start. Whether you're fresh out of school or a seasoned professional, there's always something new to learn. Be open-minded, stay curious, and never stop exploring. And always remember to keep your eye on the end goal: creating something that can change the world.

And always remember "YOU GOT THIS !!"

• How did you crack Google SWE Internship?

• Which resources did you follow?

• Can you share some interview tips for my upcoming interview?

• Was it on-campus or off-campus?

... and many more. So, I decided to pen down my thoughts so that I can share this article next time someone asks me anything about my SWE Internship.

What can you expect from this blog?

These are the few things that I would like to cover in this blog

• How I optimized my resume.

• How I approached (or, usually approach) referrals.

• Resources I referred to while preparing for DSA.

• Interview tips

NOTE: Although this article is about Google, the recommendations will be useful if you apply to other product-based companies.

Introduction

Hi, I am Arnab Sen, a Software Developer from India 🇮🇳. I interned at Google as an SWE for the summer of 2022 and even grabbed a Pre-Placement Opportunity. I will join Google India as a Full Time, in July 2023. When I was in my third year Google, unfortunately, didn't visit our campus. So I had to take the off-campus route to land at Google and here is my journey:

Decoding the Google Application Process

Let's understand how Google usually hires candidates for the Software Engineering Role:

1. Resume Shortlisting

2. Online Aptitude Round

3. Telephonic Round

4. Technical Interviews

5. Hiring Committee

I'll go through each round and talk about what happened and how I prepared.

The same rounds are usually followed by most of the Product based companies like Microsoft, Amazon, Meta .etc.

Round 1: Resume Shortlisting

It was in Aug 2021 that I came across the Google India SWE Intern opening in Google's career portal and I applied with a referral from one of my seniors.

Resume building tips

Your resume is the first thing that creates an impression. Even if you are super talented and skilled, a poor resume can negate all your abilities. So, you need to create a kickass resume that showcases your talents and impresses everyone!

Originally I used to create my resume in LaTeX but later on, it became a hassle to maintain and make updates. Hence I moved on to this website called flowcv.com.

Their interface is user-friendly and offers pre-defined templates while allowing ample options for personalizing your resume. Check out the template I personally used.

I prefer a single-column resume. If you want to stick to LaTeX and you are looking for a resume template, my suggestion would be to use DeedyCV. It looks very professional and compact.

Resume tips to keep in mind

Here are some of my tips for your resume:

• If you wanna look like a pro, make sure your stuff doesn't have any spelling or grammar mistakes. You can use tools like Google Docs spelling check or Grammarly to help you out.

• Just get straight to the point and don't use fancy words. Nobody cares about your language skills, they just want to know what you can do as a developer.

• It would be great if you could add some proof to your work. If you have built a project, published a paper, or completed a certification, you must provide a link for the respective work.

• Don't stuff everything into one page of your resume. Leave some space to breathe. It's totally fine to use 2 pages if you need to. I checked with some Google peeps and recruiters and they all said it's all good to have a 2-page resume if one page just ain't cutting it.

Now, let me share with you some hacks that can use:

• If you don't have achievements, try to take part in Leetcode, GFG, Hackerrank, or HackerEarth contests or contests of college fests. You can put those ranks in your achievements. To know the upcoming contests, check out clist.by.

• If you don't have projects, try to take part in hackathons with your peers. "I can't think of a good project" is what I hear from a lot of my juniors. Right now, if I ask you to think of anything, you won't be thinking of anything. But if I asked you to think of a fruit, you might be thinking of a banana, apple, mango, etc. Every hackathon has a unique theme, so by the end of the hackathon you will have a somewhat unique project. You can find such hackathons at Devpost.

• If you don't have enough work experience, pick up a good open-source project from GitHub and try to make contributions to that for like 1-2 months. By the end of it, you will have enough experience and contributions that you can put it under the "Open Source Experience" category.

If you are interested to know what my resume looks like, check out: arnabsen.dev/resume.

Elevate your referral game

Companies like Google actually let their own employees refer potential candidates. Although a referral doesn't necessarily guarantee you a job offer, it does mean your application will stand out from the rest of the applicants, which can only mean one thing - your chances of getting hired just got a whole lot higher. In some cases, a Good Resume + Referral can fast forward in the application rounds. In my case, I was directly skipped to the Technical Interview rounds.

Now few things to keep in mind:

• The number of referrals doesn't matter. It's the quality of the referral that makes the difference. I'm familiar with Google's internal referral portal. Employees must respond to questions such as, why do you believe this candidate is a good fit, have you previously worked with the candidate, and so on. Hence, if the person referring you understands your strengths and weaknesses, the referral will be stronger, and your chances will be better.

• I was lucky enough to have a senior in my college who was ready to refer me. We knew each other since day 1 of my college so he was able to write a good referral for me. Shoutout to Suryansh Gupta ✨. So you can always reach out to your seniors regarding referrals.

• If you don't have such seniors in your college, no worries, you can get referrals over LinkedIn, Twitter, etc. When asking for referrals make sure you provide all the necessary information:

  • The job id of the role you are applying for.

  • Your resume.

  • Your achievements and relevant skills.

  • Why do you think you are a good fit for the role?

College tip: active interaction with seniors in your college can have a positive effect on your career. So, don't be shy.

Now, let's say your resume got shortlisted. Then what?

Round 2: Online Aptitude Round

Even though I skipped the OA round, here are some preparation tips:

• Google usually asks pretty standard questions which are of the Leetcode Medium-Hard difficulty. So don't just solve easy questions. Also, no need to waste time by solving very difficult questions either.

• Make sure you know all the standard algorithms, I will share the list after this.

• I also prepared by participating in Competitive Programming contests, especially from Atcoder and LeetCode. This was done to ensure that I could answer the questions swiftly and within the time limit, exactly like an actual OA.

My advice would be to go through the past OA rounds questions. Here is the list I referred to for Google's questions:

• https://leetcode.com/discuss/interview-question/352460/Google-Online-Assessment-Questions

• https://algo.monster/problems/google_online_assessment_questions

• https://www.codingninjas.com/codestudio/problem-lists/top-google-coding-interview-questions

• https://www.interviewbit.com/google-interview-questions/

You can find similar collections of questions for other companies as well.

Standard topics to prepare for OAs

• Binary search

• BFS/DFS/Flood fill

• Tree traversals

• Hash tables

• Linked list, Stacks, Queues, Two pointers/Sliding window

• Binary heaps

• Dynamic programming

• Union find

• Ad hoc/string manipulations

• Trie (good to know)

• Segment trees/Fenwick trees (good to know)

• Bitmasks (good to know)

Once you are done with the OA you will be taken to the 3rd round.

Round 3: Telephonic Round

So, for this round, they're gonna call you up and hit you with some theoretical Data Structures and Algorithms questions. Some might even have multiple-choice answers. They might throw in some programming language questions too. Here are some examples of questions that you can expect to get an idea:

• What are the pros and cons of using an adjacency list over an adjacency matrix for a graph problem?

• What is the data structure used in a priority queue?

• What is the difference between Binary Search Tree and Heap?

• How is a C++ map different from unordered_map from an implementation point of view?

• Which algorithm would you use for a network flow problem?

• What are the time complexities of Bellman-Ford and Dijkstra's algorithm?

If you have a strong fundamental understanding of Data Structures and Algorithms then these won't be very difficult for you.

Resources to study DSA

I studied DSA mainly from these two sources:

1. Introduction to Algorithm by CLRS (a.k.a the Bible of DSA)

1. MIT 6.006 Introduction to Algorithms, Fall 2011

   

So, I made sure to take notes while watching those videos and reading the book chapters. And when I got stuck with any problem, I just Googled it and checked out the first link - be it an article or a video.

Once you complete the telephonic round you will be invited for the technical interview rounds.

Round 4: Interview Rounds

It was on 13th September I received a mail stating that I have my interview lined up after a week. I freaked out a little. Because first, I didn't expect to hear back from them. And second, how am I supposed to prepare for a Google interview in 7 days?

I took a smart approach. I checked out other people's experiences with Google Interviews and saw that they like to ask about DP, Graphs, Binary Search, Lazy Sum, and Sliding Windows stuff. I went back and reviewed all the notes I took from that MIT course and honed in on those types of questions. I practiced topic-wise from LeetCode itself.

Another question I get is if LeetCode premium is really worth it. To be honest, in today's time it isn't anymore. There are tonnes of other platforms which have similar questions for free that are only available in LeetCode premium. But 2 years back, I felt spending Rs 10.5k for 1 year of LC premium might be worth it. Because if I am able to crack a job like Google, that amount invested will be just a fraction of the monthly salary. So, I got 1 year of LC premium.

My style of practicing for interview

After discussing with my seniors I realized that solving a question in an interview setting is completely different than practicing questions by myself. Most of these Google interviews happen on an editor-based portal where you cannot run the code and it also doesn't support intellisense. So, I would create a Google doc where I would solve all the questions and write code. This way I didn't have the privilege of syntax highlighting or intellisense. As they say, "Hope for the best, Prepare for the worst".

The strategy of breaking down Google interviews

A Google Interview happens for 45 mins and they usually ask 2 questions. So, I divided the total time into (20 + 20 + 5) mins. 20 mins for each question and 5 mins as a buffer (for an emergency like network issues or Q&A at the end).

Within those 20 mins, I would spend 3-5 mins understanding the question asking clarifying questions, and explaining my approach. 10 mins for implementing the optimized solution and 5 mins for dry running or debugging. Note that you have to come up with the optimal solution as quickly as possible. So, I would insist on not wasting time trying to implement the brute-force solution.

So, even during practice when I was solving the questions myself I followed the same strategy. I would pretend that I am in an interview setting. I would try to read my thoughts out loud and explain everything. I even made the habit of doing a dry run every time I wrote the final code for a problem.

Mock Interviews are an absolute necessary

To succeed in any field, preparation is key. Just as astronauts undergo rigorous training to prepare for the unique conditions of zero gravity, so too must job seekers prepare for the challenge of a job interview. One effective way to do this is by practicing mock interviews. That way you can know your weakness, and you can prepare for them before the actual interview.

I also watched a lot of Mock Interview videos on Youtube to give me an idea of what it would be like during an interview setting. I created a playlist of all the videos that I referred to.

Link to the playlist 📼.

I gave a bunch of mock interviews (around 5-6) by taking the help of my seniors and using online free platforms like Pramp.

Importance of clarifying questions

The initial question that will be given to you, will be intentionally very vague. They will expect you to ask some clarifying questions to identify the fine details of the question. Let's take a very simple example, you are given an array of n numbers, and you have to find the subarray with the maximum sum.

Now, your first thought might be using Kandane's Algorithm. But wait, what if all the numbers are positive? In that case, the answer is simply the entire array. As you can see one small change in the constraints, had a big effect on the final solution. So always practice asking clarifying questions.

Interview tips to keep in mind

• Treat the interview more like a conversation with a fellow software developer and not like a viva. This will also help you tackle your nervousness.

• During the interviews, you should also think about other ways to improve the solution. Share the different options or tradeoffs that you're considering.

• Think out loud because that way the interviewer will know your thought process and can give you signals if you are going in the wrong direction.

• Discuss tradeoffs if possible. How would you improve your solution? How do you make it faster? How to optimize the space complexity? Know the time and space tradeoffs of the solution/data structures that you are picking.

• Try to avoid taking hints, but always remember solving with hints is much better than not solving at all.

• Avoid syntactical errors like missing semicolons, not closing parenthesis, wrong indentation, etc. Even though they are not deal-breakers, writing high-quality code can showcase your skills as a developer.

• Finally, use proper variable names and if required add meaningful comments. But remember you have limited time so just don't spend minutes trying to come up with an appropriate variable name.

• Make it a habit to always calculate the Time and Space complexity of your solution.

If you don't remember the exact syntax of a library class or method, that is fine. You can just let the interviewer know, and replace it with a meaningful substitute.

All of these points might be hard to keep in mind during the interview, which is why the more mock interviews you give, the better you get at remembering these subtle points.

Here are some more resources that I referred to:

• Big-O-Cheatsheet

For some last-minute revision, I also followed this collection of LC questions.

https://seanprashad.com/leetcode-patterns/

My interview experience

I cannot reveal the actual questions because of NDA, but in my first interview, I had a question on Sliding Windows and Dynamic Programming.

In my second interview which was held the next day, I had 3 questions. One was a Scheduling Problem based on the heap, the second was again a DP question. I was able to quickly answer these 2 questions, and there were around 10-12 mins left. So the interviewer gave me another DP question for which he just asked me to write the recursive function and how I would memoize it. I was able to do that as well.

Both my interviews went well from my end, I didn't take any hints, wrote high code quality, and was able to successfully dry run all my solutions with edge cases.

Round 5: Hiring Committee

After I finished my interviews, I had a long wait. According to what I've heard, there is a hiring committee that comprises four to five persons that have prior interview experience and are familiar with the hiring standards. They would review your resume, your code during interviews, and the feedback from the interviewers before making a final decision.

For me, it was a long long wait. I very clearly remember the day. It was Maha Shasthi (the onset of Durga Puja), I woke up to a mail saying:

"Congratulations on your SWE Summer Intern offer with Google India!"

It was a perfect blessing to me. I was ecstatic and overjoyed. It was an absolute dream come true moment for me and my family. Looking back I remember I felt very depressed when I was rejected by Microsoft twice. I gave my best in the Microsoft Engage Aptitude round yet wasn't even selected for the project building round. Then again when Microsoft came on-campus I was rejected after the OA round which still baffles me. But as they say "Whatever happens, happens for Good".

So whenever something unexpected or bad happens to you, remember these lines:

I hope you liked my blog, do follow me for more such content. I wrote another blog sharing some tips on how to convert your internship to PPO.

You can learn more about me at arnabsen.dev/about. Have a nice day !!

It happened to me, and after speaking with one of my seniors, I discovered that this condition is known as burnout. Then I began investigating this particular topic, and it turns out that it's a rather common occurrence that can impact anyone, particularly those in high-stress professions like software development. However many individuals are unaware of it and fail to take the required steps to recover from it. The sooner you recuperate, the more energized you will feel about your task, which will eventually result in better results and increased productivity.

In this article, I'll try to go over some of the indicators that you might be burnt out, and then I'll provide some suggestions that helped me get back on track. It should be noted that burnout can progress to depression, in which case you should consult a medical expert.

A little note: Because I am a Software Developer, my thoughts on dealing with Burnout will be slightly skewed and will be more relevant towards this profession.

My burnout experience

I was attempting to achieve several things at the same time in the second half of 2021.

1. I was a mentee at Summer of Bitcoin and I was working on adding USDT (User Statically-Defined Tracing) tracepoints to the UTXO (Unsigned Transaction Output) set of Bitcoin. Furthermore, I had no prior knowledge of Bitcoin, tracepoints, or working with a huge C++ codebase.

2. My internship placements were ongoing, and I was being turned down both on and off campus. I was devastated when I was denied Microsoft's Engage program and then again from their OA round.

But at the time, I was all about having a decent internship, so I went all in. Ended up cracking Wells Fargo SDE Internship and Google SWE Internship. Also after completing my Summer of Bitcoin project got an internship opportunity at Chaincode Labs.

But then I changed from being a highly driven, hyperactive person to someone who had very little ambition to accomplish anything and felt sluggish all day. My productivity dropped, which made me feel even worse, and I became trapped in a vicious loop.

Signs of Burnout: How to Recognize Them Early

These will be some of the early signs that you will notice:

1. Decreased productivity: You will find yourself struggling to complete tasks or meet deadlines.

2. Lack of motivation: There is a feeling of disengagement or lack of interest in your work. You might not even find your work pleasurable anymore.

3. Chronic fatigue: You're constantly tired, despite getting enough sleep.

4. Irritability: Burnout can lead to feelings of frustration, anger, and irritability.

5. Decreased quality of work: Burnout can cause you to make more mistakes or produce lower-quality work than you normally would.

Now, let's understand the causes of Burnout and how to avoid them.

Avoiding Burnout

Maintaining balance

In life, balance is crucial. It's incredibly easy for software engineers to work longer hours because all we need is a laptop and an internet connection. As a result, we occasionally forget our limits and push ourselves. It may produce short results, but it is hazardous in the long run. If we spend all our time and energy at work and neglect our personal life and self-care, it can lead to physical, mental, and emotional exhaustion.

As a result, we must strike a good and sustainable balance between many aspects of our lives, such as jobs, hobbies, family, and friends. This varies from person to person, therefore you must reflect and establish ground rules for yourself.

• Post 6 pm, no work-related calls/talks.

• Go outside to play some sports with friends.

• Taking a vacation once in a while and travelling.

• Having dinners with family or friends.

Taking Breaks

Taking breaks is incredibly important in avoiding burnout. Breaks allow us to rest, recharge, and reset, which can help us manage stress and prevent exhaustion. When we continuously work without breaks, we can become overwhelmed, stressed, and exhausted, leading to burnout.

Breaks can take many forms, from short breaks throughout the day to longer vacations. Short breaks, such as taking a few minutes to stretch, walk around, or chat with a colleague, can help us refocus and re-energize during the workday. Longer breaks, such as weekends, holidays, or vacations, allow us to disconnect from work and engage in activities that bring us joy and relaxation.

Next time when you are stuck fixing a bug, go ahead take a walk and come back. 8 out of 10 times you will be able to fix it.

Having a clear understanding of your work

Employees can experience stress, dissatisfaction, and burnout when they are confused about their tasks and responsibilities or are given ambiguous expectations.

Employees who lack clear direction may struggle to prioritize their tasks or become overwhelmed by competing requests. This can lead to feelings of powerlessness and a lack of control over their work, which can lead to burnout. Workers may also feel that they are not fulfilling expectations or that their efforts are not being recognized, leading to feelings of discontent and disengagement.

Clear communication is the solution. If you have any questions about the tasks, you should speak with your manager, senior, or leader. If you believe you will have problems completing the work, mention this as well. This will be beneficial in two ways. One, your manager will have a clear knowledge of your comprehension and will be able to provide extra resources to assist you. Two, you will feel relieved, which will lower your stress levels.

Self-Care

Finally, you must take care of yourself. By practicing self-care, individuals can reduce stress levels and promote overall well-being, which can help prevent burnout. Some self-care practices that can help prevent burnout include:

1. Prioritizing sleep: Getting enough sleep is crucial for overall health and well-being and can help individuals manage stress levels. Consistently getting 7-8 hours of sleep per night can help prevent burnout.

2. Engaging in physical activity: Regular exercise can help reduce stress levels, promote relaxation, and improve overall health. Engaging in activities such as walking, jogging, or yoga can help prevent burnout.

3. Practicing mindfulness: Mindfulness practices, such as meditation or deep breathing, can help individuals manage stress and improve emotional well-being. By staying present at the moment and reducing distractions, individuals can prevent burnout.

4. Eating a healthy diet: Eating a balanced diet that includes whole foods, fruits, and vegetables can help individuals maintain physical and mental health. A healthy diet can help individuals manage stress levels and prevent burnout.

5. Engaging in leisure activities: Engaging in activities that bring joy and relaxation, such as reading, painting, or spending time with friends, can help individuals manage stress and prevent burnout.

Seeking professional help

If things are getting out of hand, it's always a good idea to seek expert assistance. Counseling or therapy can offer individuals assistance and guidance in dealing with stress and avoiding burnout. The faster you recuperate, the better you will be able to focus on your task, leading to an increase in total productivity.

Final Thoughts

I have been following these few advices that I received from my seniors and leads at Google, and to be honest the outcomes have been fantastic.

• I completed my Google internship securing a PPO.

• I worked as a Technical Content Writer for Codedamn and authored few high-quality posts. You can find them right here.

• I won Showwcase's Develevate event.

• I joined a Singapore-based web3 startup called Cypherock as an SDE Intern.

• I joined Pupilfirst as a Teaching Assistant.

• At the same time, I've been taking care of myself by going to the gym on a regular basis and eating healthily.

Yet I didn't experience any burnout along the process. I hope these few tips will assist you in increasing your work productivity and avoiding burnout.

TLDR;

Maintaining balance, taking breaks, having a clear understanding of work responsibilities, practicing self-care, and seeking professional help if needed are all effective strategies to prevent and recover from burnout. Prioritizing our well-being and seeking help when necessary is essential to achieving long-term success and productivity. Remember, taking care of ourselves is not a luxury but a necessity, and it is essential to make self-care a priority in our daily lives.

All the images in this article were generated by Dall-E.

I have been wanting to write this blog for a long time, I will try to include all the important points that I followed and have learned during my internship which helped me bag a PPO. I will break them down into 2 parts mainly, Technical and Non-Technical. Before that let me give you a little background.

Background

I applied for Google Summer SWE Intern Opening on the Google's careers portal with a referral. My Resume got selected and I was called for 2 Rounds of Interviews. After that, I got the offer and interned at Google for 10 weeks remotely with the Google Ads team. And finally, I was able to crack the PPO and got a full-time offer from Google as an SWE. I am planning to join them in July 2022.

From

To

Conversion Process

Before sharing the pieces of advice, I feel it's important to learn about the conversion steps. This will vary from company to company but overall the process remains the same.

1. Every Intern is assigned a project and a mentor (at Google we call them host) and a co-host. The project is usually a part of the product that the team is working on. The mentor is a senior engineer who will be your guide throughout the internship. You will be working with your mentor and other engineers on the team.

2. There will be two evaluations during the internship. The first evaluation called Mid Evaluation happens after 5 weeks (since my internship period was 10 weeks) and the Final Evaluation happens in the last week of your internship. The Mid Evaluation is more of a check-in to see how you are doing and if you are on track. The Final Evaluation is more of a review of your work and the project. The host and co-host also get to give feedback on your performance and your project progress. This feedback plays a huge role in the final conversion.

Technical Advice

1. Learn the basics

You must have strong problem-solving abilities to work on the project that will be assigned to you. There are some basic Computer Science concepts that one should be familiar with. I will list them down below.

a. Version Control Systems

No matter which company you are working at, you will be using a version control system. You must be familiar with the basics of version control systems. I would recommend you learn Git and GitHub. You can learn it from here.

At Google, we didn't use git, instead, we used Mercurial. But the fundamental concepts are the same. If you know how git works, learning a new version control system will just require you to learn the commands specific to that version control system.

b. Basic Commandline

There is a high chance that you will be working in a cloud environment and you must be comfortable with the terminal. You should be able to navigate through the file system, create directories, create files, delete files, etc. You should also be able to use basic commands like cat, grep, sed, awk, etc. You can learn about these commands from the man pages. For example, to learn about the cat command, you can run man cat in your terminal.

c. Data Structures and Algorithms

You should be familiar with the basic data structures and algorithms. You should be able to implement them in your language of choice and most importantly be aware of when to use which data structure and algorithm.

Note: You don't need to be an expert in these topics. You just need to be familiar with them. But most of these companies have DSA questions in their interviews. So, you must be familiar with them.

I love the MIT 6.006 Introduction to Algorithms lectures, they have helped me build the foundation of algorithms. You can find the playlist here:

2. Knowing the syntax of the language you will be using

Sometimes you can ask the recruiter about the tech stack that you will be using. But most of the time, you won't be able to know the tech stack beforehand. But, you can still learn the basics of the language that is popularly used in the company. For example, I knew that Google uses C++, Java, and Golang a lot in their backend. I already knew C++ and Golang. So I learned the basics of Java before joining Google. This helped me save a lot of time during the internship picking up the language.

3. Learn Design Patterns

Design patterns are important for developers to understand because they provide proven solutions to common software design problems. By using design patterns, developers can improve the quality, reusability, and maintainability of their code.

While you don't need to know every design pattern in existence, it's helpful to have a good understanding of the most common design patterns and when to use them. This knowledge can save a lot of time and effort by allowing you to reuse tried-and-true solutions to common problems, rather than having to reinvent the wheel every time you encounter a new challenge.

4. Write clean code

I can't stress this enough. Writing clean code is very important especially if you are interning at a big organization like Google. Every change you make goes through a rigorous review process. If your code is not clean, it will be rejected. So, you must write clean code. Here are some points that you should keep in mind while writing clean code.

• Keep it simple: When writing code, it's important to avoid unnecessary complexity. Simple, easy-to-understand code is usually easier to maintain and debug.

• Use clear and descriptive names: Choosing clear and descriptive names for variables, functions, and other elements of the code can make it much easier to read and understand. Avoid using abbreviations or single-letter names, unless they are well-known and widely understood.

• Use comments: Comments can be a helpful tool for explaining the purpose and functionality of different parts of the code. However, it's important to use comments sparingly and only when they are truly necessary, as too many comments can make the code harder to read.

• Follow a consistent style: Consistency is key when it comes to writing clean code. Using a consistent style for indentation, spacing, and other elements of the code can make it easier to read and understand. It's also a good idea to follow established coding standards and guidelines, such as those provided by the programming language's community or the company you work for.

5. Learn testing

Testing is an important part of software development. It's important to test your code to make sure it works as expected and to catch any bugs that might have been introduced during development. Testing can also help you to ensure that your code is clean and easy to understand. There are different types of testing like unit testing, integration testing, etc. Knowing these testing types will help you a lot.

Here is a free course on Test Driven Development.

6. Know how to write a Design Doc

At Google, we heavily focus on the Design Doc. Even before we write a piece of code for a particular implementation we have to make sure we have analyzed every possible edge case, and technical and non-technical requirements. Design documents also help with communication among team members and can serve as a reference for future maintenance and updates to the software.

Certain key points need to be covered in a Design Doc like Architecture, Requirements, Design, User Interface, Testing .etc.

Here is a great post that talks in depth about Design Docs.

This is a sample template of a Design Document.

Non-Technical Advice

In addition to the technical advice, I would also like to share some non-technical advice that I followed during my internship.

1. Be proactive

This was something my host pointed out in my mid-evaluation. I was being too dependent initially on my mentor, waiting for her to tell me the next steps. But later I took the advice. So I started planning my work, assigning tasks to myself, and taking the initiative to solve the problems. But obviously, I kept my mentor in the loop. This takes me to the next point.

2. Keep your mentor (host and co-host) in the loop

Remember, your host and co-host will be evaluating you at the end of your internship. Also, they have their deadline and they will be busy with their work. So, you must keep them in the loop. You should be able to communicate with them and tell them what you are working on and what problems you are facing. This will help them to guide you better.

We had a weekly meeting with our host and co-host but even then I would update them on my progress every alternate day so that they are aware of what I am working on. I would also disclose the problems I faced and the solutions I found.

There is a really good article on Writing a brag document that I would recommend you read.

3. Be open to feedback

No matter how big a company you are interning at, you should remember that you are still very new to this field and there is a lot to improve. Feedback from your host/co-host or other team members can help you identify areas of their work that needs improvement. It might be difficult to accept the feedback at first but you should be open to it. You should be able to take the feedback positively and work on improving yourself.

Not always your mentor will reach out to you to give you feedback. If required you might have to reach out to them and ask for feedback. You can ask them to give you feedback on your code, your communication skills, your problem-solving skills, etc. This also shows that you are proactive and you are willing to make improvements.

A good place to get feedback from your host/co-host is through 1:1 meetings. Here is a good video on The Art Of The 1:1 Meeting.

4. Time management

Plan out the tasks that you will have to do, assign priority to them and then start working on them. This will help you to manage your time better. You should also be able to estimate the time that you will need to complete a task. Initially, you may end up overestimating or underestimating the time but as you gain more experience, you will be able to estimate the time better.

5. Ask questions

An internship usually lasts 10-12 weeks, which is not a lot of time. So if you are stuck at something, you can't afford to lose time. You should be able to ask questions to your mentor and other team members. But before that make sure you have done a fair amount of research on your end as well.

You have to ask the right questions to get quick answers. If you came across an error message, explain properly how to reproduce the error, and what the error message is. Share the code snippet that might be causing the error, or share the files you are working on. Further, the approaches you have already taken to solve the problem. This will help the person you are asking the question to understand the problem better. Also, make sure you are asking the question in the right channel. For example, if you are stuck on a coding problem, you should ask your mentor or other team members. But if you are stuck in a process, you should ask your host/co-host.

Here is a great article on How to ask questions the smart way.

Podcast w/ Aniket

I was recently on a podcast where we went through some of these points in more depth. Watch the video here:

Conclusion

These are some of the things that I followed during my internship. I hope this article will help you to prepare for your internship. One thing you should also keep in mind is to enjoy the internship. It is a great opportunity to learn and grow, network with people and build your resume. Don't burnout yourself before or during the internship. Take breaks, go out with your friends, play games, watch movies, etc. This will help you to stay motivated and perform better.

If you have any questions, feel free to ask in the comments. I will try my best to answer them. Also, if you have any other advice that you would like to share, feel free to do so in the comments.

Introduction

Redis stands for REmote DIctionary Server.

In simple words, Redis is an in-memory data structure store and is mainly popular for its very low-level latency. The fact that it is in-memory means that the time taken to access the data is very low compared to in-disk databases. This can be useful for applications that require fast data access and manipulation, such as real-time bidding systems or gaming servers.

Redis was created by Salvatore Sanfilippo, also known as "antirez", in order to provide a fast, in-memory database that could be used for a variety of purposes. It has since become widely popular for its performance, reliability, and flexibility.

It is because of its low latency and high performance that Redis is used by a lot of major companies, such as:

• Twitter

• Snapchat

• GitHub

• Pinterest

• Craigslist

How does Redis work?

Redis is a key-value store, which means that it stores data in the form of key-value pairs. The key is used to access the value, and the value can be of any type, such as a string, a list, a set, a sorted set, a hash. These data types are supported natively by Redis and provide a rich set of commands for manipulating and querying the data.

In addition to these data types, Redis can be extended with Redis Modules. Some of these modules are provided by Redis Labs, and some are provided by the community. Some of the modules provided by Redis Labs are:

• RediSearch: This module provides full-text search capabilities for Redis, allowing users to index, search, and query text data stored in Redis similar to Elasticsearch.

• RedisGraph: This module provides graph database capabilities for Redis, allowing users to store, query, and manipulate graph data structures similar to Neo4j.

• RedisTimeSeries: This module provides time series data management capabilities for Redis, allowing users to store, query, and visualize time series data similar to InfluxDB.

• RedisJSON: This module provides native support for JSON data in Redis, allowing users to store, manipulate, and query JSON data using Redis commands similar to MongoDB.

• RedisBloom: This module provides probabilistic data structures for Redis, allowing users to store and query data using bloom filters, count-min sketches, and other data structures that can approximate data sets.

These are just a few examples of the many Redis Modules that are available. There are many more modules that provide a wide range of additional functionality for Redis. For a complete list of available modules, you can visit the Redis Modules registry at the following URL: redis.io/resources/modules.

This makes Redis a multi-model database.

Installing Redis

Redis can be installed on Linux, Windows, and macOS using the following steps:

Linux

Begin by updating your local package index with the latest package information:

curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list

sudo apt-get update
sudo apt-get install redis

Windows

Redis is not officially supported on Windows. However, you can install Redis on Windows for development by following the instructions mentioned in the website.

MacOS

Install the Redis server and command-line tools using Homebrew:

brew install redis

Once the installation is complete, you can start the Redis server using the following command:

brew services start redis

To enable Redis to start automatically when the system boots, you can use the following command:

brew services enable redis

Once, you have Redis installed, you can start it by running the following command:

redis-server

Using Redis

Redis provides a command-line interface (CLI) that can be used to interact with the Redis server. You can start the Redis CLI using the following command:

redis-cli

Basic Redis Commands

You can set a key-value pair in Redis using the SET command. For example, to set the key name to the value John, you can use the following command:

SET name John

It will return OK if the operation is successful.

Now, to get the value of the key name, you can use the GET command:

GET name

It will return John.

127.0.0.1:6379> SET name John
OK
127.0.0.1:6379> GET name
"John"

You can also delete a key using the DEL command:

DEL name

It will return 1 if the operation is successful. Now, if you try to get the value of the key name, it will return (nil).

127.0.0.1:6379> SET name John
OK
127.0.0.1:6379> GET name
"John"
127.0.0.1:6379> DEL name
(integer) 1
127.0.0.1:6379> GET name
(nil)
127.0.0.1:6379>

To get a list of all the keys in the database, you can use the KEYS command:

KEYS *

Here is what it looks like:

127.0.0.1:6379> SET fname John
OK
127.0.0.1:6379> SET lname Doe
OK
127.0.0.1:6379> SET age 30
OK
127.0.0.1:6379> KEYS *
1) "lname"
2) "age"
3) "fname"

To get the number of keys in the database, you can use the DBSIZE command:

DBSIZE

It will return 3 in this case.

Setting Expiration Time

The expiration time of a key refers to the amount of time that the key will remain in the database before it is automatically deleted. We can set the expiration time of a key using the EXPIRE command. For example, to set the expiration time of the key name to 10 seconds, you can use the following command:

EXPIRE name 10

It will return 1 if the operation is successful. Now, if you try to get the value of the key name, it will return (nil) after 10 seconds.

To find out the remaining time before the key expires, you can use the TTL command:

TTL name

It will return -2 if the key does not exist, -1 if the key does not have an expiration time or the number of seconds remaining before the key expires.

127.0.0.1:6379> SET id 1001
OK
127.0.0.1:6379> GET id
"1001"
127.0.0.1:6379> EXPIRE id 10
(integer) 1
127.0.0.1:6379> TTL id
(integer) 7
127.0.0.1:6379> TTL id
(integer) 6
127.0.0.1:6379> TTL id
(integer) 4
127.0.0.1:6379> TTL id
(integer) 3
127.0.0.1:6379> TTL id
(integer) 1
127.0.0.1:6379> TTL id
(integer) -2
127.0.0.1:6379> GET id
(nil)

Handling Lists

Redis provides a data structure called a list that can be used to store a collection of strings. You can create a list using the RPUSH command. For example, to create a list named colors and add the values red, green, and blue to it, you can use the following command:

RPUSH colors red green blue

Here RPUSH refers to the right push operation. It will return 3 (the number of elements pushed) if the operation is successful.

Note: Like RPUSH there are other commands like LPUSH (left push), LPOP (left pop), and RPOP (right pop) that can be used to push and pop elements from the list.

To get all the elements in the list, you cannot use the GET. This is because you are dealing with lists, so you have to use the LRANGE command:

LRANGE colors 0 -1

Here 0 refers to the starting index and -1 refers to the ending index. It will return all the elements in the list.

127.0.0.1:6379> RPUSH colors red green blue
(integer) 3
127.0.0.1:6379> GET colors
(error) WRONGTYPE Operation against a key holding the wrong kind of value
127.0.0.1:6379> LRANGE colors 0 -1
1) "red"
2) "green"
3) "blue"
127.0.0.1:6379>

Handling Sets

Sets are similar to lists, but they do not allow duplicate elements. You can create a set using the SADD command. For example, to create a set named employee_ids and add the values 1001, 1002, and 1003 to it, you can use the following command:

SADD employee_ids 1001 1002 1003

To get all the elements in the set, you can use the SMEMBERS command:

SMEMBERS employee_ids

If you try to add an element that already exists in the set, it will be ignored. For example, it will be ignored if you try to add the value 1001 to the set employee_ids.

127.0.0.1:6379> SADD employee_ids 1001 1002 1003
(integer) 3
127.0.0.1:6379> SMEMBERS employee_ids
1) "1001"
2) "1002"
3) "1003"
127.0.0.1:6379> SADD employee_ids 1001
(integer) 0 
127.0.0.1:6379> SMEMBERS employee_ids
1) "1001"
2) "1002"
3) "1003"

In the case of sets, to check if an element exists in the set, you can use the SISMEMBER command:

SISMEMBER employee_ids 1001

127.0.0.1:6379> SISMEMBER employee_ids 1002
(integer) 1
127.0.0.1:6379> SISMEMBER employee_ids 1005
(integer) 0

To remove an element from the set, you can use the SREM command:

SREM employee_ids 1001

127.0.0.1:6379> SMEMBERS employee_ids
1) "1001"
2) "1002"
3) "1003"
127.0.0.1:6379> SREM employee_ids 1001
(integer) 1
127.0.0.1:6379> SMEMBERS employee_ids
1) "1002"
2) "1003"

Handling Hashes

Hashes are also key-value pairs. So, using hashes in Redis is like using key-value pairs inside key-value pairs.

To create a hash, you can use the HSET command. For example, to create a hash named employee and add the values id, fname, and lname to it, you can use the following command:

HSET employee id 1001 fname John lname Doe

And to get the value of a field in the hash, you can use the HGET command:

HGET employee id

127.0.0.1:6379> HSET employee id 1001 fname John lname Doe
(integer) 3
127.0.0.1:6379> HGET employee id
"1001"
127.0.0.1:6379> HGET employee fname
"John"
127.0.0.1:6379> HGET employee lname
"Doe"

To get all the fields in the hash, you can use the HKEYS command:

HKEYS employee

To get all the values in the hash, you can use the HVALS command:

HVALS employee

To get everything in the hash, you can use the HGETALL command:

HGETALL employee

127.0.0.1:6379> HGETALL employee
1) "id"
2) "1001"
3) "fname"
4) "John"
5) "lname"
6) "Doe"
127.0.0.1:6379> HKEYS employee
1) "id"
2) "fname"
3) "lname"
127.0.0.1:6379> HVALS employee
1) "1001"
2) "John"
3) "Doe"

To delete a field from the hash, you can use the HDEL command:

HDEL employee id

To check if a field exists in the hash, you can use the HEXISTS command:

HEXISTS employee id

127.0.0.1:6379> HEXISTS employee id
(integer) 1
127.0.0.1:6379> HDEL employee id
(integer) 1
127.0.0.1:6379> HEXISTS employee id
(integer) 0

You can see initially the field id exists in the hash employee so the output was 1. After deleting it, it no longer exists and hence the value is 0.

Another very useful command is flushall. It will delete all the keys in the database. So, if you want to delete all the keys in the database, you can use the following command:

127.0.0.1:6379> KEYS *
1) "name"
2) "age"
3) "employee"
127.0.0.1:6379> flushall
OK
127.0.0.1:6379> KEYS *
(empty array)

Conclusion

• In conclusion, Redis is a popular and robust data management tool that is widely used by organizations for a variety of purposes.

• Its key features and benefits include high performance, flexibility, and support for a wide range of data structures.

• Once you have installed it, you can use a variety of basic Redis commands, such as SET, GET, DEL, and KEYS, to manage and manipulate data in Redis. By understanding and using these commands effectively, you can unlock the full potential of Redis to support your specific use cases.

I would encourage you to experiment with Redis and explore its capabilities to gain a deeper understanding of how it can help your organization.

If you like this article, please share it with your friends and colleagues who are interested in Redis. If you have any questions or comments, please feel free to leave them below. I will be happy to answer them.

My Website: arnabsen.dev

My Twitter: @ArnabSen1729