What Is Lexical Analysis?

In computer science, lexical analysis is performed similarly to how it is done in linguistics. Lexical analysis groups a sequence of letters or sounds into sets of units representing meaningful syntax. In linguistics, it is called parsing, and in computer science, it can be called parsing or tokenizing. To put this in more concrete terms, consider the following sentence "The cat sat on the mat". If you were doing lexical analysis on this sentence, you would break it down into words like this: "the", "cat", "sat", "on", and "mat". Each of these words would have meaning and could be used by themselves (or with other words) to create new sentences. In computer science, the same process happens when you break down a program into tokens, the individual pieces of information like numbers, letters and punctuation marks that make up your code. A tokenizer breaks down your code into these tokens so that when you run your program, each instruction gets executed one at a time instead of all at once! Lexical analysis is a computer science term that breaks down streams into tokens, the basic units of meaning. For example, if you're trying to read this sentence, your brain does lexical analysis as it analyzes each word. You don't have to think about it too hard. It's just something that happens automatically in your brain when you read! So, does that mean computers do the same thing? Well…no. Computers do lexical analysis differently than humans do. They use programs called lexers (or tokenizers) or scanners to break streams down into their most minor possible pieces, called lexemes or tokens. The distinction between these two terms is subtle: while both refer to the smallest unit of meaning in a language, they differ slightly in their use. In computer science, lexemes are individual characters within a text string, whereas tokens represent more complex meaning-related sub-units. Lexical analysis programs are explicitly designed around these concepts – so when we speak about them here on our website, we'll use them interchangeably!