Python Tokens

Each logical line in Python is broken down into a series of python tokens, which are basic lexical components. Python converts characters into tokens, each of which corresponds to one of Python’s lexical categories. It is critical to learn and understand its technical jargon, namely Python tokens. Let’s dive in deeper to know about python tokens – keyword, identifier, literal, operator, punctuator in detail. 

What are Tokens in Python?

Tokens or lexical units are the smallest fractions in the python programme. A token is a set of one or more characters having a meaning together. There are 5 types of tokens in python which are listed below:

1. Keywords                
2. Identifiers                 
3. Literals                        
4. Operators                  
5. Punctuators

1. Keywords

A keyword is a reserved word in a computer language that has a specific meaning. Python keywords form the vocabulary of the python language. Keywords aren’t allowed to be used as identifiers. They are used to define the Python language’s “Syntax” or “Structure.”

There are as in all 33 keywords used in Python programming language version 3.7. Here are a few of them:

• and, not, or: logical Operators
• as: To create an alias
• assert: For debugging
• break: To break out of a loop
• if: To create a conditional statement
• while: To create a while loop

2. Identifiers

Just as identity refers to a characteristic that distinguishes a person, the same principle is a python identifier, a token in python. In Python, an identifier is a name given to a Class, Function, or Variable. It aids in distinguishing one entity from others.

Characteristics of Python Identifier

• The initial letter of the identifier should be any letter or underscore (_).
• Upper and lower case letters have distinct characteristics.
• Except for the initial letter, any digit from 0 to 9 can be part of the identification.
• It shouldn’t be used as a keyword 
• Except for the underscore (_), an identifier cannot contain any special characters.
• Identifiers can be as long as you want them to be.
• Case matters when it comes to identifier names. Myself and myself, for example, are not the same thing.

3. Operators

Operators are tokens that, when applied to variables and other objects in an expression, cause a computation or action to occur. Operands are the variables and objects to which the computation is applied. There are 7 different operators.

i)Arithmetic Operators

It performs all the mathematical calculations. Here are a few of them:

• ( + ) Operands on either right and left sides of the operator are added.
• ( – ) Subtract the right-hand operand from the left-hand operand with the subtraction operator.
• (✖️) operator – Multiplies both sides of the operator’s operands.
• (➗) the left-hand operand by the right-hand operand with the division operator.
• (%) a percentage divides the left-hand operand by the right-hand operand and returns the remainder with the modulus operator.

ii) Relational Operators

A relational operator is a type of operator that examines the relationship between two operands. Some of the relational operators are:

• (== ) Check if two operands’ values are equal.
• (!= )Check if two operands’ values are not equal.
• (>) Check if two operands’ values are not identical (same as the!= operator).

iii) Assignment Operators

The assignment operators are employed to allocate a value to a variable. A few examples are:

• (+=)It adds the right side input to the left side input and then assigns the result to the left side input.
• (-= )Augmented assignment operator- It takes the right side operand and subtracts it from the left side operand, then assigns the result to the left side operand.

iv) Logical Operators

The logical operators compare two boolean expressions and yield a boolean result. Like

• The logical AND operator makes a condition true if both operands are true or non-zero.
• The logical OR operator returns true if one of the two operands is true or non-zero.

v) Bitwise Operators

The bitwise operator manipulates individual bits in one or more bit patterns or binary numbers. For example, If a binary XOR operator (^)  is set in one input value but not both, it copies the matching binary 1 to the result.

vi) Membership Operators

The membership operator checks for membership in successions, such as a string, list, or tuple. Like in a membership operator that fetches a variable and if the variable is found in the supplied sequence, evaluate to true; otherwise, evaluate to false.

vii) Identity Operators

When comparing the memory locations of two objects, identity operators are used. If two variables point to separate objects, it does not return true; otherwise, it returns false.

4. Literals

Literals,  tokens in Python, are data elements with a fixed value. Literals return a value for an object of the specified type. Python supports a variety of literals:

• String Literals
• Numeric Literals. These are further of three types, integer, float, and complex literals.
• Boolean Literals
• Literal Collection

Lists, tuples, dictionaries, and sets are all examples of literal collections in Python.

• A list is a collection of elements enclosed in square brackets and separated by commas. These variables can be of any data type, and their values can be altered.
• Tuple: A comma-separated list of elements or values in round brackets is also known as a tuple. Values can be of any data type, but they cannot be modified.
• Dictionary: It’s an unsorted collection of key-value pairs.
• The “set” is an unordered collection of objects enclosed in curly braces.

5. Punctuators

Punctuators are tokens in python employed to put the grammar and structure of syntax into practice. Punctuators are symbols that are used to structure programming sentences in a computer language. Some commonly used punctuators are: ‘, ‘ ,#, \ ,( ) ,{ },[ ] ,@ ,: , =

Conclusion

A tokenizer python language’s lexical structure is the set of fundamental principles that control how you build programmes in that language. Tokens in python define the language’s lowest-level structure, such as how variable names should be written and which characters should be used to represent comments.