TF-IDF Score

easyPRONLP FundamentalsMachine Learning 2

Compute TF-IDF scores for terms across documents.

For each term in each document:

TF (Term Frequency): $\frac{\text{count of term in doc}}{\text{total terms in doc}}$
IDF (Inverse Document Frequency): $\ln\frac{N}{\text{number of docs containing term}}$
TF-IDF = TF × IDF

where $N$ is the total number of documents and $\ln$ is the natural logarithm.

Return the vocabulary (sorted) and the TF-IDF matrix, rounded to 4 decimal places.

Example

Input

documents = ["the cat", "the dog", "the bird"]

Output

(['bird', 'cat', 'dog', 'the'], [[0.0, 0.5493, 0.0, 0.0], [0.0, 0.0, 0.5493, 0.0], [0.5493, 0.0, 0.0, 0.0]])

Reasoning

Input: documents holds $N = 3$ short texts, each split into words: "the cat", "the dog", "the bird". Every document has exactly 2 words.
Vocabulary: the distinct words in alphabetical order are bird, cat, dog, the. They are the 4 columns; each of the 3 rows is one document.
TF: in "the cat", each of the and cat appears once out of 2 words, so $TF = \frac{1}{2}$ . Absent words have $TF = 0$ .
IDF: the is in all 3 documents, so $IDF = \ln\frac{3}{3} = \ln 1 = 0$ . Each animal word is in just 1 document, so $IDF = \ln\frac{3}{1} = \ln 3 \approx 1.0986$ .
TF-IDF: for cat in document 1, $\frac{1}{2} \times 1.0986 = 0.5493$ ; for the, $\frac{1}{2} \times 0 = 0$ . Rounded to 4 decimal places, row 1 is [0.0, 0.5493, 0.0, 0.0], and rows 2 and 3 put 0.5493 under dog and bird.
Notice the the column is all zeros: a word that appears in every document cannot tell documents apart, so its IDF wipes it out.

Constraints

documents: list of strings (split on spaces, lowercase)
Return tuple (vocabulary, tfidf_matrix)
tfidf_matrix: 2D list (n_docs x vocab_size)
Use natural log (ln), round to 4 decimal places

More from Machine Learning 2

Kernel PCAhard PCA via Covariance Matrixmedium Variance Explained Ratioeasy Feature Scalingeasy GAN Discriminator Losshard

Concept primerPREVIEW

Background Knowledge

The TF-IDF (Term Frequency-Inverse Document Frequency) score is a widely used technique in Natural Language Processing (NLP) to evaluate the importance of words in a document based on their frequency and rarity across a collection of documents. The Term Frequency (TF) component measures the frequency of a term in a document, while the Inverse Document Frequency (IDF) component measures the rarity of a term across all documents. The product of TF and IDF gives the TF-IDF score, which helps in filtering out common words (like "the", "and", etc.) that do not carry much meaning in the context of the document.

The TF is calculated as the ratio of the count of a term in a document to the total number of terms in the document. This gives a measure of how important the term is in the document. The IDF, on the other hand, is calculated as the natural logarithm of the ratio of the total number of documents to the number of documents containing the term. This gives a measure of how rare the term is across all documents. The TF-IDF score is then calculated as the product of TF and IDF, giving a measure of the importance of the term in the document, taking into account its rarity across all documents.

Understanding the concept of vocabulary is also crucial in this context. The vocabulary refers to the set of unique terms across all documents. In the context of TF-IDF, the vocabulary is used to create a matrix where each row represents a document, and each column represents a term in the vocabulary. The cell at row $i$ and column $j$ contains the TF-IDF score of term $j$ in document $i$ . This matrix is known as the TF-IDF matrix.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Creating a vocabulary of unique terms across all documents
Calculating the TF and IDF scores for each term in each document
Calculating the TF-IDF score for each term in each document
Creating a TF-IDF matrix where each cell contains the TF-IDF score of a term in a document

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results