📘

TF-IDF Score

EasyNLP Fundamentals 📚 Machine Learning 2

Compute TF-IDF scores for terms across documents.

For each term in each document:

TF (Term Frequency): $\frac{\text{count of term in doc}}{\text{total terms in doc}}$
IDF (Inverse Document Frequency): $\ln\frac{N}{\text{number of docs containing term}}$
TF-IDF = TF × IDF

where $N$ is the total number of documents and $\ln$ is the natural logarithm.

Return the vocabulary (sorted) and the TF-IDF matrix, rounded to 4 decimal places.

Example:

Input:

documents = ["the cat", "the dog", "the bird"]

Output:

(['bird', 'cat', 'dog', 'the'], [[0.0, 0.5493, 0.0, 0.0], [0.0, 0.0, 0.5493, 0.0], [0.5493, 0.0, 0.0, 0.0]])

Reasoning:

First, we calculate the Term Frequency (TF) for each term in each document: for example, in the first document "the cat", $TF_{cat} = \frac{1}{2}$ and $TF_{the} = \frac{1}{2}$ .
Then, we calculate the Inverse Document Frequency (IDF) for each term across all documents: for example, $IDF_{cat} = \ln\frac{3}{1}$ , $IDF_{the} = \ln\frac{3}{3} = 0$ .
Next, we compute the TF-IDF score for each term in each document by multiplying the TF and IDF values: for example, $TF-IDF_{cat} = TF_{cat} \times IDF_{cat} = \frac{1}{2} \times \ln{3}$ .
The final output is a sorted vocabulary and the TF-IDF matrix, rounded to 4 decimal places, where each row represents a document and each column represents a term in the vocabulary.

Constraints:

documents: list of strings (split on spaces, lowercase)
Return tuple (vocabulary, tfidf_matrix)
tfidf_matrix: 2D list (n_docs x vocab_size)
Use natural log (ln), round to 4 decimal places

More from Machine Learning 2

Kernel PCAHard PCA via Covariance MatrixMedium Variance Explained RatioEasy Feature ScalingEasy GAN Discriminator LossHard

Background Knowledge

The TF-IDF (Term Frequency-Inverse Document Frequency) score is a widely used technique in Natural Language Processing (NLP) to evaluate the importance of words in a document based on their frequency and rarity across a collection of documents. The Term Frequency (TF) component measures the frequency of a term in a document, while the Inverse Document Frequency (IDF) component measures the rarity of a term across all documents. The product of TF and IDF gives the TF-IDF score, which helps in filtering out common words (like "the", "and", etc.) that do not carry much meaning in the context of the document.

The TF is calculated as the ratio of the count of a term in a document to the total number of terms in the document. This gives a measure of how important the term is in the document. The IDF, on the other hand, is calculated as the natural logarithm of the ratio of the total number of documents to the number of documents containing the term. This gives a measure of how rare the term is across all documents. The TF-IDF score is then calculated as the product of TF and IDF, giving a measure of the importance of the term in the document, taking into account its rarity across all documents.

Understanding the concept of vocabulary is also crucial in this context. The vocabulary refers to the set of unique terms across all documents. In the context of TF-IDF, the vocabulary is used to create a matrix where each row represents a document, and each column represents a term in the vocabulary. The cell at row $i$ and column $j$ contains the TF-IDF score of term $j$ in document $i$ . This matrix is known as the TF-IDF matrix.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Creating a vocabulary of unique terms across all documents
Calculating the TF and IDF scores for each term in each document
Calculating the TF-IDF score for each term in each document
Creating a TF-IDF matrix where each cell contains the TF-IDF score of a term in a document

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.