TF-IDF Vectorizer
Implement TF-IDF vectorization for a corpus of documents.
TF (term frequency): tf(t, d) = count of t in d / total words in d IDF (inverse document frequency): idf(t) = log(N / df(t)) where N = total documents, df(t) = documents containing term t.
TF-IDF = tf × idf
Input:
- Line 1: N (number of documents)
- Next N lines: documents (space-separated words, lowercase)
- Last line: query word
Output: TF-IDF score of the query word in each document, one per line, rounded to 4 decimal places.
Example:
3 the cat sat the dog sat the cat and dog cat
0.1352 0.0000 0.1352
- We calculate the term frequency (tf) of the query word "cat" in each document:
- Document 1: tf(cat,d1)=31,
- Document 2: tf(cat,d2)=0,
- Document 3: tf(cat,d3)=51
- We calculate the inverse document frequency (idf) of the query word "cat":
- N=3,
- df(cat)=2,
- idf(cat)=log(23)
- We calculate the TF-IDF score for the query word "cat" in each document by multiplying tf and idf:
- Document 1: TF−IDF=31⋅log(23),
- Document 2: TF−IDF=0⋅log(23)=0,
- Document 3: TF−IDF=51⋅log(23)
- The final output is the TF-IDF score of the query word in each document, rounded to 4 decimal places, resulting in the given sample output values.
Constraints:
- Use natural log (np.log)
- If word not in document, TF-IDF = 0
- If word not in any document, IDF = 0
- Round to 4 decimal places
More from LLM 3: Applications & Evaluation
Background Knowledge
The TF-IDF (Term Frequency-Inverse Document Frequency) technique is a widely used method in natural language processing and information retrieval for transforming text into a matrix of TF-IDF features. The goal is to normalize the importance of words in a document based on their frequency in the document and their rarity across the entire corpus. Term Frequency (TF) measures how often a word appears in a document, while Inverse Document Frequency (IDF) measures how rare a word is across all documents. The product of these two values gives the TF-IDF score, which can be used to weigh the importance of words in a document.
The Term Frequency (TF) is calculated as the number of times a word appears in a document divided by the total number of words in the document. This gives a measure of how important the word is in the document. The Inverse Document Frequency (IDF), on the other hand, is calculated as the logarithm of the total number of documents divided by the number of documents containing the word. This gives a measure of how rare the word is across all documents. By multiplying TF and IDF, we get a score that reflects both the importance of the word in the document and its rarity across the corpus.
Understanding the concept of vectorization is also crucial. Vectorization involves converting text data into numerical vectors that can be processed by machine learning algorithms. In the context of TF-IDF, vectorization involves creating a matrix where each row represents a document, and each column represents a word in the corpus. The cell at row i and column j contains the TF-IDF score of word j in document i. This allows for efficient comparison and analysis of documents based on their content.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.