Chain-of-Thought Extractor
Extract the final answer from a chain-of-thought response.
Given an LLM response that contains reasoning steps followed by a final answer, extract just the answer. The final answer is always on the last line that starts with "Answer:" or "Therefore:" or "The answer is".
Input: Multi-line LLM response (read until EOF)
Output: The extracted answer (text after the marker, stripped).
If no answer marker is found, output "NO_ANSWER".
Example:
Let me think step by step. First, 2 + 3 = 5. Then, 5 * 2 = 10. Answer: 10
10
- The input is read line by line until the end of the file (EOF) is reached.
- Each line is checked for the presence of answer markers ("Answer:", "Therefore:", or "The answer is") at the start of the line.
- The last line that starts with an answer marker is identified as "Answer: 10", so the text after the marker is extracted and stripped of any leading/trailing whitespace.
- The extracted text, 10, is then output as the final answer.
Constraints:
- Answer markers (case-insensitive): "Answer:", "Therefore:", "The answer is"
- Use the LAST matching line if multiple exist
- Strip whitespace from the extracted answer
- Output NO_ANSWER if no marker found
More from LLM 3: Applications & Evaluation
Background Knowledge
The problem of extracting the final answer from a chain-of-thought response involves natural language processing (NLP) and text parsing. Chain-of-thought responses are generated by large language models (LLMs) and typically include a series of reasoning steps followed by a final answer. To extract the answer, we need to identify the specific line that contains the answer, which is marked by keywords like "Answer:", "Therefore:", or "The answer is". This requires an understanding of string matching and text manipulation techniques.
In the context of NLP, tokenization is the process of breaking down text into individual words or tokens. This can be useful for analyzing the structure of the input text and identifying the answer marker. Additionally, regular expressions can be used to match patterns in the text, such as the answer markers. Understanding these concepts will help you develop an effective approach to solving the problem.
The problem also involves input/output operations, as we need to read the multi-line LLM response from the input and output the extracted answer. This requires an understanding of how to handle EOF (end-of-file) and how to process the input text line by line.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.