Would you voluntarily spend another five hours on this kind of problem? Why?
AI Engineering · 90–120 min
Build an evaluated answer finder—not an AI demo
Experience retrieval, evaluation and refusal boundaries before adding a generative model.
Start the experiment01 · Setup
What you need
- Basic Python or JavaScript with a local test runner already set up. Use five short documents you wrote or may legally use. A paid model or API key is not required.
You may stop when the timebox ends. An incomplete result is still useful evidence.
02 · Task
The work sequence
- 01
Write eight questions: five answerable from the documents, two deliberately unanswerable and one ambiguous. Record the expected source for each answerable question.
- 02
Build a deterministic baseline that ranks documents by normalized keyword overlap and returns the top source plus its score. Keep the code small and inspectable.
- 03
Run all eight questions. Measure source recall and choose a score threshold below which the system says it lacks evidence instead of inventing an answer.
- 04
Improve tokenization or add a small synonym map using only the failures you observed. Rerun every case and keep before/after results.
- 05
If you have approved model access, optionally generate an answer only from the retrieved text and require a source. Mark this extension separately; never upload private documents. Write cost, latency and hallucination risks.
03 · Pay attention
Watch your reaction, not only the output
- Did measuring failures change what you wanted to build?
- Can your system refuse when evidence is missing, and can you prove that with a test?
Plot twist
Difficulty does not mean the career is a bad fit. The useful question is whether curiosity returned after the difficulty.
04 · Reflect
Three honest questions
Which part gave you energy, and which part drained it?
What evidence—not a salary video—would you gather before choosing this path?
This site does not save your answers. Put them in your own notebook if useful.
05 · Where to go next