Case-to-Skill Agent
A case-based LLM workflow that converts solved engineering cases into reusable, evaluated skills.
Designed for knowledge distributed across documents, reports, and technical files.
Industrial AI / Agent Systems / Case-Based Reasoning
Instead of returning isolated passages, the system retrieves validated procedures that can be adapted, executed, evaluated, and safely retained.
- Input the engineering caseLoad the task, files, and evaluation target.
- Retrieve and run a validated skillAdapt a reusable procedure instead of returning isolated passages.
- Evaluate, retain, or recoverKeep successful work and route failed attempts to retry or fallback behavior.
Overview
Engineering knowledge is often distributed across documents, reports, and technical files. Even when a similar problem has already been solved, the underlying reasoning can be difficult to locate and reuse.
This project explored a case-based alternative to conventional document retrieval. Instead of retrieving passages alone, the workflow stores solved cases together with reusable skills: concise procedures describing how similar tasks can be approached and evaluated.
The problem
Standard retrieval can surface relevant information, but it does not necessarily preserve the reasoning process required to solve a technical task.
How can reusable procedures be represented from previously solved cases?
How can a relevant, validated skill be selected for a new task?
How can generated results be evaluated before they are retained or reused?
My contribution
I co-engineered selected parts of the workflow, with a focus on retrieval and context design, structured case representation, objective evaluation, fallback handling, caching, and cost-aware orchestration. My work focused on improving the reliability of the system around the LLM rather than relying on prompt changes alone.
My scope
Contributed to selected workflow components and the evaluation and reliability logic around the LLM.
Co-engineered
Retrieval and context design, structured outputs, evaluation loops, fallback paths, caching, and LLM usage controls.
Broader project
A case-based agent workflow for converting solved technical cases into reusable skills that can be retrieved and adapted to related tasks.
System workflow
- 01Prepare cases
Structure prior tasks, associated files, expected results, and reusable procedures.
- 02Retrieve a validated skill
Find a relevant solved case using task and artifact information.
- 03Adapt the procedure
Use the retrieved skill as guidance for solving the new task.
- 04Execute the workflow
Process the available documents or technical files and produce a structured result.
- 05Evaluate the result
Compare the output with objective criteria instead of accepting it because it appears plausible.
- 06Retain or recover
Keep an improved skill when the result passes evaluation; otherwise use retry or fallback paths.
Technical decisions
Retrieve procedures, not only passages
case representation- Problem
- A relevant document may contain useful information without explaining the steps needed to solve the task.
- Decision
- Represent solved cases together with concise, reusable skills that encode the associated reasoning procedure.
- Result
- A new task can reuse an evaluated method rather than reconstructing the full approach from retrieved passages each time.
Evaluate before retaining
objective evaluation- Problem
- A generated answer or skill should not be accepted only because it is well written or appears plausible.
- Decision
- Evaluate structured outputs against objective criteria and retain a new skill only when it improves the previous result.
- Result
- The external memory can reject unsuccessful attempts instead of accumulating every generated output.
Add explicit recovery and usage controls
cost-aware orchestration- Problem
- Repeated LLM calls and unnecessarily large contexts can increase cost without guaranteeing a better result.
- Decision
- Use selective context, caching, usage tracking, and explicit fallback paths around the main workflow.
- Result
- The workflow can reuse previous work, limit unnecessary calls, and recover when an adapted skill does not produce an acceptable result.
Results
These results come from separate experiments and model configurations. They should be interpreted individually, not as the performance of one combined final model.
- Baseline
- Passage retrieval with direct generation
- Variant
- Validated skill retrieval with objective evaluation
- Evaluation
- Public numerical engineering benchmark
- Baseline
- Passage retrieval with direct generation
- Variant
- Validated skill retrieval with objective evaluation
- Evaluation
- Public numerical engineering benchmark
- Tolerance
- ±10% relative error
Results are drawn from separate public-benchmark experiments and should not be interpreted as one combined final-model score.
Lessons & takeaways
- Reliable LLM workflows depend on the control system around the model, not only on prompt quality.
- Retrieving a reusable procedure can be more useful than retrieving isolated passages.
- Generated outputs should be evaluated before they are added to long-term memory.
- Fallback and usage controls should be designed as part of the workflow rather than added only after failures occur.