# Prepare reviewed examples for a smaller-model experiment

Build a local JSONL dataset and holdout set without assuming a MandAPI training service.

## Start with the dataset

Distillation uses a more capable model's outputs to help train another model. This recipe covers a practical earlier step: preparing reviewed examples and a separate evaluation set. It does not train, fine-tune or upload a model through MandAPI.

Start with a task narrow enough to judge: answering your own business FAQ, for example. Use facts you own or can use. Before adding generated answers, check the relevant model and provider terms for your intended training use; an available inference model does not itself establish training permission.

## Create a small reviewed corpus

Save this as prepare_examples.py and run it in a new working directory. Python's standard library is sufficient. The fictional FAQ and answers are human-authored demonstration data, so the script makes no teacher-model requests.

```python
import json
from pathlib import Path

# Fictional, human-authored FAQ: no model requests or customer records.
examples = [
    {"id": "hours", "question": "What are your hours?", "answer": "Monday to Friday, 09:00-18:00 Mexico City time.", "split": "train"},
    {"id": "delivery", "question": "How long is delivery?", "answer": "Demo delivery takes 2-4 business days after dispatch.", "split": "train"},
    {"id": "refund", "question": "Can you refund my order?", "answer": "Please contact a support person to review the request.", "split": "train"},
    {"id": "tracking", "question": "Where is order 123?", "answer": "A support person must check that order; I cannot look it up.", "split": "holdout"},
]
seen = set()
rows = {"train": [], "holdout": []}
for example in examples:
    question = " ".join(example["question"].split())
    normalized = question.casefold()
    if not question or not example["answer"].strip() or normalized in seen:
        raise ValueError("Empty or duplicate example")
    seen.add(normalized)
    split = example["split"]
    if split not in rows:
        raise ValueError("Unknown split")
    rows[split].append({
        "example_id": example["id"],
        "source": "human-authored-demo-faq-v1",
        "reviewed": True,
        "messages": [
            {"role": "user", "content": question},
            {"role": "assistant", "content": example["answer"]},
        ],
    })

output = Path("distillation-demo")
output.mkdir(exist_ok=False)
for split, records in rows.items():
    with (output / (split + ".jsonl")).open("x", encoding="utf-8") as handle:
        for row in records:
            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
print(json.dumps({name: len(records) for name, records in rows.items()}))
print("Teacher requests: 0. Training jobs: 0.")
```

```powershell
python prepare_examples.py
```

The output directory must not already exist. Each row retains an example ID, a source version and a review flag alongside the conversation. These are local review records, not a promise that another training provider accepts this exact schema.

The script rejects empty and duplicate normalized questions. Different phrasings of the same question still need a human check; a string comparison cannot detect semantic duplicates.

## Add teacher candidates carefully

If you later generate candidate answers with MandAPI, use ordinary Chat Completions at https://api.mandapi.com/v1 with MANDAPI_API_KEY and a real MANDAPI_MODEL_ID. Record the model and prompt version in your own manifest. Keep generated candidates out of the approved corpus until you have checked factual correctness, permissions and private information.

A separate training tool or provider owns the actual training job and its accepted file format. This guide does not assume a MandAPI fine-tuning endpoint or a gateway flag that certifies training rights.

## Evaluate and check the result

The script should produce three training rows and one holdout row, with zero teacher requests and zero training jobs. Parse both JSONL files and confirm all conversation rows contain a question and reviewed answer.

Expand the holdout with unseen, independently written questions. Keep related questions together when splitting a larger dataset. Compare any smaller model with the original on the same factual-answer and human-handoff rubric. Report measured outcomes and mistakes; preparing a dataset alone does not demonstrate a quality or cost improvement.

## References

- [https://openrouter.ai/docs/cookbook/evaluate-and-optimize/distillation](https://openrouter.ai/docs/cookbook/evaluate-and-optimize/distillation)
- [https://docs.python.org/3/library/json.html](https://docs.python.org/3/library/json.html)
- [https://mandapi.com/en/docs](https://mandapi.com/en/docs)