Dataset: Distillation Pipeline from Gemini
==========================================

Supports main.txt section 3.
This IS the distillation - the dataset is how we transfer Gemini's
tool-calling knowledge into the student model.


## 1. Problem

No large-scale public dataset exists for on-device assistant tool
calling that is:
  (a) Focused on device tools (timers, smart home, messages - not
      enterprise APIs)
  (b) Diverse in query style (direct, indirect, disfluent, garbled)
  (c) Diverse in tool count (1-10 per example)
  (d) Diverse in call patterns (single, multi, none)
  (e) High quality with validated JSON

Existing datasets (xlam-60k, ToolBench, API-Bank) are too small,
focused on web APIs, or lack query diversity.


## 2. Distillation via Gemini

We distill Gemini's tool-calling behavior by generating (query, tools,
answer) triples. Gemini acts as the teacher: given tools and a scenario,
it produces the query a user would ask and the correct tool call(s).
The student model learns to reproduce these outputs.

The quality of the student is upper-bounded by the teacher. If Gemini
gets a tool call wrong, the student learns that error. This is an
explicit trade: we get millions of examples cheaply at the cost of
being limited by Gemini's accuracy on tool calling (which is high but
not perfect).


## 3. Diversity Control

The key contribution is not "we used Gemini to generate data" - it's
how we control diversity along multiple axes to produce a robust
training distribution.

### 3.1 Tool Pools

33 hand-crafted tool pools covering on-device domains (~350 tools).
Tools are hand-written to ensure schema quality - Gemini generates
QUERIES and ANSWERS, not the tool definitions themselves.

### 3.2 Tool Synthesis

~10% of examples use SYNTHESIZED tool definitions - Gemini invents
novel tools in random domains. Prevents memorizing the fixed pools,
forces generalization to unseen schemas at inference.

### 3.3 Tool Count Distribution

Weighted sampling: front-loaded toward 3-5 tools (common real-world
case), 1-2 and 6-10 covered at lower frequency. Adjustable post-hoc
via `needle rebalance-tools`.

### 3.4 Call Types

Sampled per-example from a weighted distribution:

  single         (19%): exactly 1 tool call
  multi          (19%): 2-3 calls to different tools
  multi_few_tools(25%): 2-4 calls reusing 1-3 tools, different args
  multi_long     (12%): 2-3 calls with verbose argument values
  none           ( 6%): no matching tools -> answer = []
  near_miss      ( 6%): related but unsatisfiable -> answer = []
  indirect       (11%): implicit intent ("it's freezing" -> set_thermostat)
  disfluent      (10%): speech disfluencies (um, uh, false starts)
  garbled        ( 9%): typos and ASR errors
  no_tools       ( 2%): no tools provided -> answer = []

This is the key dataset design decision. The model sees:
  - Clean commands (common case)
  - Noisy/disfluent/garbled queries (realistic speech input)
  - Implicit requests (natural human behavior)
  - Negative examples (teaches when NOT to call)
  - Multi-call patterns (batch operations)

### 3.5 Scenario Seeds

~200 hand-crafted scenarios biasing generation toward specific patterns.
Hints, not templates - same scenario produces diverse queries across
Gemini calls.

### 3.6 Validation and Rephrasing

Every example validated: JSON parse, schema, tool name, argument key,
deduplication. Tool descriptions randomly rephrased per-example to
prevent memorizing description strings.


## 4. External Data

Salesforce/xlam-60k merged for API-style tools and cross-domain
diversity. Deduplicated, uploaded to HuggingFace (Cactus-Compute/
tool-calls).


## 5. Tokenization

  Encoder (max 1024): [query_tokens... <tools> tool1... tool2...]
  Decoder (max 512):  [EOS <tool_call> answer_tokens...]

Tools shuffled to prevent positional bias. Token-level loss mask
labels each decoder token as base/name/value/key for structured
weighting during training (see architecture.txt section 7).

Sequence packing with segment IDs for block-diagonal attention.
Contrastive data extracted separately: query-only tokens, individual
tool tokens, positive/negative labels.


## 6. Ablations (maps to main.txt 4.4)

  A. Data Scale Curve [MUST-HAVE]
     10%, 25%, 50%, 100% of dataset. Shows data efficiency.

  B. Call Type Distribution [MUST-HAVE]
     Single-only vs clean-only vs full distribution. Proves query
     diversity is necessary for robust inference.

  C. Negative Example Importance [MUST-HAVE]
     With vs without negatives. Proves the model needs to learn when
     NOT to produce a call.

  D. Synthesized vs Curated-Only Tools [STRONG]
     Eval on seen and novel tools. Proves synthesis prevents
     overfitting to the training tool catalog.
