Dolci-Instruct-SFT-Tool-Use-FR

# Dolci-Instruct-SFT-Tool-Use-FR A French tool-calling SFT set of **186,176 single-turn samples**, derived from [`allenai/Dolci-Instruct-SFT-Tool-Use`](https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT-Tool-Use) and targeted at the French categories of the Berkeley Function Calling Leaderboard (BFCL). Prompts and schema descriptions are French. **Function names, parameter names, enum values and argument values stay English**, mirroring how BFCL's French split is built — so the skill being taught is *French intent → English arguments*. Decontaminated against BFCL v2 French with the Olmo 3 recipe (see below). ## Schema Standard [TRL tool-calling format](https://huggingface.co/docs/trl/main/en/dataset_formats#tool-calling): `messages` + `tools`. | column | type | contents | |---|---|---| | `id` | `string` | source id from Dolci, for provenance / joins | | `messages` | `list<struct<role, content, tool_calls>>` | the conversation. `tool_calls` is a list of `{type, function:{name, arguments}}`; `arguments` is a **dict** | | `tools` | `list<Json>` | available tools as OpenAI JSON schemas | ```python { "id": "...", "messages": [ {"role": "user", "content": "Quelle est la météo à Paris ?", "tool_calls": []}, {"role": "assistant", "content": "", "tool_calls": [ {"type": "function", "function": {"name": "get_weather", "arguments": {"city": "Paris"}}}]} ], "tools": [{"type": "function", "function": {"name": "get_weather", "description": "...", "parameters": {...}}}] } ``` `arguments` and `tools` use the `datasets` `Json()` feature type. **`arguments` is a dict, not a JSON string** — this is what `apply_chat_template` requires. Passing a JSON string fails with `TypeError: Can only get item pairs from a mapping`. For refusal samples the assistant message has prose `content` and an empty `tool_calls`. There are no `tool` role messages (single-turn, no tool responses), so no `tool_call_id` field. ## Composition | category | n | shape | |---|---|---| | `multiple` | 68,515 | 1 call, several tools offered | | `parallel_multiple` | 61,880 | several calls to different tools | | **`irrelevance`** | **31,180** | no tool fits — correct answer is prose, no call | | `simple` | 12,990 | 1 call, 1 tool offered | | `parallel` | 11,611 | several calls to the same tool | Origins: 85,442 natively single-turn Dolci rows · 73,663 harvested as the first exchange of a multi-turn row · 14,323 synthesized refusals · 11,850 single-tool conversions · 898 clarifications. All single-turn: one user message, one assistant reply. Rows shuffled (seed 0). Category is not a column (kept minimal) but is recoverable from `id`. Rendered through Qwen3.5's template, assistant targets use its XML call format, **not** Qwen3's JSON-in-`<tool_call>`: ``` <tool_call> <function=get_weather> <parameter=city> Paris </parameter> </function> </tool_call> ``` ## The refusal (`irrelevance`) subset — read this before reweighting Knowing when *not* to call is scored by two BFCL categories totalling 1,124 of 3,638 entries, and it is the easiest thing to accidentally destroy by adding call data. An earlier revision of this dataset diluted refusals from 17.3% to 9.4% and lost **17.8 points** of `live_irrelevance` while gaining on every call category. Refusals are **16.7%** here, deliberately. **Shape matters as much as share.** BFCL's refusal entries mostly offer exactly *one* unrelated tool: | tools offered | this dataset | BFCL `live_irrelevance` | BFCL `irrelevance` | |---|---|---|---| | **1** | **42.1%** | 57.5% | 100% | | 2 | 19.4% | 10.3% | – | | 3 | 16.6% | 7.0% | – | | 4+ | 21.8% | 25.2% | – | The three refusal sources: | origin | n | what it is | |---|---|---| | `single` | 15,959 | Dolci rows that natively have no call, translated | | `synth` | 14,323 | **mismatched pairing**: an existing French prompt offered a semantically distant tool set, refusal generated in French | | `clarify` | 898 | **underspecified requests** whose correct reply asks for the missing argument | `synth` is built the way BFCL builds the category — a real benchmark entry offers `get_current_weather` against a query about VirusTotal DNS lookups. A prompt therefore appears twice in this dataset: once with its real tools (→ call) and once with distant tools (→ refuse). That is the intended contrast — **the decision depends on the tools, not the query**. These are distinct rows, not duplicates. Guarding against the expensive error (offering a tool that *can* serve the query, then labelling a refusal) is two-layer: a lexical filter requiring zero shared content words with both the query and its real tool, then `gemma-4-31B-it` judging each pair and answering `APPELABLE` instead of refusing, biased to answer `APPELABLE` whenever unsure. It rejected 1.0% (223 of 22,000). Control check — the same 150 queries offered their **real** tools: | tools offered | `APPELABLE` | refusal written | |---|---|---| | real | **150 / 150** | 0 | | mismatched | 1 / 150 | 149 | so the filter is doing the work rather than the judge rubber-stamping refusals. **`simple` is deliberately inflated to balance this.** BFCL's single-tool entries are ~1:1 call vs refuse (808 call / 748 refuse). Adding 13k single-tool refusals against the 1,151 single-tool *calls* Dolci provides would teach "one tool offered ⇒ refuse". So 12,000 `multiple` rows were converted by dropping their uncalled tools — the gold call is untouched, so the label stays correct. Published ratio is **1.07:1**. If you subsample this dataset, preserve that ratio. ## Training with TRL ### ⚠️ `messages` alone trains on the full sequence On **Qwen3.5**, `assistant_only_loss=True` is rejected outright: ``` ValueError: Assistant-only loss is not yet supported for vision-language models. ``` Qwen3.5 ships a `video_preprocessor_config.json`, so TRL classifies it as a VLM. TRL *does* bundle `qwen3_5_nothink_training.jinja` with the `{% generation %}` markers, but the VLM guard fires first. Verified with `trl 1.9.1`: feeding `messages` + `tools` straight to `SFTTrainer` yields `masked=0` — **loss on every token, including all the tool schemas**. To get completion-only loss, split into TRL's *conversational prompt-completion* type. Both halves stay lists of dicts, and the model's stock chat template is left untouched: ```python from datasets import load_dataset from trl import SFTTrainer, SFTConfig ds = load_dataset("kurakurai/Dolci-Instruct-SFT-Tool-Use-FR", split="train") def to_prompt_completion(ex): m = ex["messages"] return { "prompt": [{"role": m[0]["role"], "content": m[0]["content"]}], "completion": [m[1]], "tools": ex["tools"], } ds = ds.map(to_prompt_completion, remove_columns=["messages"]) trainer = SFTTrainer( model="Qwen/Qwen3.5-0.8B", args=SFTConfig(max_length=8192, loss_type="nll"), train_dataset=ds, ) trainer.train() ``` Verified with this split: the loss mask is contiguous over the assistant turn only, and the masked prefix is **byte-identical to the prompt BFCL sends at eval time** (ending `<|im_start|>assistant\n<think>\n\n</think>\n\n`, first trainable token `<tool_call>`). Do **not** pass `assistant_only_loss=True` or `chat_template_path` — either makes TRL substitute `qwen3_training.jinja`, which documents Qwen3's **JSON** tool format instead of Qwen3.5's XML format, and that patched template gets saved with your checkpoint. Guard against it: ```python base = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B") assert tok.chat_template == base.chat_template, "chat template drifted from Qwen3.5" ``` `loss_type="nll"` avoids a crash in TRL's default `"chunked_nll"` path on current transformers (`'functools.partial' object has no attribute '__func__'`). ### ⚠️ Raise `max_length` **Nothing here is truncated or length-filtered.** But `SFTConfig.max_length` defaults to **1024**, and TRL truncates then *drops examples whose completion was cut away*: | `max_length` | samples over the limit | |---|---| | 1024 (default) | **~28% of the dataset silently dropped** | | 4096 | 11,182 | | 8192 | 11 | | 16384 | 2 | Lengths (Qwen3.5 tokenizer, rendered): mean 1,056, p50 742, p95 5,558, p99 5,696, max 27,148. ≈ **197M tokens** total. `max_length=8192` is the sweet spot. ## How it was built 1. **Filter** Dolci to BFCL-v1/v2-shaped single-turn rows: exactly one user turn, first assistant message is the call, every call parses with `ast`, no function name contains a space. Plus the opening `user → assistant call` exchange of multi-turn rows, which is itself a valid single-turn sample (later turns and tool responses discarded — French BFCL is 100% single-turn). 2. **Translate** with `google/gemma-4-31B-it` (bf16, greedy, reasoning off), 8×H100 data-parallel. Units are *individual strings*, never JSON blobs, so the translator can never corrupt a schema and never sees names, enums or types. Deduplicating description strings cut translated volume from 697M chars to 2.35M. 3. **Repair the refusal subset** — mismatched pairing, clarification harvest, single-tool call conversion (see above). 4. **Decontaminate** against BFCL v2 French, 8-gram / 0.5 overlap, both languages: 121 removed. Separately, 27 no-call rows whose "refusal" was really a bare LaTeX answer (`\boxed{844-937-8679}`) were dropped: BFCL scores them correct (no call emitted), but as a training target they teach answer-in-LaTeX rather than decline. 5. **Two publication gates**, both of which caught real corruption: - *decode round-trip*: render each target through Qwen3.5's template, decode it with BFCL's own `QwenFCLocalHandler.decode_ast`, require the exact original call back. **100.0000%** over 154,996 call samples. - *storage round-trip*: the `datasets` `Json()` encoder truncates floats (`31.23788289124186` → `31.2378828912`, `8.854e-12` → `0.0`). Write parquet, read it back, compare `arguments`, drop mismatches — 2,176 dropped. **Argument values are BFCL type-safe.** 38,209 values are stored JSON-quoted because their bare form does not survive the decoder. Qwen3.5's template renders a scalar as `args_value | string`, so the *string* `"10009"` goes on the wire as bare `10009` and BFCL's decoder reads it back as an *int* — likewise `"true"` → `True`, `"1,2"` → a tuple, `"0xe9e7…"` → a hex int. Quoting is applied only to values that fail the round-trip, so everything else is untouched. ## Decontamination Uses the **Olmo 3 recipe** — Olmo 3 applies the Tulu 3 procedure to all post-training data: **8-gram matching with a 0.5 overlap threshold** on the instruction field. Implemented as: for eval instance `E` and train instance `T`, `overlap = |8grams(E) ∩ 8grams(T)| / |8grams(E)|`, dropping `T` when `overlap > 0.5` for any `E`. The denominator is the *eval* instance — "more than half of an eval question appears in this training example". Using the train instance as denominator would let a long training document hide a fully-copied eval question. Text is NFKC-normalized, lowercased, stripped of punctuation, whitespace-tokenized. Run on **both** sides, against all 13 BFCL v1+v2 French categories (3,638 entries): | side | compared | flagged | |---|---|---| | FR | translated prompt vs BFCL-fr prompt | 22 | | EN | English source prompt vs English BFCL prompt | 101 | | **union removed** | | **121 (0.120%)** | The English pass matters: **99 of the 121 were caught only on the English side.** Translation is a paraphrase, so it dissolves n-gram overlap that is obvious in the source — decontaminating the French text alone would have missed most of the leakage. This is *in addition to* the source's own decontamination: 200k of the 227,579 Dolci rows come from an explicitly `bfclv3-decontaminated` mix. ## Limitations - **Synthesized refusals are formulaic in their opening.** 99.9% of the 14,323 are distinct texts, but **91.8% begin "Je suis désolé, mais je…"**. BFCL only scores whether a call was emitted, so this does not affect the benchmark, but a model trained on it will produce a canned refusal opening. The 15,986 translated `single` refusals are stylistically diverse and offset this somewhat. Resample the openings if refusal style matters for your use case. - **Decontamination is n-gram based**, so it cannot catch paraphrase-level leakage. **13.2% of BFCL-fr prompts (482 of 3,638) are shorter than 8 tokens** and are therefore invisible to 8-gram matching. A stricter `n=5` diagnostic flagged nothing new, which is evidence the set is clean rather than the window being too wide — but not proof. - **`simple` is inflated by construction** (11,850 of 12,990 are converted `multiple` rows). This is intentional (see above) but means it is not an independent sample of single-tool tasks. - Mismatched pairing reuses prompts, so **~14k prompts appear twice** with different tool sets and opposite targets. Intended, but it makes naive prompt-level dedup delete the signal. - No English samples, so format retention on English input is untrained and untested. - Translation and refusal generation are machine-produced and unreviewed by humans. - Single-turn only: no multi-turn, no tool-response follow-ups. Nothing here trains the `tool` role, so it will not help on BFCL v3+ multi-turn categories. - Java/JavaScript type vocabulary is absent (Dolci is pure OpenAPI: string/object/integer/number/ array/boolean/null), so `simple_java` / `simple_javascript` are not directly targeted. ## License and attribution Released under **ODC-BY 1.0**, inheriting the license of the source dataset. Derived from `allenai/Dolci-Instruct-SFT-Tool-Use` (Ai2, ODC-BY), created for Olmo 3. Please credit Ai2 alongside this dataset, and follow Ai2's [Responsible Use Guidelines](https://allenai.org/responsible-use). ```bibtex @misc{olmo2025olmo3, title={Olmo 3}, author={Team Olmo}, year={2025}, eprint={2512.13961}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2512.13961} } ```