Evaluating the reliability of function calling by large language models for ukrainian-language requests

Abstract

Function calling by large language models is increasingly used to connect natural-language requests to software functions, yet its reliability for Ukrainian-language requests remains poorly characterized. We introduce UA-ToolCall, a paired English–Ukrainian benchmark with twenty bilingual mock-tool contracts, twenty-four development cases, and one hundred and twenty fixed test cases (one hundred required calls, ten clarification cases, and ten no-tool cases). It varies query and description languages, catalogue size, and distractor similarity while keeping machine identifiers canonical. Models from the Qwen3, Granite 3.3, and SmolLM3 families were evaluated in quantized form using a shared text prompt for unconstrained one-step generation of structured function-call outputs and greedy decoding, producing three thousand six hundred outputs. Under English descriptions and the full catalogue of twenty tools, Ukrainian queries reduced strict success from ninety-four to seventy-four percent, from forty-three to nine percent, and from twenty-six to nine percent, respectively; all paired effects remained significant after Holm correction. Qwen kept one hundred percent tool-selection accuracy, so its loss came from arguments and canonicalization; Granite lost selection and arguments; SmolLM3 was dominated by decision-field errors. Ukrainian and bilingual descriptions improved SmolLM3 by eleven and seventeen percentage points, respectively, but did not produce statistically significant gains for the other models; bilingual descriptions increased mean prompt length by fifty-four to eighty-three percent. Within the single catalogue order generated with seed 1701, neither catalogue contrast was statistically significant; order variability was not estimated. These results support reporting semantic and executable correctness separately and indicate that localization benefits are model-dependent. The findings apply to prompt-based one-step generation of structured function-call outputs by the recorded Q4_K_M builds, not native function-calling interfaces, constrained decoding, or multi-step tool interaction.