Laya: where did the important words go?
Published
Laya is a local AI model that chooses from options. Its input builder limits option text before inference, so adding options can cut off the words that distinguish them.
Explanation & code
Measured with Laya 0.3.5 input-building functions and a pinned English tokenizer. No model prediction, accuracy or latency was measured. Shortening text must preserve important conditions.
Understand it. Then fix it.
Inspect the actual model input
A long billing description ends with “refunds and invoices.” With two options, those words survive. With twenty long options at head_max_len=192 and max_len=512, they disappear during input construction.
Count tokens, not words
Tokens are pieces of text. In this crowded-options branch, floor((192 - 16) / 20) gives 8 slots per option. One slot is a marker, leaving 7 text tokens. The real decoded billing input is shown below. This formula applies to this branch, not every possible input.
billing: This team handles incoming customerPreserve meaning with less filler
Replace repeated filler with short, distinct descriptions. With the same twenty options and budgets, both refunds and invoices survive. The pinned tokenizer splits the short description into seven pieces; token counts depend on the tokenizer.
criteria = {
"billing": "refunds and invoices",
"frontend": "buttons and layout",
# ...18 other short descriptions
}If detail matters, increase the budgets
For the original long descriptions, our 768/1024 input-building example preserves the missing words. Review checkpoint limits and measure real inference before choosing budgets. More room is not a guarantee of better decisions.
agent.cfg["head_max_len"] = 768
agent.cfg["max_len"] = 1024Reproduce the observation
The companion downloads pinned source and tokenizer files, verifies their hashes, and runs unchanged upstream input-building functions. It downloads no model weights. Separate prediction tests are still needed.
Code blocks are teaching excerpts. Keep the surrounding error handling and application requirements.
Save the code excerpts ↓Read the full transcript
Laya is a local AI model that picks from options. Why can adding options hide important words? Its input builder shares a text budget. More options can mean shorter descriptions. Tokens are the small text pieces it reads. In our twenty-option example, each gets eight slots: one marker and seven text tokens. The word refunds disappears before the model sees it. We tested the tokenizer, not a prediction. Can we keep the words that matter? Yes. Replace repeated filler with short, distinct descriptions. Here, billing says refunds and invoices. With the same twenty choices, both words survive. If important details will not fit, raise the input budgets and test again. More text costs memory and time. So it ignored my instructions? No. Your input builder ate the evidence.
