Original release notes
kv-cells: look up the n-gram history in the sequence position index (#28040)
get_prev_tokens() rebuilt a (seq, pos) -> token hash map on every ubatch by walking all used cells, while llama_kv_cells already keeps an ordered index of the positions of each sequence in seq_pos, updated on every cell mutation to serve seq_pos_min() and seq_pos_max().
The index now stores (pos, cell) pairs in a std::set instead of a position -> count map, so a repeated position (cache reuse via rm + add, vision inputs with shared positions) yields distinct entries and the removal of a cell erases its own pair. The new seq_pos_tok_le() returns the token of the cell at the largest position 72.7 t/s (+4.9%), pp unchanged at ~2720 t/s, greedy output identical, needle retrieved.
Website: -
Attestations: -
UI:
