persist llama.cpp KV cache across idle auto-unload (slot save/restore) - #7204
Conversation
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Code Review
This pull request implements KV cache persistence across idle unloads for the llama-server backend. It introduces functionality to save slot KV states to disk before an idle unload and restore them upon model reload, significantly speeding up resumed chats. The feature includes disk space estimation, size limits, and automatic cleanup of stale cache files. Corresponding backend settings, API endpoints, and frontend UI toggles have been added, along with comprehensive unit tests. The review feedback suggests replacing a symlink in the tests with directory traversal to ensure cross-platform compatibility on Windows environments.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c3e3f0726
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fc3c0aed6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| not self.is_loaded | ||
| or not self._slot_save_dir | ||
| or not self._gguf_path | ||
| or self._prompt_cache_disabled | ||
| ): |
There was a problem hiding this comment.
When the loaded GGUF is vision/multimodal via --mmproj, llama.cpp currently rejects /slots/{id}?action=save with 501 not_supported_error (see ggml-org/llama.cpp#19466). Because keep-KV now defaults on, every idle unload of those models still enters this save path, creates no manifest, and silently drops the chat context even though the setting/UI says it will be kept; guard this path for multimodal loads or surface that KV persistence is unsupported for them.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Current llama-server gates slot save on slot content, not model capability ("a multimodal model may hold a pure-text slot" — server-context.cpp). Text-only slots on mmproj models save fine; media-holding slots return a per-slot error that the save loop already tolerates and cleans up after. Disabling saves for all multimodal loads would regress the supported case.
| # Slot-state dir: Studio owns it for KV persistence across idle unload. | ||
| frozenset({"--slot-save-path"}), |
There was a problem hiding this comment.
Because user extra args are appended after Studio's flags, a saved override containing --no-slots still passes validation and disables the /slots endpoint that this new keep-KV path relies on; llama.cpp documents slot persistence as POST /slots/{id_slot}?action=save|restore and exposes that endpoint behind the --slots/--no-slots switch (https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md). In that configuration idle unload still runs, but every save silently returns no manifest, so include the endpoint flag in the managed/denylisted group or force --slots when enabling --slot-save-path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
POST /slots/{id_slot}?action=save|restore is gated only on --slot-save-path (checked in both current master and older releases); --slots/--no-slots gates only the GET /slots monitoring endpoint, which this feature never calls. --slot-save-path is already Studio-managed and denylisted.
There was a problem hiding this comment.
💡 Codex Review
unsloth/studio/backend/core/inference/llama_keepwarm.py
Lines 368 to 370 in 2f04636
Avoid dropping the resume manifest during reload
This stale-stash cleanup runs before the idle loop acquires the lifecycle gate, so it can execute while an auto-switch reload is still holding that gate: load_model() has already marked the backend healthy, but _load_model_impl has not yet called note_model_loaded(... ) to restore the saved slots. In that race the idle loop sees a loaded model and deletes _kv_resume/its files here, causing the first request after an idle unload to lose the chat context that was just saved.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b02e03d2d9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 374fae2181
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d763b504da
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7cffb7e119
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0956329df1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
here is some feedback:
|
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 841e0a2d43
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Resolve settings/i18n/test conflicts by keeping both the new keep-KV toggle and main's 60-second idle-unload floor (clamp on read, reject 1-59s on write).
for more information, see https://pre-commit.ci
|
Pushed a few changes to move this forward. Merge and conflict resolution
Hardening in the slot save/restore path (studio/backend/core/inference/llama_cpp.py)
Added regression tests for each; the full backend suite is green (test_llama_cpp_slot_resume, test_openai_auto_switch, test_llama_server_args, test_llama_cpp_mtp_detection). Went through the outstanding review comments: the fingerprint, cleanup, purge-on-disable, re-read-before-unload, and prompt-cache items are all handled in the current code. Left the multimodal-gating and --slots suggestions as not applicable per your notes (per-slot save tolerates media-slot errors and cleans them up, and save/restore is gated on --slot-save-path rather than the monitoring endpoint). |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9392feb710
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| estimate = self._estimate_kv_cache_bytes( | ||
| self._effective_context_length or self._context_length or 0, | ||
| self._cache_type_kv, | ||
| n_parallel = self.effective_parallel_slots, | ||
| ) | ||
| # Skip before writing anything when the estimate alone blows the cap, | ||
| # rather than fully writing a slot and discarding it afterwards. | ||
| if estimate > _SLOT_SAVE_MAX_BYTES: | ||
| logger.debug( | ||
| "Skipping slot save: estimated %d bytes exceeds cap %d", | ||
| estimate, | ||
| _SLOT_SAVE_MAX_BYTES, | ||
| ) | ||
| return None | ||
| if shutil.disk_usage(save_dir).free < estimate + (1 << 30): | ||
| logger.debug("Skipping slot save: insufficient free disk") |
There was a problem hiding this comment.
When _estimate_kv_cache_bytes() cannot derive the model metadata it returns 0, but this code treats that as a zero-byte cache: it accepts just 1 GiB of free space and starts the save. For a loadable GGUF with incomplete KV metadata, a single slot file can actually be many GiB, so the save can exhaust the disk before the later post-write cap check deletes it. Treat an unavailable/zero estimate as unsafe (or require enough space for the configured cap) before issuing the save request.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7e1403b: a 0 estimate (insufficient metadata) is now treated as worst-case, requiring free space for the full cap plus 1 GiB headroom before issuing any slot save, rather than accepting a 1 GiB floor.
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…e) (unslothai#7204) * Studio: persist llama.cpp KV cache across idle auto-unload (slot save/restore) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address KV persistence review feedback * Studio: guard KV restore on launch config * Studio: fix KV resume purge race, fingerprint requested ctx, purge on disable * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: re-check idle/keep-KV settings after slot save, ns file identity * Studio: shard-aware KV guard, honor user --no-cache-prompt, early save cap * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: honor LLAMA_ARG_CACHE_PROMPT env in slot-save guard * Studio: derive prompt-cache state from final argv for slot saves * Studio: stat LoRA/control-vector sidecars in KV restore fingerprint * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: parse csv and FNAME:SCALE sidecar syntax in KV fingerprint * Studio: address codex review on idle-unload KV resume * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: harden slot-save cleanup, cap accounting, stale-KV guard, save timeout * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: treat unavailable KV estimate as full-cap for slot-save disk check --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Idle auto-unload currently frees the model but loses all KV state, so a resumed chat re-prefills its entire history after the reload. This saves per-slot KV state to disk before the idle unload and restores it when the same model loads
again.