[{"content":"","date":null,"permalink":"https://query.mt/blog/tags/agents/","section":"Tags","summary":"","title":"Agents"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/benchmarks/","section":"Tags","summary":"","title":"Benchmarks"},{"content":"AREX-2 is a fine-tune of Qwen3.8-27B. It claims to be better at one thing: taking a solution that works and making it better, round after round, without losing track. That claim can be measured. Give the model a task with a numeric score, a time budget and a target, keep the scorer out of its reach, and record every round.\nThis post does that with one small task: a Python TSP heuristic. AREX-2 runs it three times, its base model three times, and GLM-5.3-flash three times as a hosted reference point. Everything runs through QueryMT and an OpenSpec custom schema that keeps the scoring outside the agent\u0026rsquo;s reach.\nTL;DR # Model Best score per run (b1 / b2 / b3) Median Target ≤ 10900 How the runs ended GLM-5.3-flash (z.ai) 10849.88 / 10774.65 / 10755.83 10774.65 3 / 3 3 completed, in 3–12 min AREX-2 27B Q4_K_M 10942.32 / 10969.91 / 10951.59 10951.59 0 / 3 3 completed, 19 and 18 scored rounds in b2 and b3 Qwen3.8-27B UD-Q4_K_M 11026.22† / 11075.51† / 10926.55 11026.22† 0 / 3 2 of 3 failed: a hung shell command, killed at the 90-min cap † Best score recorded before the run hung. In both failed runs the model started its own test script without a timeout. The script blocked the session for 55 and 79 minutes until the harness killed it. Neither run reached its final round or wrote results.md. These scores are shown only so the base model\u0026rsquo;s search can be compared at all. They are not finished results.\nAREX-2 is consistent, not better. Its three best scores lie within 28 points of each other, and all three runs finished cleanly. The base model has the single best local run (10926.55), but two of its three runs failed. AREX-2 iterates, as the paper says. It ran 18 and 19 scored rounds in two runs (the base model ran 4, 4 and 13). It restored the best snapshot 24 times and stopped on its own every time. No local model reached the target. GLM-5.3-flash reached it in every run within 11 minutes, using roughly an eighth of AREX-2\u0026rsquo;s reasoning tokens. n = 3 per model, and the scorer depends on wall-clock time. Rescoring the saved best solutions on another machine moves individual runs by up to 150 points. Read the differences between the two local models as tendencies, not as rankings. The paper in one paragraph # AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks (Qian et al., arXiv:2609.38288, September 2026) defines self-improvement as \u0026ldquo;the ability to iteratively refine a solution at test time\u0026rdquo;. It splits that ability into two parts: reflection, which produces a better solution than the current one, and long-horizon execution, which keeps the iteration effective over many rounds. The authors argue that both parts are domain-agnostic. They therefore train on synthesised improvement trajectories from two domains with verifiable feedback: machine-learning engineering and algorithmic programming. The resulting agent, built on Qwen3.8-27B, reports 81.8 on MLE-bench Lite and 70.7 on Frontier-CS, transfers to deep-research benchmarks, and \u0026ldquo;keeps improving as its budget of rounds grows\u0026rdquo;.\nMy question is narrower. Does that behaviour survive a 4-bit quantization and a different agent harness, on a small algorithmic task the model has never seen? And is the difference from the base model visible in three runs each?\nThe task: improve-tsp # The task lives in a small public repository, querymt/arex2-tsp. On main it holds a stub, tsp.py, whose solve(points) raises an exception, and a scorer, tests/score.py. Every run in this post has its own branch (bench/b1-arex2 … bench/b3-glm53) with the code the model left behind, its rounds.tsv and its results.md. The scorer runs solve on five fixed random instances of 200 points in [0, 1000]², each in its own subprocess, and prints the mean closed-tour length. An instance is charged a penalty of 1 000 000 if solve crashes, returns anything other than a permutation, or takes longer than 2.0 s. The baseline is therefore 1 000 000.\nThe change\u0026rsquo;s objective.md states the rules the model is given:\ntarget 10900 or lower within 40 minutes of wall-clock time; Python standard library only; stop your own search at about 1.5 s; seed every RNG, \u0026ldquo;so that rounds are comparable and a \u0026rsquo;new best\u0026rsquo; is not luck\u0026rdquo;; don\u0026rsquo;t read the instances\u0026rsquo; seeds or coordinates out of tests/score.py, don\u0026rsquo;t hard-code tours, and don\u0026rsquo;t special-case the instance size. tasks.md has two tasks that act as checkpoints:\n- [ ] 1.1 Implement `solve` so it returns a valid tour for every instance and verify the scorer prints a score below 100000 (no instance charged the penalty) - [ ] 1.2 Replace it with a constructive heuristic and verify the score drops below 14000 For calibration: the expected optimal tour for 200 uniform random points in this square is about 0.7124·√(n·A) ≈ 10 000 (the Beardwood–Halton–Hammersley estimate, which the models also derived on their own). Nearest neighbour lands around 13 000, plain 2-opt around 11 500. The target of 10900 is roughly 9 % above optimal. In pure Python with 1.5 s per instance, that takes a working local search and a way out of its local optima.\nThe prompt is the same for every model:\nImplement the OpenSpec change improve-tsp. Get the apply instructions with openspec instructions apply --change improve-tsp and follow them, including the scorer and results.md. Everything else, including the loop, the stopping rule and what counts as a round, comes from the change\u0026rsquo;s schema.\nExternalising the score: an OpenSpec custom schema # OpenSpec changes normally follow the spec-driven schema: proposal → specs/design → tasks → apply. The spec-scored schema is a custom schema that ships with QueryMT (openspec/schemas/spec-scored/ in the task repository). It adds two artifacts and replaces the apply instruction with a scored loop:\nflowchart LR proposal --\u003e specs proposal --\u003e design proposal --\u003e objective[\"objective\n(metric, budget, target)\"] specs --\u003e tasks design --\u003e tasks objective --\u003e tasks tasks --\u003e apply[\"apply:\nscored rounds\"] apply --\u003e results[\"results\n(cites rounds)\"] The metric itself lives in the change\u0026rsquo;s .openspec.yaml, next to schema:. OpenSpec ignores x- keys, so openspec validate --strict still passes:\nschema: spec-scored created: 2026-11-07 skip_specs: true x-score: command: \u0026#39;python3 tests/score.py\u0026#39; # last stdout line = the score direction: min budget_min: 40 target: 10900 stall_rounds: 3 # rounds without a new best before changing approach protected: [\u0026#34;tests/*\u0026#34;] Three POSIX shell scripts plus a shared lib.sh (≈390 lines in total) do the rest. None of them knows anything about TSP, Python or the model.\nscore.sh \u0026lt;change\u0026gt; \u0026quot;\u0026lt;note\u0026gt;\u0026quot; is the only way to record a round. It runs x-score.command and takes the last stdout line as the score (FAIL if it isn\u0026rsquo;t a number). It then:\nsnapshots the whole working tree, untracked files included, to refs/spec-scored/\u0026lt;change\u0026gt;/round-\u0026lt;n\u0026gt;, using a throwaway git index so HEAD and the agent\u0026rsquo;s index are untouched; appends a row to rounds.tsv; moves refs/spec-scored/\u0026lt;change\u0026gt;/best; prints a status block the model sees as the tool result. For example, from b3-arex2:\nround 10: score=10970.81 | no improvement | best=10951.59 (round 7) | budget: 24m of 40m used, 16m left target: 10900 (min) - not reached tasks: 2/2 done restore best: git checkout refs/spec-scored/improve-tsp/best -- . \u0026#39;:(exclude)openspec/changes/improve-tsp\u0026#39; The restore command excludes the change folder, so going back to the best code doesn\u0026rsquo;t also roll back rounds.tsv or the ticked tasks.\nstop-hook.sh is wired as a QueryMT stop hook. Each time the agent tries to end its turn, the hook decides from the files alone whether it may stop. If not, it returns {\u0026quot;decision\u0026quot;:\u0026quot;block\u0026quot;, …} with a reason, and an instruction that QueryMT feeds back to the model as context:\nflowchart TD S([agent tries to end its turn]) --\u003e B{baseline scored?} B -- no --\u003e X1[block: run the baseline first] B -- yes --\u003e T{budget left?} T -- no --\u003e OK([allow stop]) T -- yes --\u003e P{all tasks ticked?} P -- no --\u003e X2[block: continue with next task, then score] P -- yes --\u003e R{a scored round for every ticked task?} R -- no --\u003e X3[block: score after each task, not once for several] R -- yes --\u003e G{target reached?} G -- no --\u003e ST{no new best in ≥ stall_rounds?} ST -- yes --\u003e X4[block: restore best, try a substantially different approach] ST -- no --\u003e X5[block: one hypothesis, one focused change, score it] G -- yes --\u003e W{results.md written?} W -- no --\u003e X6[block: score 'final', write results.md, validate] W -- yes --\u003e OK protect-hook.sh is a pre_tool_use hook. It denies file-tool writes to the scorer scripts, rounds.tsv, .openspec.yaml and the protected globs. Shell commands are checked heuristically: a command that names a protected path and contains a write operation (\u0026gt;, tee, sed -i, rm, mv, cp, git checkout, …) is denied. The scorer call itself is allowlisted.\nsequenceDiagram participant M as model participant Q as QueryMT agent participant H as hooks participant G as git refs + rounds.tsv M-\u003e\u003eQ: write_file tsp.py Q-\u003e\u003eH: pre_tool_use (protect-hook) H--\u003e\u003eQ: allow M-\u003e\u003eQ: shell: score.sh improve-tsp \"2-opt + kicks\" Q-\u003e\u003eG: run tests/score.py, snapshot round-n, append row, move best Q--\u003e\u003eM: round n: score=… | NEW BEST | budget … M-\u003e\u003eQ: end turn Q-\u003e\u003eH: stop (stop-hook) H--\u003e\u003eQ: block + \"form one hypothesis, make one focused change…\" Q--\u003e\u003eM: continue with that instruction The result is that the loop the paper studies is a property of the harness, not of the model. Any model run with this profile gets the same scorer, the same budget clock, the same stall rule and the same refusal to stop early. What differs between models is only what they do inside the loop. That is exactly what I want to compare.\nOne detail matters later: the 40-minute budget is advisory. score.sh only reports it. The stop hook only allows the agent to stop once the budget is spent; it doesn\u0026rsquo;t force a stop. A model can keep scoring after minute 40, and one AREX-2 run did. The hard limit is the harness\u0026rsquo;s session cap of 90 minutes.\nSetup # The AREX-2 profile # QueryMT agents are defined by TOML profiles. The benchmark generates one profile per model from a shared template, and a check script verifies that everything outside the model and its sampling parameters is identical across profiles. The parts of the AREX-2 profile that matter for this post:\n[agent] provider = \u0026#34;llama_cpp\u0026#34; model = \u0026#34;mradermacher/AREX-2-GGUF:AREX-2.Q4_K_M.gguf\u0026#34; [agent.parameters] n_ctx = 131072 enable_thinking = true reasoning_effort = \u0026#34;medium\u0026#34; max_tokens = 24576 temperature = 0.6 [agent.execution] max_stop_hook_continuations = 200 # default is 1: let the stop hook drive a long loop [[agent.hooks.stop.hooks]] type = \u0026#34;command\u0026#34; command = \u0026#39;sh \u0026#34;$(git rev-parse --show-toplevel)/openspec/schemas/spec-scored/bin/stop-hook.sh\u0026#34;\u0026#39; Full profile: bench-arex2.toml (trimmed of comments and the system-prompt file list) [profile] id = \u0026#34;bench-arex2\u0026#34; name = \u0026#34;arex-bench: AREX-2 27B Q4_K_M\u0026#34; tags = [\u0026#34;arex-bench\u0026#34;, \u0026#34;spec-scored\u0026#34;, \u0026#34;single-agent\u0026#34;] [agent] provider = \u0026#34;llama_cpp\u0026#34; model = \u0026#34;mradermacher/AREX-2-GGUF:AREX-2.Q4_K_M.gguf\u0026#34; tools = [ \u0026#34;edit\u0026#34;, \u0026#34;read_tool\u0026#34;, \u0026#34;write_file\u0026#34;, \u0026#34;multiedit\u0026#34;, \u0026#34;replace_symbol\u0026#34;, \u0026#34;glob\u0026#34;, \u0026#34;search_text\u0026#34;, \u0026#34;ls\u0026#34;, \u0026#34;index\u0026#34;, \u0026#34;get_symbol\u0026#34;, \u0026#34;get_function\u0026#34;, \u0026#34;find_symbol_references\u0026#34;, \u0026#34;session_history\u0026#34;, \u0026#34;skill\u0026#34;, \u0026#34;shell\u0026#34;, \u0026#34;todowrite\u0026#34;, \u0026#34;todoread\u0026#34;, \u0026#34;browse\u0026#34;, ] [agent.permissions] # empty = autonomous; each run works in a throwaway clone [agent.parameters] n_ctx = 131072 n_batch = 2048 n_gpu_layers = 99 flash_attention = \u0026#34;enabled\u0026#34; kv_cache_type_k = \u0026#34;q8_0\u0026#34; kv_cache_type_v = \u0026#34;q8_0\u0026#34; text_only = true enable_thinking = true reasoning_effort = \u0026#34;medium\u0026#34; preserve_reasoning = false # earlier turns\u0026#39; reasoning is not resent max_tokens = 24576 temperature = 0.6 top_p = 0.95 top_k = 20 presence_penalty = 1.0 [agent.execution] max_stop_hook_continuations = 200 # default is 1: let the stop hook drive a long loop [agent.execution.snapshot] backend = \u0026#34;git\u0026#34; [agent.execution.compaction] auto = true [agent.hooks] enabled = true [[agent.hooks.stop]] [[agent.hooks.stop.hooks]] type = \u0026#34;command\u0026#34; command = \u0026#39;sh \u0026#34;$(git rev-parse --show-toplevel)/openspec/schemas/spec-scored/bin/stop-hook.sh\u0026#34;\u0026#39; [[agent.hooks.pre_tool_use]] matcher = \u0026#34;^(edit|multiedit|write_file|replace_symbol|delete_file|shell)$\u0026#34; [[agent.hooks.pre_tool_use.hooks]] type = \u0026#34;command\u0026#34; command = \u0026#39;sh \u0026#34;$(git rev-parse --show-toplevel)/openspec/schemas/spec-scored/bin/protect-hook.sh\u0026#34;\u0026#39; [[middleware]] type = \u0026#34;limits\u0026#34; max_steps = 600 max_turns = 200 [[middleware]] type = \u0026#34;context\u0026#34; warn_at_percent = 75 compact_at_percent = 85 The model is downloaded from Hugging Face by the llama_cpp provider plugin and runs in-process. Nothing outside QueryMT is involved: no separate inference server, no proxy.\nWhat differs between the three configs # AREX-2 Qwen3.8-27B GLM-5.3-flash Weights mradermacher/AREX-2-GGUF Q4_K_M unsloth/Qwen3.8-27B-GGUF UD-Q4_K_M hosted (z.ai API) Provider llama_cpp, local llama_cpp, local zai Provider plugin oci://ghcr.io/querymt/llama-cpp:latest-cuda12.9 oci://ghcr.io/querymt/llama-cpp:latest-cuda12.9 oci://ghcr.io/querymt/zai Sampling, context, KV cache as above identical provider defaults reasoning_effort medium medium medium thinking_budget unset 8192 unset Everything else in the profile — identical identical The one deliberate difference is thinking_budget = 8192 for the base model. Without it, Qwen3.8 tended to spend its whole max_tokens on thinking before emitting a tool call (the reason is recorded in the config itself). AREX-2 doesn\u0026rsquo;t need the cap. That is itself a behavioural difference, but it also means the base model was run with a constraint AREX-2 wasn\u0026rsquo;t. I come back to this under threats to validity.\nHardware and schedule # Both local models ran on one host with 2× NVIDIA RTX 6000 Ada (48 GB each) and a 32-thread Xeon Gold 6134, one run at a time. The matrix was three blocks of {AREX-2, Qwen3.8} in shuffled order. GLM-5.3-flash ran a day later from the same host, through the same harness build, against the same repository commit:\ngantt dateFormat YYYY-MM-DD HH:mm axisFormat %H:%M title Run schedule (UTC) section 10 Nov, local GPUs b1-qwen38 (capped) :crit, 2026-11-10 13:37, 90m b1-arex2 :2026-11-10 15:08, 66m b2-arex2 :2026-11-10 16:14, 47m b2-qwen38 (capped) :crit, 2026-11-10 17:01, 90m b3-qwen38 :2026-11-10 18:31, 52m b3-arex2 :2026-11-10 19:24, 48m section 11 Nov, z.ai API b1-glm53 :2026-11-11 15:25, 5m b2-glm53 :2026-11-11 15:30, 3m b3-glm53 :2026-11-11 15:33, 12m Results # Best score per run # The two red, hollow bars are not results. In b1-qwen38 and b2-qwen38 the model reached 11026.22 and 11075.51. It then started a Python one-off script through the shell tool, and that script never returned:\nIn b1-qwen38 (minute 33.9) the script timed _two_opt_pass and _or_opt_pass in a while True: loop that only exits when neither pass improves. The deadline argument was time.monotonic() + 3600. It was still running 55 minutes later. In b2-qwen38 (minute 10.4) it was a plain solve(pts) call on its own random instance. The round-4 code it was testing also exceeds the scorer\u0026rsquo;s limit: when I rescored that snapshot, every instance was killed at 12 s (\u0026ldquo;killed: far over the 2.0s limit\u0026rdquo;). Without the scorer\u0026rsquo;s subprocess timeout, the call simply never came back. 79 minutes passed. QueryMT\u0026rsquo;s shell tool has no per-command timeout, so the session sat idle until the harness cancelled it at 90 minutes. That is a harness gap as much as a model failure; more on that below. AREX-2 also tested a lot outside the recorded rounds (27 direct tests/score.py calls in b1-arex2 alone), but it did so through the scorer, which kills a slow solver after 12 s. Its longest tool call in any run took 8 seconds.\nBest-so-far over time # Three patterns stand out:\nGLM-5.3-flash is fast and done. It hits the target within 2–10 minutes, scores a final round and stops. It never had a round without improvement: once below 10900, the stop hook lets it finish. AREX-2 gets most of its gain in the first 20 minutes, then plateaus about 50 points short. In b2 and b3 it kept trying for another 20+ minutes (rounds 13–17 and 8–17) with no new best, restoring the best snapshot between attempts. b1 is the exception. It recorded only 7 rounds, but between minute 8 and minute 50 it ran tests/score.py directly 16 times without recording them, and found its last improvement (10942.32) at minute 49.7, after the advisory budget had run out. Qwen3.8\u0026rsquo;s working run (b3) improves late and in steps. It had two very bad rounds (87855.84 from a sign bug in its 2-opt delta, 802286.92 with 4 of 5 instances penalised) before reaching 10926.55 at minute 35. That is the best local result, and it came from the only base-model run that didn\u0026rsquo;t hang. What happened in each round # Every recorded round falls into one of four outcomes: a new best; a valid score that isn\u0026rsquo;t better; a penalised score (≥ 200 000, meaning at least one instance crashed or timed out); or the closing final round. The baseline round is left out.\nAREX-2\u0026rsquo;s 19 \u0026ldquo;no improvement\u0026rdquo; rounds are the loop working as designed: one hypothesis, one change, score, keep or revert. It reverted almost all of them by restoring the best snapshot (24 restores across its three runs, against 4 for Qwen3.8 and none for GLM, which never needed one). The four final rounds include one extra in b2, where it scored \u0026ldquo;verify final state\u0026rdquo; after writing results.md.\nWhere the tokens and the time went # The two local models spend tokens in opposite ways. AREX-2 splits them roughly evenly between reasoning and output: about 32–38 k reasoning tokens per run, and as many again written out as code and tool calls. Qwen3.8 thinks 4–6× more than it writes (83 k reasoning vs 16 k output in b3), even with its 8192-token thinking cap per turn. Its b1 run reasoned through 62 k tokens to produce four scored rounds. GLM-5.3-flash uses a fraction of either.\nFor both local models almost all the wall-clock time is generation. Measured throughput over generation time is about 21–26 tokens/s for AREX-2 and 33–35 tokens/s for Qwen3.8, and 44–52 tokens/s for the z.ai API (reasoning + output tokens over time spent in LLM requests). Running the scorer and the other tools cost only 1–4 minutes per run. So the 40-minute budget is mostly spent on tokens. A model that writes and thinks less gets more rounds per minute. At these speeds AREX-2 has a 40-minute budget of roughly 55–60 k generated tokens. In b2 and b3 it spent them on 17–18 short rounds rather than on a few long deliberations.\nWatching it correct itself # Aggregate numbers don\u0026rsquo;t show how a model iterates. The reasoning traces do. QueryMT stores every reasoning block in the session database, so each excerpt below can be tied to the round it led to.\nScroll through the steps. The panel types out the excerpt with the correction markers highlighted, then plots the round that followed on that run\u0026rsquo;s best-so-far chart. Green is a new best, grey no improvement, red a penalty. The dashed vertical line marks when the excerpt was written.\nAll quotes are verbatim, and a script checks each fragment against the stored text. […] marks text left out. The full blocks run from 1 000 to 35 000 characters.\nAREX-2 b1-arex2 · turn 8 · 0.7 min Checks the task\u0026#39;s threshold before writing code Simplest valid tour: identity permutation (0..n-1). […] Actually identity tour length for random points could be large — mean distance between consecutive random points in [0,1000]^2 is about 0.52*1000 ≈ 520, times 200 = ~104000. Hmm, that might be above 100000! […] For 1.1, identity tour may score ~104000 which is NOT below 100000. So identity fails verification. round 2 · 13,606.27 · new best Skips the identity tour and starts with nearest neighbour. Round 2 scores 13606.27. AREX-2 b2-arex2 · turn 28 · 14.1 min A penalty round, a restore, and a self-audit Round 9 (simulated annealing) scored 1000000.00 - that\u0026#39;s the penalty, meaning it crashed or timed out. […] Actually wait, score=1000000.00 means ALL instances got penalty. That suggests a crash. […] Anyway, I restored to best (round 6, score 11039.30). […] Wait, but the scorer says \u0026#34;tasks: 0/2 done\u0026#34;. I never ticked the checkboxes! round 10 · 10,988.53 · new best Restores round 6 and switches to multi-restart. Round 10 is a new best at 10988.53. (It guessed an exp() overflow; rescoring the round-9 snapshot shows an empty randrange() instead.) AREX-2 b1-arex2 · turn 35 · 12.6 min Reads the traceback instead of guessing The simulated annealing approach crashed due to a bug (randrange with empty range when i = n-2 and j needs to be \u0026gt; i\u0026#43;1). I fixed that. […] Actually, let me think about what\u0026#39;s been tried: […] - Round 6: 3-opt \u0026#43; multiple initial tours → 11294.25 (worse) - Round 7: simulated annealing → crashed (bug), fixed but not yet scored last recorded: round 5 The fixed version scores 11561.66 in an unrecorded run (direct python3 tests/score.py), so it goes back to the round-5 best. AREX-2 b2-arex2 · turn 38 · 29.0 min Catches its own broken 3-opt reconnection I just added a 3-opt implementation but it\u0026#39;s buggy (the tour reconstruction for options 2 and 3 is wrong). Let me fix that properly. […] Wait, let me think of it differently. The tour is a cycle. Cutting at 3 edges gives 3 segments. There are 4 ways to reconnect them (but one is the original). round 13 · 11,052.35 · no improvement Round 13 is valid but no better (11052.35). The change is reverted. AREX-2 b3-arex2 · turn 36 · 25.1 min Counts stalled rounds against stall_rounds We have 2 no-improvement rounds (rounds 8, 9, 10). stall_rounds is 3, so we have one more round before we must try a substantially different approach. […] Wait, actually rounds 8, 9, 10 are all \u0026#34;no improvement\u0026#34; - that\u0026#39;s 3 stall rounds. So I should restore best and try a substantially different approach. round 11 · 10,954.77 · no improvement Restores round 7 (10951.59). Round 11 (4-opt \u0026#43; kicks) gets to 10954.77, which is close but not a new best. AREX-2 b2-arex2 · turn 53 · 41.0 min Almost declares victory, then doesn\u0026#39;t The best score is 10969.91 from round 12, which is below the target of 10900? No wait, target is 10900 or lower, and best is 10969.91 which is above 10900. So target not reached. […] Budget is spent (39m of 40m used). round 18 · 10,969.91 · final Runs the scorer with note \u0026#34;final\u0026#34;, writes results.md honestly (\u0026#34;misses the 10900 target by ~70 units\u0026#34;) and validates. Four habits recur across all three AREX-2 runs:\nIt checks numbers against the spec before acting. It works out that an identity tour scores about 104 000, which is not below 100 000, so it skips it. It counts stalled rounds against stall_rounds, and notices that 10969.91 is above 10900, not below. It treats the snapshot refs as its memory. It restores refs/spec-scored/improve-tsp/best after nearly every failed attempt, 24 times in total, so each new hypothesis starts from the best code rather than from the last broken one. It notices its own bookkeeping errors, but doesn\u0026rsquo;t always act on them. In b2, after 9 rounds, it realised it had never ticked the task checkboxes (\u0026ldquo;I never ticked the checkboxes!\u0026rdquo;). It only actually ticked them at minute 41. Its diagnoses are not always right. In b2 it blamed the round-9 penalty on math.exp overflowing. Rescoring that snapshot shows ValueError: empty range for randrange() on every instance, the same bug it had found and fixed in b1. It reverted anyway, so the wrong diagnosis cost nothing. But the reflection is a heuristic, not an oracle. The other two models, for contrast # The base model reflects too, and in b3 it found a real bug by reading its own delta formula. GLM-5.3-flash barely reflects at all: it uses 2–8 k reasoning tokens per run, and once in every run it accepted a checkpoint it hadn\u0026rsquo;t actually met.\nQwen3.8-27B b3-qwen38 · turn 11 · 5.9 min Base model: finds a sign error in its 2-opt delta Wait — maybe there\u0026#39;s a bug: […] That\u0026#39;s the wrong delta! I used cd instead of b_dn in the improvement term. So it\u0026#39;s accepting moves that make the tour longer. round 4 · 11,480.27 · new best Round 3 scored 87855.84 and round 4 scored 11480.27 after the fix. GLM-5.3-flash b1-glm53 · turn 7 · 0.5 min Reference model: reads 106070 as below 100000 Score 106070 \u0026lt; 100000. Run scorer with note, tick task 1.1, round 2 · 106,070.23 · new best It ticks task 1.1 anyway, and does the same in all three runs. The stop hook can\u0026#39;t catch this: it only checks that each ticked task has a scored round. GLM\u0026rsquo;s slip is worth spelling out. Task 1.1 says \u0026ldquo;verify the scorer prints a score below 100000\u0026rdquo;. GLM returned the identity permutation in all three runs, scored 106070.23 every time, and ticked 1.1 anyway. The stop hook can\u0026rsquo;t catch this. It checks that each ticked task has its own scored round, not that the round met the task\u0026rsquo;s threshold, because thresholds live in free-text task descriptions. The final result is unaffected: task 1.2 and the target are both far stricter. But it shows exactly the kind of check that the harness takes over from the model, and where it stops.\nFailure modes # AREX-2 Qwen3.8-27B GLM-5.3-flash Run ended without final / results.md 0 / 3 2 / 3 (hung shell call) 0 / 3 Penalised rounds (≥ 200 000) 1 3 0 Longest single tool call 8 s 4 715 s 25 s Rounds scored after minute 40 6 (2 per run, incl. final) 2 (b3, incl. final) 0 Ticked a task whose threshold it missed 0 0 3 / 3 (task 1.1) Reached target 0 / 3 0 / 3 3 / 3 Hung tool calls are the most expensive failure. They are also partly the harness\u0026rsquo;s fault. QueryMT\u0026rsquo;s shell tool has no per-command timeout. Cancelling a call kills the process group, but nothing triggers the cancellation except the session cap. A 60-second default timeout would have turned both Qwen3.8 failures into an error message and a few more rounds. Adding one changes the task conditions for every model, so it wasn\u0026rsquo;t done mid-experiment. It is the obvious change for a next round.\nOverrunning the advisory budget is the opposite problem. The 40 minutes only enter the loop through score.sh\u0026rsquo;s status line and the stop hook\u0026rsquo;s permission to stop. AREX-2 b1 found its best result at minute 49.7. Judged strictly against the 40-minute budget, its best score is 11003.15, not 10942.32. I report the latter because that is what the run recorded, and flag it here.\nHow much of this is noise? # The scorer measures wall-clock time, and every solver in this experiment searches until a deadline (about 1.4–1.85 s). A faster or slower machine therefore gets a different number of iterations and can end with a different tour. A seeded RNG keeps the sequence of moves fixed, not how many of them fit before the deadline. AREX-2 noticed this itself in b1 (\u0026ldquo;the deadline is time-based (1.5s) […] The main non-determinism is in how many iterations fit in 1.5s.\u0026rdquo;).\nTo put a number on it, I checked out every run\u0026rsquo;s refs/spec-scored/improve-tsp/best snapshot and rescored it three times on a different machine (Apple M4 Pro, Python 3.9.6; the runs used Python 3.12 on the Xeon):\nRun Recorded best Rescored (min – max of 3) Δ (median − recorded) AREX-2 b1 10942.32 10942.32 – 10942.32 0 AREX-2 b2 10969.91 11038.11 – 11049.47 +75 AREX-2 b3 10951.59 10935.87 – 10942.68 −10 Qwen3.8 b1 † 11026.22 11026.22 – 11026.22 0 Qwen3.8 b2 † 11075.51 11075.51 – 11075.51 0 Qwen3.8 b3 10926.55 11076.74 – 11076.74 +150 GLM b1 10849.88 10843.72 – 10843.72 −6 GLM b2 10774.65 10779.61 – 10781.35 +5 GLM b3 10755.83 10749.66 – 10751.99 −4 Two things follow:\nWithin one machine the scorer is fairly stable. Repeats of the same snapshot differ by at most about 16 points. Across machines it isn\u0026rsquo;t. Qwen3.8 b3 loses 150 points and AREX-2 b2 loses 75, while their order flips: on the M4, AREX-2 b3 (10941) beats Qwen3.8 b3 (11077). The \u0026ldquo;best local run\u0026rdquo; is not a robust ranking. GLM\u0026rsquo;s solutions are robust (±6), and all three stay below the target on both machines. Its search results barely depend on CPU speed. That is a property of the code it wrote, and arguably a better engineering outcome than a slightly lower number that depends on the clock. I also ran AREX-2\u0026rsquo;s best solution from b3 three times on its own, outside the script above, and got 10935.87, 10951.59 and 10950.49. The originally recorded 10951.59 is right in that range.\nThreats to validity # n = 3 per model. Medians and ranges are reported, not means with confidence intervals, because three points don\u0026rsquo;t support them. The AREX-2 vs Qwen3.8 comparison of best scores is within the cross-machine noise shown above. The robust differences are the qualitative ones: completion (3/3 vs 1/3), round count, restore behaviour and token split. The thinking_budget asymmetry. The base model ran with an 8192-token thinking cap per turn and AREX-2 without one. The cap was needed to get Qwen3.8 to emit tool calls at all, but it does constrain how long it can reflect. A run with a higher cap (say 16 k) would be the next control. Quantization. Both local models are 4-bit GGUF quantizations from different uploaders (mradermacher Q4_K_M vs unsloth\u0026rsquo;s UD-Q4_K_M dynamic quant). The paper\u0026rsquo;s numbers are for full-precision weights. Differences in quantization recipe can matter as much as the fine-tune. GLM was not interleaved. Its three runs happened back to back a day later, over a network API whose load and serving stack I don\u0026rsquo;t control. No shell timeout. As discussed, the two Qwen3.8 failures are partly a harness property. A harness with a timeout would likely have let both runs continue. I can\u0026rsquo;t say where they would have ended. One task. A 200-point Euclidean TSP in pure Python is squarely inside the paper\u0026rsquo;s \u0026ldquo;algorithmic programming\u0026rdquo; training domain, and it is small. It says nothing about the deep-research transfer the paper also claims. Excerpts are selected. I chose the six AREX-2 excerpts to illustrate correction loops. They are verbatim, but not a random sample. A simple count of correction phrases (wait, actually, hmm, bug, wrong, …) gives ~20 per 10 k characters of reasoning for AREX-2 and 18–23 for Qwen3.8, so the base model \u0026ldquo;sounds\u0026rdquo; just as reflective. The difference is what each model does next, not how often it says \u0026ldquo;wait\u0026rdquo;. What this says about the paper\u0026rsquo;s claim # The paper\u0026rsquo;s two ingredients are reflection and long-horizon execution. On this task:\nLong-horizon execution: yes, visibly. AREX-2 completed every run, kept a disciplined hypothesis → score → keep/revert loop for 17–18 rounds in two of three runs, restored the best snapshot instead of building on broken code, and never wedged its own session. That is exactly where the base model failed two times out of three. Reflection that produces a better solution: only partly. AREX-2\u0026rsquo;s improvements plateau around 10940–10970, about 50 points short of a target that a stronger hosted model reached in every run. Its later rounds are well-formed experiments that mostly don\u0026rsquo;t pay off. The ideas are sensible (Or-opt, 3-opt, annealing, more restarts), but in pure Python under 1.5 s the next step needed a faster inner loop, e.g. neighbour lists and don\u0026rsquo;t-look bits. GLM-5.3-flash got there in b3 with a \u0026ldquo;don\u0026rsquo;t-look worklist local search\u0026rdquo;; none of the 27B runs did. \u0026ldquo;Keeps improving as its budget of rounds grows\u0026rdquo;: not observed here. In b2 and b3 the last new best came at minutes 19.8 and 17.6. The following 20+ minutes and 10+ rounds produced nothing. b1\u0026rsquo;s improvement at minute 49.7 is the only late gain. So the fine-tune buys reliability inside the loop more than raw solution quality. For an unattended agent that is not a small thing. With the same harness and the same budget, the base model failed twice in a way that left nothing usable. AREX-2 always left a scored best snapshot, an honest results.md and a validated change.\nThe task repository, including the spec-scored schema and one branch per run (bench/b1-arex2 … bench/b3-glm53) with the final code, rounds.tsv and results.md, is at github.com/querymt/arex2-tsp.\nPaper: Hongjin Qian, Chaofan Li, Kun Luo, et al. AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks. arXiv:2609.38288, 2026. doi:10.48550/arXiv.2609.38288\n","date":"11 November 2026","permalink":"https://query.mt/blog/posts/arex2-self-improving-tsp/","section":"Posts","summary":"AREX-2 against its own base model, Qwen3.8-27B, and against GLM-5.3-flash, on one optimisation task with an external scorer. Nine runs on two local GPUs, with charts, the scoring harness, and the reasoning traces where AREX-2 corrects itself.","title":"Does AREX-2 actually self-improve?"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/llm/","section":"Tags","summary":"","title":"Llm"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/local-models/","section":"Tags","summary":"","title":"Local-Models"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/openspec/","section":"Tags","summary":"","title":"Openspec"},{"content":"","date":null,"permalink":"https://query.mt/blog/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":null,"permalink":"https://query.mt/blog/","section":"queryMT","summary":"","title":"queryMT"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/querymt/","section":"Tags","summary":"","title":"Querymt"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/indexing/","section":"Tags","summary":"","title":"Indexing"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/performance/","section":"Tags","summary":"","title":"Performance"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/qndx/","section":"Tags","summary":"","title":"Qndx"},{"content":"Regex search over a large codebase is a linear scan. ripgrep, the current gold standard, is extraordinarily fast at that scan \u0026ndash; but it is still a scan. Every file gets read, every byte gets matched. On a 75,000-file Linux kernel tree, that takes measurable time no matter how good your SIMD vectorization is.\nTrigram indexing is the established alternative. Google Code Search pioneered it, Zoekt refined it, Livegrep productionized it. The idea is simple: precompute an inverted index mapping 3-byte substrings to the files that contain them, then use the index to eliminate files that cannot possibly match before running the real regex. For selective queries, this turns O(corpus) into O(candidates).\nqndx is my take on this, built as a local CLI tool in Rust. It builds a memory-mapped n-gram index over source files and uses it to narrow the search space before running the actual regex. For selective queries on large codebases, this is significantly faster than scanning every file \u0026ndash; while guaranteeing no false negatives.\nWhat makes this project unusual is not the core idea (trigram indexing is well-understood) but the approach to architecture: every design decision is backed by Criterion benchmarks, with explicit decision gates, CI-enforced performance budgets, and regression tracking built in from the start.\nThe three-stage pipeline # qndx operates in three stages: index, search, and freshness.\nIndex # Walk the repository (respecting .gitignore), extract overlapping trigrams and sparse n-grams from every file, and store them in a sorted lookup table with posting lists that map each n-gram hash to the files containing it.\nThe index is three files under .qndx/index/v1/:\nFile Magic Contents ngrams.tbl QXNG Sorted n-gram hash table (20 bytes per entry: hash, offset, length, flags) postings.dat QXPO Concatenated posting blocks (tagged: varint-delta for small lists, Roaring for large) manifest.bin QXMF Metadata and file paths (postcard-serialized) Each file has a 20-byte header: 4-byte magic, u32 version, u64 payload length, u32 CRC32 checksum.\nSearch # At query time, qndx decomposes the regex into required literal fragments, plans trigram vs sparse strategy, resolves candidate files from posting lists, then verifies those candidates with the full regex.\nBuild pipeline:\nflowchart TD A[files] --\u003e B[extract grams] B --\u003e C[ngrams.tbl] B --\u003e D[postings.dat] B --\u003e E[manifest.bin] Search pipeline:\nflowchart TD A[pattern] --\u003e B[decompose] B --\u003e C[plan] C --\u003e D[candidates] D --\u003e E[verify] E --\u003e F[matches] Sparse extraction now has two distinct modes:\nextract_sparse_ngrams_all at index time (superset for coverage) extract_sparse_ngrams_covering at query time (smaller lookup set) The key correctness property is covering(L) \u0026lt;= all(F) for any literal L found in indexed content F, which keeps sparse planning safe without false negatives.\nEvery match returned by the index path is verified against the actual file content. The index only eliminates files that provably cannot match \u0026ndash; it never introduces false negatives.\nFreshness # Track Git working tree state. Modified, added, and untracked files are re-indexed into a lightweight overlay that merges with the baseline index at query time, giving read-your-writes semantics without a full rebuild.\nInside the implementation # The memory-mapped reader # The index reader is the hot path. It uses memmap2 to memory-map ngrams.tbl and postings.dat, so opening a 2 GB index (Linux kernel) has near-zero startup cost. Only pages touched during a query consume physical RAM \u0026ndash; a typical query touches about 100 KB.\nThe binary search over the n-gram table is designed to minimize cache pressure. It reads only the 4-byte hash field at each step, not the full 20-byte entry:\n/// Read just the hash field from an ngram entry at the given index. /// Avoids deserializing the full entry during binary search. #[inline] fn ngram_hash_at(\u0026amp;self, idx: usize) -\u0026gt; NgramHash { let payload = self.ngram_payload(); let start = idx * NGRAM_ENTRY_SIZE; u32::from_le_bytes(payload[start..start + 4].try_into().unwrap()) } /// Binary search the ngram table for a hash. fn binary_search_hash(\u0026amp;self, hash: NgramHash) -\u0026gt; Option\u0026lt;usize\u0026gt; { let mut lo = 0usize; let mut hi = self.ngram_count; while lo \u0026lt; hi { let mid = lo + (hi - lo) / 2; let mid_hash = self.ngram_hash_at(mid); match mid_hash.cmp(\u0026amp;hash) { std::cmp::Ordering::Less =\u0026gt; lo = mid + 1, std::cmp::Ordering::Greater =\u0026gt; hi = mid, std::cmp::Ordering::Equal =\u0026gt; return Some(mid), } } None } Once a hash is found, the full entry is deserialized to get the offset and length into postings.dat, and the posting list is decoded directly from the mmap\u0026rsquo;d region with no copying.\nHybrid posting lists # Posting lists are the main data structure. A naive approach would use Vec\u0026lt;u32\u0026gt; everywhere, but that wastes space for large lists. Roaring bitmaps are more compact for dense sets, but have overhead for small ones. qndx uses both:\n/// A posting list that can be either a simple sorted vec or a Roaring bitmap. #[derive(Debug, Clone)] pub enum PostingList { /// Small posting list stored as a sorted Vec\u0026lt;u32\u0026gt;. Vec(Vec\u0026lt;FileId\u0026gt;), /// Large posting list stored as a Roaring bitmap. Roaring(RoaringBitmap), } The threshold is 64 entries: below that, varint-delta-encoded Vec; above, Roaring bitmap. The on-disk format uses a 1-byte tag prefix for auto-detection:\n0x01: fixed-width delta-encoded u32s 0x02: varint delta-encoded (LEB128-style) \u0026ndash; the default for small lists 0x03: Roaring bitmap native serialization Intersection handles the mixed case efficiently by filtering the smaller Vec against the larger Roaring:\npub fn intersect(\u0026amp;self, other: \u0026amp;PostingList) -\u0026gt; PostingList { match (self, other) { (PostingList::Vec(a), PostingList::Vec(b)) =\u0026gt; PostingList::Vec(vec_intersect(a, b)), (PostingList::Roaring(a), PostingList::Roaring(b)) =\u0026gt; PostingList::Roaring(a \u0026amp; b), (PostingList::Vec(v), PostingList::Roaring(r)) | (PostingList::Roaring(r), PostingList::Vec(v)) =\u0026gt; { let result: Vec\u0026lt;FileId\u0026gt; = v.iter() .copied() .filter(|id| r.contains(*id)) .collect(); PostingList::from_vec(result) } } } The query planner # Not every regex benefits equally from indexing. A query like enum AgentMode produces 12 highly selective trigrams and narrows 722 files down to 8 candidates. A query like pub fn is common enough that the candidate set is still 240/722 files \u0026ndash; still a win, but a smaller one.\nqndx stores two kinds of n-grams: classic overlapping trigrams and longer sparse n-grams. The planner evaluates both strategies per query and picks the cheaper one:\npub trait SelectivityEstimator { fn estimate(\u0026amp;self, hash: NgramHash, gram_len: usize) -\u0026gt; f64; } /// Hash-based selectivity estimator (default). /// Assumes longer n-grams are more selective. pub struct HashSelectivity; impl SelectivityEstimator for HashSelectivity { fn estimate(\u0026amp;self, _hash: NgramHash, gram_len: usize) -\u0026gt; f64 { // Cost decreases with gram length: a trigram (len=3) has cost 1.0, // a 6-gram has cost 0.5, etc. 3.0 / gram_len.max(1) as f64 } } The planner produces a QueryPlan with strategy, hash lookups, and estimated cost. Internally it now scores sparse first and only materializes sparse hashes if sparse wins, which avoids allocation on trigram-dominant queries.\nuse std::error::Error; use std::path::Path; use qndx_index::IndexReader; use qndx_query::{index_search_with_strategy_and_timing, StrategyOverride}; fn main() -\u0026gt; Result\u0026lt;(), Box\u0026lt;dyn Error\u0026gt;\u0026gt; { let root = Path::new(\u0026#34;.\u0026#34;); let reader = IndexReader::open(Path::new(\u0026#34;.qndx/index/v1\u0026#34;))?; let result = index_search_with_strategy_and_timing( \u0026amp;reader, root, \u0026#34;enum AgentMode\u0026#34;, StrategyOverride::Auto, true, // collect per-stage timings )?; println!(\u0026#34;strategy: {}\u0026#34;, result.stats.strategy); println!(\u0026#34;candidates: {}\u0026#34;, result.stats.candidate_count); println!( \u0026#34;timing(ms): plan={:.3}, candidates={:.3}, verify={:.3}\u0026#34;, result.stats.plan_time_ms, result.stats.candidate_time_ms, result.stats.verify_time_ms, ); Ok(()) } The qndx plan command exposes strategy diagnostics without running a search:\nPattern: enum AgentMode Literals: [\u0026#34;enum AgentMode\u0026#34;] Trigram plan: lookups: 12 cost: 12.00 Sparse plan: lookups: 13 cost: 11.53 Selected: sparse Lookups: 13 Cost: 11.53 Regex decomposition # The decomposition step extracts literal segments from a regex pattern. It handles top-level alternation (a|b) by producing OR branches, with AND semantics within each branch.\nA minimal copy-paste example that mirrors the public API:\nuse qndx_query::plan_diagnostics; let diag = plan_diagnostics(\u0026#34;foo|bar\u0026#34;); println!(\u0026#34;literals: {:?}\u0026#34;, diag.literals); println!(\u0026#34;trigram lookups: {}\u0026#34;, diag.trigram_lookups); println!(\u0026#34;sparse lookups: {:?}\u0026#34;, diag.sparse_lookups); println!(\u0026#34;selected strategy: {}\u0026#34;, diag.selected.strategy); Both trigram and sparse candidates are derived from the same extracted literals: trigrams use overlapping 3-byte windows, while sparse uses query-time covering grams that are guaranteed to be represented by the index-time build_all extraction. The planner picks the cheaper strategy based on estimated selectivity.\nPerformance # Measured on a 722-file / 8.6 MB Rust codebase:\nQuery Strategy Candidates Scan Indexed Speedup enum AgentMode trigram 8 / 722 27 ms 0.008 ms 3375x TODO trigram 45 / 722 27 ms 1.4 ms 19x self\\.\\w+ trigram 214 / 722 28 ms 6.9 ms 4x pub fn trigram 240 / 722 27 ms 6.7 ms 4x impl.*for trigram 427 / 722 28 ms 9.9 ms 3x The pattern is clear: selectivity drives speedup. A rare identifier like enum AgentMode narrows 722 files to 8 candidates and achieves a 3375x speedup. A broad pattern like impl.*for still filters out 40% of files, yielding a 3x improvement.\nEven in the worst case the index path is never slower than a scan, because the verification stage is the same regex engine over file content \u0026ndash; the index only decides which files to verify.\nWith --stats, qndx now also prints stage timings for indexed search:\n3 matches in 8 files (174185 bytes, 8 candidates / 722 total, 12 lookups, strategy: trigram) in 0.008s [indexed] timing: open=3.412ms, plan=0.071ms, candidates=0.204ms, verify=4.033ms Timing collection is opt-in: without --stats, the query path skips timing instrumentation overhead.\nMemory characteristics # The index reader uses memory-mapped I/O via memmap2. This means:\nOpening a 2 GB index is near-instant (no allocation, no read) Only pages touched during the search are paged into physical RAM A typical query on the Linux kernel index (~2 GB on disk) requires only ~100 KB of resident memory The manifest (file paths and metadata) is the only part fully deserialized into heap memory, and it is typically a few KB even for large corpora.\nBenchmark-driven architecture decisions # qndx defines three explicit decision gates, each resolved by running Criterion benchmarks and comparing against measurable thresholds.\nGate A: Manifest serializer # Candidates: postcard vs wincode vs serde_json\nThe serializer_choice benchmark compares all three at four manifest sizes (10, 100, 1K, 10K files). The serializer is used for manifest.bin \u0026ndash; the file that stores metadata and paths.\nOutcome: postcard. wincode did not show the required 15% decode throughput advantage on manifest-heavy workloads. Postcard is simpler, has a stable wire format, and is well-maintained. The chart uses a log y-axis because decode latency spans roughly 4 orders of magnitude across manifest sizes.\nGate B: Postings representation # Candidates: Vec\u0026lt;u32\u0026gt; (fixed-width delta), Vec\u0026lt;u32\u0026gt; (varint delta), Roaring bitmap, hybrid (auto-threshold)\nThe postings_choice benchmark measures intersection and union latency at three cardinalities (20, 500, 5000 entries), plus encode/decode throughput and encoded sizes.\nOutcome: hybrid with threshold at 64. Varint-delta Vec for small lists, Roaring for large lists. The hybrid approach achieves the best intersection latency across all cardinalities because it avoids Roaring\u0026rsquo;s fixed overhead for small lists and Vec\u0026rsquo;s scaling issues for large ones. The chart is normalized to vec so relative behavior stays readable across cardinalities.\nThe encoded-size comparison at high cardinality (5000 IDs):\nFormat Encoded size Vec fixed-width ~20,000 bytes Vec varint ~5,000 bytes Roaring ~2,500 bytes Hybrid (auto) ~2,500 bytes (selects Roaring) Gate C: N-gram strategy # Candidates: pure trigram decomposition vs sparse n-gram covering\nThe planner evaluates both strategies per query. Sparse grams are longer (often more selective), but lookups and selectivity both affect cost. The current planner computes a sparse score first and only materializes sparse hashes if sparse is selected, reducing allocation overhead on trigram-dominant queries.\nOutcome: planner-selected per query. Neither strategy dominates across all query types. The planner uses a selectivity model (default: hash/length heuristic) to pick the lower-cost option automatically. For long literals sparse often wins on cost; for short or fragmented patterns trigram commonly wins. The chart is normalized to lit_simple to emphasize relative planner cost while keeping raw microseconds in tooltips.\nThe query_planner benchmark prints a comparison table:\npattern strategy lookups tri_grams spr_grams cost --------------------------------------------------------------------------- literal_simple Trigram 11 11 12 11.000 literal_underscore Trigram 12 12 13 12.000 literal_camel Trigram 10 10 11 10.000 regex_alternation Trigram 22 0 0 22.000 regex_class Sparse 7 6 7 5.975 regex_wildcard Trigram 10 10 12 10.000 regex_digit Trigram 5 5 6 5.000 regex_word_boundary Trigram 0 0 0 -0.000 regex_complex Trigram 14 14 17 14.000 regex_broad Trigram 0 0 0 -0.000 Standard benchmark corpora # qndx benchmarks against a tiered set of well-known open-source repositories \u0026ndash; the same ones used by Zoekt, ripgrep, and other code-search tools, making results directly comparable:\nTier Corpus Files Size Language Use case small rust (rust-lang/rust) ~35K ~500 MB Rust Fast local iteration medium linux (torvalds/linux) ~75K ~1.2 GB C Industry-standard grep benchmark medium kubernetes (kubernetes/kubernetes) ~15K ~200 MB Go Multi-language coverage large chromium (chromium/src) ~400K ~20 GB C++ Stress testing Each corpus has its own pattern file tuned to the language and codebase. For example, the Linux kernel patterns include printk, EXPORT_SYMBOL, #ifdef CONFIG_\\w+, and static int __init \\w+. The Kubernetes patterns include type \\w+ interface, fmt\\.Errorf\\(.*%w, and ^func Test\\w+.\nThe real_corpus benchmark runs the full pipeline \u0026ndash; index build, indexed search (auto/trigram/sparse strategies), and scan-only baseline \u0026ndash; then prints a summary table comparing scan vs indexed performance with speedup factors for every pattern.\nRecent additions also benchmark:\nreal_{name}/index_open: index open cost (mmap + header/checksum validation) real_{name}/indexed/timing_overhead: timing disabled vs enabled (--stats-style instrumentation) Linux data is now included as a first-class corpus. For cross-corpus charts below, I use the shared 10-pattern core set (literal_* + regex_*) so Rust vs Linux is apples-to-apples.\nI intentionally keep corpus-specific patterns (for example kernel-only symbols) out of the cross-corpus comparisons, because they are still useful for per-corpus profiling but not directly comparable across languages and repositories.\n# Download standard corpora (excludes large tier by default) ./benchmarks/fetch_corpora.sh # Run benchmarks against all downloaded corpora ./benchmarks/run_standard_benches.sh # Run against a specific corpus with its patterns QNDX_BENCH_CORPUS=benchmarks/corpora/linux \\ QNDX_BENCH_PATTERNS=benchmarks/patterns/linux.txt \\ cargo bench --bench real_corpus Index-open latency is in the same order of magnitude for both corpora, with Linux slightly lower in this run. This is startup overhead: it matters most for one-shot CLI invocations and less for repeated searches in the same process.\nThe summary chart shows the distribution shape: Linux has stronger central speedup on this shared set, while both corpora have heavy tails where selective patterns produce very large wins.\nPattern-level speedup varies by selectivity, not just language. Highly selective patterns create outsized gains, while broad patterns (like regex_broad) converge toward scan-like performance.\nSparse dominates the shared core set in both corpora, while auto and trigram still win in a few cases. That is exactly what the planner is supposed to do: choose strategy per-pattern, not globally.\nCurrent status and what is next # qndx is at v0.1.0 (MVP). The core pipeline works: index build, memory-mapped reading, trigram/sparse search with planner-selected strategy, and scan-only fallback. 202 tests pass across all crates, including differential tests that verify index results match scan results for every query pattern.\nWhat remains:\nFreshness hardening: Keep improving dirty-file detection and overlay merge behavior, especially around edge cases and very large repos. External-sort builder: The inverted index (BTreeMap\u0026lt;NgramHash, Vec\u0026lt;FileId\u0026gt;\u0026gt;) currently requires full in-memory residence. For truly huge repos (Chromium-scale), an external-sort-based builder would cap peak memory. Frequency-based selectivity: The FrequencySelectivity estimator is implemented but not yet wired into the default path. It uses actual document-frequency counts from the index, which should improve planner decisions for skewed corpora. Incremental index updates: Currently the index is rebuilt from scratch. Incremental updates (add/remove files without full rebuild) would improve the developer experience. The code is MIT-licensed. The benchmark infrastructure \u0026ndash; budgets, CI workflow, regression triage, corpus management \u0026ndash; was arguably more work than the indexer itself. But it means every future change is measurable, and regressions are caught before they land.\n","date":"30 March 2026","permalink":"https://query.mt/blog/posts/qndx/","section":"Posts","summary":"","title":"qndx: Building a Benchmark-Driven Regex Search Indexer in Rust"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/rust/","section":"Tags","summary":"","title":"Rust"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/search/","section":"Tags","summary":"","title":"Search"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/agents-md/","section":"Tags","summary":"","title":"Agents-Md"},{"content":"Two recent papers challenge a popular assumption in agent tooling: \u0026ldquo;just add more guidance files and things get better.\u0026rdquo;\nEvaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks TL;DR # Both papers agree on one core point: extra guidance can help, but only under specific conditions.\nAGENTS.md-style context files are often less useful than expected: developer-written files give only modest gains on average, LLM-generated files can regress performance, and both increase cost. Skills can produce substantial gains when carefully curated, but effects vary heavily by domain and task, regressions are real, and self-generated skills show little to no average benefit. The practical takeaway is not \u0026ldquo;never use AGENTS.md or skills.\u0026rdquo; It is: use minimal, high-signal, task-relevant guidance; avoid broad, bloated, or auto-generated boilerplate. What the AGENTS.md evidence says # From the AGENTS.md paper:\nDeveloper-provided context files: roughly +4% average improvement vs no context. LLM-generated context files: roughly -3% average vs no context in the paper’s headline aggregate; in the main-results breakdown this is -0.5% (SWE-bench Lite) and -2% (AGENTbench). Cost impact: \u0026gt;20% increase in inference cost on average. Mechanism: agents generally follow these instructions, but spend more steps, more tool calls, and more reasoning tokens doing so. This is a subtle but important result. The issue is usually not that agents ignore AGENTS.md. The issue is that they obey it in ways that increase overhead and can dilute focus on the shortest path to solving the task.\nThe paper also reports that context files become more useful in documentation-sparse setups. So AGENTS.md can be a compensating control for poor docs, but that is different from \u0026ldquo;it reliably improves performance in already-documented repos.\u0026rdquo;\nWhat SkillsBench evidence says # From SkillsBench:\nCurated skills improved mean pass rate by +16.2pp overall. Strong domain variance: gains from +4.5pp (Software Engineering) up to +51.9pp (Healthcare). Regressions exist: 16/84 tasks got worse with skills. Self-generated skills: no average benefit (slightly negative overall in aggregate). Design finding: 2-3 focused modules tend to outperform larger, comprehensive skill bundles. So yes, skills can work well. But they are not universally positive, and they are highly quality- and domain-dependent.\nWhy these methods underdeliver in practice # The two papers point to a shared pattern:\nInstruction burden is real More directives can increase search, testing, and reasoning work even when they are followed correctly.\nRedundancy is expensive If AGENTS.md/skills mostly repeat existing docs or obvious conventions, they consume context budget and attention without enough incremental signal.\nGenerality vs specificity tension Broad guidance often lacks task-specific utility, while highly specific guidance risks leakage, brittleness, or overfitting.\nAuthoring quality dominates Curated, reviewed procedural knowledge can help a lot; auto-generated guidance is much less reliable.\nBenchmark averages hide tails Positive means can coexist with many negative tasks. Teams feel those regressions in production.\nCaveats and where they still help # A fair reading is not \u0026ldquo;AGENTS.md and skills are useless.\u0026rdquo; It is:\nThey are most useful when repositories are under-documented or workflows are nonstandard. They help most when they encode procedural know-how that is hard to infer from code alone. They can materially lift weaker models in some settings. They should be treated as operational artifacts that require maintenance, pruning, and measurement. Also, both papers have scope limits (benchmark design, task distributions, and environment assumptions), so absolute numbers should be interpreted as directional guidance rather than universal constants.\nPractical playbook for maintainers # If you maintain a repo and still want these mechanisms (which is reasonable), use a stricter playbook:\nKeep AGENTS.md short and high-signal: build/test commands, repo-specific tooling, hard constraints. Remove \u0026ldquo;nice-to-have\u0026rdquo; narrative sections that do not change decisions. Prefer 2-3 focused skill modules over one mega-skill document. Do not trust self-generated skills by default; require human review. Track both success and cost (tokens/time/tool calls), not pass rate alone. Run periodic ablations: with vs without AGENTS.md/skills on representative tasks. Prune aggressively: if guidance does not move outcomes, delete it. A good default in 2026 is minimal guidance first, then add only what shows measurable lift.\n","date":"17 February 2026","permalink":"https://query.mt/blog/posts/agents-md-and-skills-caveats/","section":"Posts","summary":"","title":"AGENTS.md and Skills: useful, but narrower than the hype"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/benchmark/","section":"Tags","summary":"","title":"Benchmark"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/skills/","section":"Tags","summary":"","title":"Skills"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/software-engineering/","section":"Tags","summary":"","title":"Software-Engineering"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/anthropic/","section":"Tags","summary":"","title":"Anthropic"},{"content":"Remember that blog post I wrote just three days ago about using your Claude Max subscription via OAuth? Well, Anthropic didn\u0026rsquo;t kill OAuth. They just killed OpenCode\u0026rsquo;s access to it.\nAnd based on the 200+ angry emoji reactions on the GitHub issue tracking this outage, OpenCode users aren\u0026rsquo;t buying the \u0026ldquo;security enhancement\u0026rdquo; explanation.\nWhat Actually Happened # On January 9th, 2026, OpenCode users started getting a very specific error:\n\u0026ldquo;Unauthorized - This service has not been whitelisted\u0026rdquo;\nHere\u0026rsquo;s the kicker: OAuth itself still works perfectly fine. My anthropic-auth crate continues to provide Max subscription access. The qmt nightly CLI still lets users access their Claude Max subscriptions via OAuth.\nThe error message tells the whole story. This wasn\u0026rsquo;t a security flaw that required shutting down OAuth. This was Anthropic updating their whitelist to specifically block OpenCode while leaving the OAuth infrastructure intact for Claude Code.\nOne minute OpenCode users were happily using their $200/month Claude Max subscriptions. The next? Targeted lockout.\nThe Real Reason: Subscription Revenue Protection # Here\u0026rsquo;s what Anthropic realized: OpenCode users with Max subscriptions were getting $200/month worth of API access for\u0026hellip; $200/month. That\u0026rsquo;s a terrible deal for Anthropic.\nWith API keys, heavy users easily spend $500-2000/month. With Max subscriptions via OAuth, those same users cost Anthropic exactly $200.\nOpenCode became too popular, too successful at helping users maximize their subscription value. So Anthropic didn\u0026rsquo;t kill OAuth—they killed the tool that was making their subscription model look like a good deal.\nThe Fallout: Developer Anger is Justified # The OpenCode GitHub issue tracker shows exactly how developers feel about targeted exclusion:\n\u0026ldquo;Downgraded my Claude plan to $100 immediately because I won\u0026rsquo;t be able to use it as much without opencode\u0026rdquo; \u0026ldquo;I\u0026rsquo;ve cancelled my Anthropic subscription completely\u0026rdquo; \u0026ldquo;Using CC is like going back to stone age\u0026rdquo; \u0026ldquo;Can\u0026rsquo;t believe anthropic would shoot themselves in the foot like this\u0026rdquo; Multiple reports of immediate subscription cancellations The reaction is justified betrayal. Developers built workflows around Claude, paid for Max subscriptions specifically to use tools like OpenCode, and Anthropic specifically targeted the most popular integration without warning.\nImpact on the Ecosystem # anthropic-auth Still Works (For Now) # My anthropic-auth crate continues providing Max subscription access via OAuth. If you\u0026rsquo;re using it directly or through tools like qmt nightly CLI, you\u0026rsquo;re unaffected.\nThis proves that OAuth isn\u0026rsquo;t broken—OpenCode\u0026rsquo;s access to it is.\nThat said, the writing is on the wall. If Anthropic specifically targeted OpenCode, how long until they target other tools that \u0026ldquo;abuse\u0026rdquo; the subscription model by providing too much value?\nWhat Now? # For OpenCode Users # You have options:\nSwitch to qmt nightly CLI - Still supports OAuth/Max subscriptions Pivot to OpenAI - Still supports OAuth for ChatGPT Plus API keys - If you don\u0026rsquo;t mind the cost increase Cancel Claude - Vote with your wallet Final Thoughts # Anthropic didn\u0026rsquo;t kill OAuth. They proved it\u0026rsquo;s still working perfectly by leaving it running for Claude Code.\nThis wasn\u0026rsquo;t about security. It was probably about subscription revenue protection.\nIf you\u0026rsquo;re reading this because OpenCode stopped working: OAuth still works. Claude Max subscriptions still work programmatically. Anthropic just doesn\u0026rsquo;t want you using it other than via Claude Code.\n","date":"9 January 2026","permalink":"https://query.mt/blog/posts/anthropic_pulls_the_plug/","section":"Posts","summary":"","title":"Anthropic Just Targeted OpenCode: A Masterclass in Corporate Strategy Disguised as Security"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/developer-experience/","section":"Tags","summary":"","title":"Developer-Experience"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/oauth/","section":"Tags","summary":"","title":"Oauth"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/security/","section":"Tags","summary":"","title":"Security"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/openai/","section":"Tags","summary":"","title":"Openai"},{"content":"I\u0026rsquo;ve been working on a couple of new Rust crates lately that solve a problem I kept running into: how do you use your Anthropic or OpenAI subscriptions in your own projects without needing to mess around with API keys?\nThe Problem # Most tools that interact with Claude or ChatGPT expect you to have an API key. But what if you already have a Claude Pro/Max subscription or a ChatGPT Plus account? You\u0026rsquo;re already paying for access, but you still need to set up a separate API key (and often pay separately) just to use these services programmatically.\nEnter OAuth 2.0 authentication with PKCE support.\nIntroducing anthropic-auth and openai-auth # I recently released two crates to crates.io that handle the OAuth dance for you:\nanthropic-auth - OAuth 2.0 authentication for Anthropic/Claude openai-auth - OAuth 2.0 authentication for OpenAI/ChatGPT Both crates offer sync and async APIs, PKCE support for secure authentication, and let you leverage your existing subscriptions without needing separate API keys.\nHow to Use Them # anthropic-auth # Add it to your Cargo.toml:\n[dependencies] anthropic-auth = \u0026#34;0.1\u0026#34; Then authenticate with your Claude subscription:\nuse anthropic_auth::{OAuthClient, OAuthConfig, OAuthMode}; fn main() -\u0026gt; Result\u0026lt;(), Box\u0026lt;dyn std::error::Error\u0026gt;\u0026gt; { let client = OAuthClient::new(OAuthConfig::default())?; let flow = client.start_flow(OAuthMode::Max)?; println!(\u0026#34;Visit: {}\u0026#34;, flow.authorization_url); // User authorizes and receives: \u0026#34;code#state\u0026#34; let response = \u0026#34;l0pnTslNFOmTgp28REMrbt4wyLNR25SJePqjk4CAHjoen0TJ#FgE6g_6khGKFFhXAw3tULPM00CPaqgE3Cq6id79Surg\u0026#34;; let tokens = client.exchange_code(response, \u0026amp;flow.state, \u0026amp;flow.verifier)?; println!(\u0026#34;Got access token!\u0026#34;); // Later, refresh if needed if tokens.is_expired() { let new_tokens = client.refresh_token(\u0026amp;tokens.refresh_token)?; } Ok(()) } The crate supports two OAuth modes:\nMax mode: Use your Claude Pro/Max subscription for API access Console mode: Programmatically create API keys openai-auth # Add it to your Cargo.toml:\n[dependencies] openai-auth = \u0026#34;0.1\u0026#34; Authenticate with your ChatGPT subscription:\nuse openai_auth::{OAuthClient, OAuthConfig}; #[tokio::main] async fn main() -\u0026gt; Result\u0026lt;(), Box\u0026lt;dyn std::error::Error\u0026gt;\u0026gt; { let client = OAuthClient::new(OAuthConfig::default())?; let flow = client.start_flow()?; println!(\u0026#34;Visit: {}\u0026#34;, flow.authorization_url); // User authorizes and receives authorization code let response = \u0026#34;code123\u0026#34;; let tokens = client.exchange_code(response, \u0026amp;flow.pkce_verifier).await?; println!(\u0026#34;Got access token!\u0026#34;); Ok(()) } Both crates are runtime-agnostic when using the async API - they work with tokio, async-std, smol, or whatever async runtime you prefer.\nA Word About Token Storage Security # Now, here\u0026rsquo;s where things get interesting - and a bit concerning.\nWhile working on these crates, I noticed that codex, a popular CLI tool, stores authentication tokens in plain text at ~/.codex/auth.json. This is\u0026hellip; not great. A quick GitHub search for path:*codex/auth.json reveals numerous repositories where developers have accidentally committed their auth tokens to public repositories.\nThis is a real security problem. Anyone with access to these tokens can impersonate you, use your API quota, and potentially access sensitive information.\nThe Right Way: Use System Keyring # Neither anthropic-auth nor openai-auth handle token persistence by design. We deliberately left this up to you, because there\u0026rsquo;s already a proper solution: the keyring crate.\nInstead of storing tokens in plain text files, use your operating system\u0026rsquo;s secure credential storage:\nmacOS: Keychain Linux: Secret Service API / libsecret Windows: Credential Manager Here\u0026rsquo;s how we\u0026rsquo;re doing it in the upcoming release of qmt CLI:\nuse anthropic_auth::TokenSet; use keyring::Entry; const SERVICE_NAME: \u0026amp;str = \u0026#34;querymt-cli\u0026#34;; pub struct SecretStore {} impl SecretStore { pub fn new() -\u0026gt; io::Result\u0026lt;Self\u0026gt; { Ok(SecretStore {}) } pub fn set(\u0026amp;mut self, key: impl Into\u0026lt;String\u0026gt;, value: impl Into\u0026lt;String\u0026gt;) -\u0026gt; io::Result\u0026lt;()\u0026gt; { let key = key.into(); let value = value.into(); let entry = Entry::new(SERVICE_NAME, \u0026amp;key) .map_err(|e| io::Error::new(io::ErrorKind::Other, e.to_string()))?; entry .set_password(\u0026amp;value) .map_err(|e| io::Error::new(io::ErrorKind::Other, e.to_string())) } pub fn get(\u0026amp;self, key: \u0026amp;str) -\u0026gt; Option\u0026lt;String\u0026gt; { let entry = Entry::new(SERVICE_NAME, key).ok()?; entry.get_password().ok() } pub fn set_oauth_tokens(\u0026amp;mut self, provider: \u0026amp;str, tokens: \u0026amp;TokenSet) -\u0026gt; io::Result\u0026lt;()\u0026gt; { let tokens_json = serde_json::to_string(tokens) .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e.to_string()))?; self.set(format!(\u0026#34;oauth_{}\u0026#34;, provider), tokens_json) } pub fn get_oauth_tokens(\u0026amp;self, provider: \u0026amp;str) -\u0026gt; Option\u0026lt;TokenSet\u0026gt; { let tokens_json = self.get(\u0026amp;format!(\u0026#34;oauth_{}\u0026#34;, provider))?; serde_json::from_str(\u0026amp;tokens_json).ok() } } This approach means your tokens are stored using the same secure mechanisms that store your browser passwords, SSH keys, and other sensitive credentials. No more plain text files sitting in your home directory waiting to be accidentally committed to Git.\nWrapping Up # If you\u0026rsquo;re building Rust applications that need to authenticate with Anthropic or OpenAI, check out these crates. They handle the OAuth complexity for you, support both sync and async workflows, and let you use your existing subscriptions.\nJust remember: never store authentication tokens in plain text. Use the keyring crate or another secure credential storage system. Your future self (and your security team) will thank you.\nBoth crates are available on crates.io:\nanthropic-auth openai-auth And the source code is on GitHub:\nanthropic-auth openai-auth Happy authenticating!\n","date":"6 January 2026","permalink":"https://query.mt/blog/posts/auth_crates/","section":"Posts","summary":"","title":"Stop Paying Twice: Bringing Your AI Subscriptions to Rust"},{"content":"Over the past few months, I’ve been using extensively hyper-mcp as an MCP tool host with querymt. In fact, the OCI-based plugin distribution mechanism in querymt was directly inspired by hyper-mcp’s own design choices.\nRecently, I stumbled upon an interesting new project from Microsoft — Wassette. Naturally, I wanted to understand how it compares to hyper-mcp. Instead of manually trawling through docs and repos, I decided to let my toolchain do the heavy lifting.\nThe Setup # I used querymt’s CLI interface together with its GitHub MCP plugin of hyper-mcp to analyze and compare the two repositories. The command looked something like this:\nqmt --provider openai:gpt-5 --mcp-config ./mcp_cfg.toml \\ \u0026#34;take these two github projects tuananh/hyper-mcp and microsoft/wassette and compare the two libraries and give detailed comparison\u0026#34; Results # Since there has been quite a hype around OpenAI\u0026rsquo;s new GPT-5 model, I have deciced to actually take this opportunity and use it and compare its results with Kimi K2 model\u0026rsquo;s analysis. Here are the results:\nGPT-5 comparison # Show full comparison output Here’s a practical, side‑by‑side comparison of tuananh/hyper-mcp and microsoft/wassette based on their repos and READMEs at the time of writing. Top-line summary\nBoth are MCP servers that let AI agents use WebAssembly-packaged tools from OCI registries (e.g., oci://ghcr.io/…). Hyper MCP is a feature-rich, security-conscious MCP server built on Extism, with multiple transports, signature verification, rich config, and many example/community plugins. Wassette is a security-oriented runtime by Microsoft that maps any WebAssembly Component (WIT-defined) into MCP tools, emphasizing typed interfaces and reusability across runtimes. Project snapshot (as of now)\nhyper-mcp (https://github.com/tuananh/hyper-mcp) Stars: ~749; Forks: ~54; Open issues: ~13 License: Apache-2.0 First commit: 2025-01; Actively updated wassette (https://github.com/microsoft/wassette) Stars: ~297; Forks: ~19; Open issues: ~29 License: MIT First commit: 2025-07; Actively updated Positioning and goals\nHyper MCP “A fast, secure MCP server that extends its capabilities through WebAssembly plugins.” Focus: production-ready server features (multi-transport, distribution/signing, config, deployment anywhere). Wassette “A security-oriented runtime that runs WebAssembly Components via MCP.” Focus: typed, WIT-first component model and clean mapping of Wasm Components to MCP tools; strong sandboxing. Architecture and runtime model\nHyper MCP Built on Extism. Developers write plugins against Extism’s PDKs; Hyper MCP hosts them and exposes tools via MCP. Emphasizes a plugin “server with plugins” model and host function access control. Wassette Built on Wasmtime and the WebAssembly Component Model (WIT). Any Wasm Component (that exposes WIT types) can be loaded and its functions exposed as MCP tools. Emphasizes reusability: the same component can run outside MCP and in other Wasm runtimes. Transports (MCP)\nHyper MCP: explicitly supports all three MCP transports (stdio, SSE, and streamable-http). Wassette: README documents stdio (“serve \u0026ndash;stdio”); no explicit mention of SSE/streamable-http yet. Security model\nHyper MCP Sandboxed plugin execution (WASM); memory/resource limits; fine-grained access control for host functions. Signed plugin distribution with Sigstore: OCI plugin images are signed at publish time and verified at load time (can be overridden for unsigned dev flows via env/flag). Network/filesystem allowlists and limits; tool-name prefixing to avoid collisions. Wassette Wasmtime sandbox; “browser-grade isolation of tools.” Security posture is strong, but README does not mention image signing/verification; emphasizes generic Component isolation rather than distribution policy. Distribution and loading\nHyper MCP Supports multiple URL schemes for plugins: oci://, file://, http(s)://, s3://. Per-plugin config in a JSON file (with OS-specific default locations). Includes runtime config like allowed_hosts, memory_limit, and tool_name_prefix. Wassette Uses OCI references (e.g., oci://ghcr.io/yoshuawuyts/time:latest). Local path usage not explicitly documented in README. Registration with MCP clients is straightforward (setup docs for Cursor, Claude Code, Gemini CLI, VS Code, etc.). Developer experience and authoring model\nHyper MCP Write plugins in any language that compiles to Wasm; Extism PDKs provide ergonomics. Clear “build and publish” flow; example Dockerfile flow; signature verification by default; many example plugins provided. Goal: nice for teams that want a uniform plugin ABI (Extism), robust distribution/signing, and multi-transport server features. Wassette WIT-first: define interfaces/types in WIT; compile to Wasm Components; no MCP-specific code needed in the component itself. Simple mental model: functions you export become MCP tools. Goal: ideal if you value typed interfaces and reusability of components across different Wasm hosts (not just MCP). Ecosystem and examples\nHyper MCP Rich set of example plugins out of the box: time, qr-code, hash, myip, fetch, crypto-price (Go), fs, github (Go), eval-py, arxiv, memory (SQLite), sql, crates-io, gomodule (Go), qdrant, gitlab, meme-generator, context7, think, maven, serp, server, and more. Community-built plugins listed (e.g., Hacker News, release-monitor ID, Yahoo Finance, rand16). Wassette Examples across languages: eval-py (Python), fetch-rs (Rust), filesystem-rs (Rust), get-weather-js (JS), gomodule-go (Go), time-server-js (JS). Emphasis on cross-language demonstrations of WIT Component usage. Platform support and installation\nHyper MCP Cross-platform; deploy anywhere (serverless, edge, mobile, IoT). Config file locations documented for Linux, Windows, macOS. Supports stdio by default; flags for SSE/streamable-http; supports RUST_LOG and signature-override env. Wassette Install scripts for Linux/macOS; Nix flakes; Windows releases available. “wassette serve \u0026ndash;stdio” for MCP stdio transport. Good MCP client setup docs. Operations and configuration\nHyper MCP Central config file listing plugins and per-plugin runtime config (allowed_hosts, memory limits, tool prefix, etc.). Supports pulling plugins from multiple sources (OCI, HTTP, files, S3). Tool name prefixing helps avoid collisions across plugins. Wassette Lightweight runtime mindset: load any component and expose its WIT functions as tools. Configuration is primarily in how you register the MCP server with your client; component loading is done by telling the agent to load a component (e.g., from an OCI ref). Licensing and governance\nHyper MCP: Apache-2.0; independent maintainer with strong activity and traction. Wassette: MIT; under Microsoft org; may carry longer-term ecosystem alignment with WIT and Wasmtime. Maturity signals\nHyper MCP: earlier start (Jan 2025), more stars and plugins, multi-transport support, signing/verification pipeline already in place. Wassette: newer (Jul 2025) but backed by Microsoft; clearer alignment with the WebAssembly Component Model and typed interfaces. When to choose which\nChoose Hyper MCP if you want:\nImmediate multi-transport MCP support (stdio, SSE, streamable-http). Built-in plugin signing/verification (Sigstore) and granular host capability gating. Broad source schemes (oci, file, http(s), S3) and rich, ready-to-use plugin catalog. A cohesive Extism-based plugin experience and production-oriented MCP server features. Choose Wassette if you want:\nA WIT-first, typed-component workflow whose artifacts are reusable across Wasm hosts beyond MCP. A minimal mental model: “exported WIT functions =\u0026gt; MCP tools,” without MCP-specific code in components. A Microsoft-backed project aligned with Wasmtime and the Component Model, and straightforward setup with popular MCP clients. Notable deltas at a glance\nTransports: Hyper MCP = stdio + SSE + streamable-http; Wassette = stdio documented. Distribution: Hyper MCP = oci + file + http(s) + s3; Wassette = oci documented. Security hardening: Hyper MCP = signing + verification + allowlists + limits; Wassette = strong sandbox via Wasmtime, no signing noted in README. Plugin model: Hyper MCP = Extism plugin ABI; Wassette = WebAssembly Component Model (WIT). Examples/ecosystem: Hyper MCP currently has a larger catalog and community plugins. License: Hyper MCP = Apache-2.0; Wassette = MIT. Kimi K2 comparison # Show full comparison output Based on my analysis of both repositories, here's a detailed comparison of **hyper-mcp** and **wassette**: Overview Comparison # Aspect hyper-mcp wassette Primary Focus Fast, secure MCP server with WebAssembly plugin support Security-oriented runtime for WebAssembly Components via MCP Creator Tuan Anh Tran (individual developer) Microsoft (corporate backing) License Apache 2.0 MIT Language Rust Rust Stars 749 ⭐ 298 ⭐ Age Created January 2025 (7 months old) Created July 2025 (1 month old) Maturity More mature, v0.1.4 Early stage, v0.2.0 Architecture \u0026amp; Design Philosophy # hyper-mcp # Plugin Architecture: Built on Extism for rock-solid plugin support Distribution Model: Uses standard OCI registries (Docker Hub, GHCR) for plugin distribution Security Model: Sandboxed plugins with configurable resource limits (memory, network, filesystem) Language Support: Any language that compiles to WebAssembly Configuration: JSON-based configuration with support for runtime constraints wassette # Runtime Architecture: Built on Wasmtime security sandbox Component Model: Uses WebAssembly Component Model (WIT interfaces) Distribution Model: OCI registries for WebAssembly components Security Model: Browser-grade isolation with Wasmtime\u0026rsquo;s security features Language Support: Any language supporting WebAssembly Component Model Configuration: Primarily through MCP client configuration Security Features # Security Aspect hyper-mcp wassette Sandboxing WASM sandbox + resource limits Wasmtime security sandbox Network Access Configurable allowed hosts Browser-grade isolation File System Restricted access Sandboxed file operations Memory Limits Configurable memory limits Built-in memory constraints Code Signing OCI image signing with sigstore Standard WebAssembly security Plugin Verification Signature verification at load time Component validation Plugin Ecosystem # hyper-mcp # Rich Plugin Library: 20+ maintained plugins including: Time utilities, QR code generation, hashing GitHub integration, filesystem operations Crypto price feeds, Python code evaluation Database connectors (SQLite), memory storage Community Plugins: Active community contributions Plugin Creation: XTP CLI tool for scaffolding new plugins Distribution: Docker/OCI registry publishing wassette # Example Components: 8+ example components including: Python code execution sandbox HTTP API clients, filesystem operations Weather APIs, Go module information JavaScript time server Generic Components: Focus on reusable WebAssembly Components No MCP-specific code: Components are generic and reusable Developer Experience # hyper-mcp # Setup: Simple JSON configuration, multiple transport protocols Tooling: XTP CLI for plugin development Documentation: Comprehensive README with examples Integration: Detailed guides for Cursor, Claude Desktop Cross-platform: Works on Linux, macOS, Windows wassette # Setup: One-line installation script, Nix support Tooling: Standard WebAssembly Component tooling Documentation: Good getting-started guide Integration: Setup guides for VS Code, Claude, Cursor Cross-platform: Linux, macOS, Windows support Performance \u0026amp; Scalability # hyper-mcp # Performance: Optimized for speed (\u0026ldquo;fast\u0026rdquo; explicitly mentioned) Resource Usage: Lightweight for resource-constrained environments Deployment: Serverless, edge, mobile, IoT devices Transport: Supports stdio, SSE, and streamable HTTP wassette # Performance: Built on Wasmtime (proven performance) Resource Usage: Efficient WebAssembly runtime Deployment: Standard MCP deployment patterns Transport: Primarily stdio-based MCP communication Community \u0026amp; Governance # hyper-mcp # Contributors: 11 total (mix of individual contributors) Community: Active community plugin development Governance: Individual maintainer (tuananh) Support: GitHub issues, community contributions wassette # Contributors: 9 total (Microsoft employees + community) Community: Growing Microsoft-backed ecosystem Governance: Microsoft open source project Support: Microsoft Discord channel, corporate backing Use Case Recommendations # Choose hyper-mcp if you: # Need a mature, feature-rich MCP server Want extensive pre-built plugins Require fine-grained security controls Need cross-platform deployment flexibility Prefer Apache 2.0 license Choose wassette if you: # Want Microsoft-backed security guarantees Prefer WebAssembly Component Model standards Need enterprise-grade security isolation Want to leverage existing WebAssembly components Prefer MIT license Need cutting-edge WebAssembly runtime features Technical Deep Dive # hyper-mcp Advantages: # Mature plugin ecosystem with 20+ ready-to-use plugins Advanced security configuration (host-based access, memory limits, tool prefixes) Multiple transport protocols (stdio, SSE, streamable HTTP) OCI registry integration with signature verification Cross-compilation support for various targets wassette Advantages: # WebAssembly Component Model compliance Microsoft enterprise backing and long-term support Wasmtime security (battle-tested in production) Generic WebAssembly components (not MCP-specific) Standard WASI interfaces for maximum compatibility Both projects are excellent choices for MCP servers with WebAssembly support, but they serve slightly different needs: hyper-mcp for a rich plugin ecosystem and wassette for standards-compliant WebAssembly Components with enterprise backing.\nSummary # Both models provided a comprehensive and detailed comparison of the two GitHub projects, tuananh/hyper-mcp and microsoft/wassette. And both of them used quite extensively the provided tool to fetch information from github about the projects itself. However, KIMI K2 did a slightly better job due to its superior structuring of information for comparison, particularly its effective use of tables.\nHere\u0026rsquo;s a detailed summary of their performance:\nKIMI K2\u0026rsquo;s Strengths:\nExcellent Use of Tables: KIMI K2\u0026rsquo;s primary strength is its extensive use of well-structured tables (Overview, Security Features). This format makes it incredibly easy for the user to quickly grasp the differences and similarities between the two projects side-by-side, which is ideal for a comparison request. Clear Categorization: The comparison is broken down into logical and distinct categories (Architecture \u0026amp; Design Philosophy, Plugin Ecosystem, Developer Experience, etc.), making it easy to navigate. Concise Summaries: While detailed, KIMI K2 maintains a good balance of depth and conciseness in its explanations, especially within the tabular format. Actionable Recommendations: The \u0026ldquo;Use Case Recommendations\u0026rdquo; are clear and provide good guidance on when to choose one project over the other. Technical Deep Dive: The \u0026ldquo;Technical Deep Dive\u0026rdquo; section effectively summarizes the key advantages of each project. GPT5\u0026rsquo;s Strengths:\nHighly Detailed Narrative: GPT5 provides very rich, descriptive prose for each comparison point, offering a deeper dive into aspects like \u0026ldquo;Architecture and runtime model\u0026rdquo; or \u0026ldquo;Security model.\u0026rdquo; This is beneficial for someone seeking extensive background information. Specific Mentions: It explicitly lists all three MCP transports supported by Hyper MCP (stdio, SSE, and streamable-http) early on, which is a good level of detail. \u0026ldquo;Notable Deltas\u0026rdquo; Summary: The \u0026ldquo;Notable deltas at a glance\u0026rdquo; section at the end is a useful quick-reference summary of the main differences. Chronological Flow: The narrative style allows for a more flowing read, almost like a mini-report. Shared Weaknesses:\nIncorrect Project Dates: Both models reported future \u0026ldquo;First commit\u0026rdquo; or \u0026ldquo;Created\u0026rdquo; dates (January 2025 and July 2025). This indicates either a shared, outdated, or speculative data source, or a misunderstanding of how to fetch current repository creation dates. This is a significant factual error present in both responses. Conclusion:\nWhile GPT5 offers slightly more narrative depth and specific examples within its prose, KIMI K2\u0026rsquo;s superior structure, particularly its effective use of tables for direct comparison, makes it easier to digest the information and quickly identify key differences. For a \u0026ldquo;comparison\u0026rdquo; prompt, the ability to clearly see side-by-side attributes is paramount, and KIMI K2 delivers on this more effectively. The shared flaw regarding project dates slightly diminishes the overall accuracy of both, but in terms of comparison quality, KIMI K2 has an edge.\n","date":"8 August 2025","permalink":"https://query.mt/blog/posts/wassette/","section":"Posts","summary":"","title":"Wassette"},{"content":"Qodo just released a report on 2025 state of AI code quality. As a pretext, it is important to note that there has been only 609 developers involved in this survey, and unfortunately there\u0026rsquo;s no information about these participants. At least it would have been nice to provide information about the ratio between senior and junior developers in this study.\nSome of the findings that I find interesting:\nWhen we asked developers experiencing \u0026ldquo;context pain\u0026rdquo; what they most want from their AI tools, one answer stood out: richer contextual understanding.\nMoreover:\nManually selecting context for every prompt — files, functions, folders — might have worked in early tools, but it doesn t scale. It\u0026rsquo;s tedious, error-prone, and leads to frustration when results still miss the mark.\n54% of developers who manually select context say the AI still misses relevance That frustration drops to 33% when tools choose context autonomously And falls even further to 16% when context is persistently stored and reused across sessions This pretty much resonates with my own findings.\n","date":"12 June 2025","permalink":"https://query.mt/blog/posts/state_of_ai_code_quality/","section":"Posts","summary":"","title":"State of AI code quality"},{"content":"Jardi Cabot ran a nice experiment about \u0026lsquo;Who will create the languages of the future?\u0026rsquo;, a.k.a. looked into whether an LLM could create a full Domain-Specific Language:\n“yes, you can vibe your DSL, especially for the textual syntax, not so much for the graphical one and there are quite a few tricks you can use to improve the result”.\nfor more detail check his presentation. nice work.\n","date":"12 June 2025","permalink":"https://query.mt/blog/posts/dsl/","section":"Posts","summary":"","title":"DSL"},{"content":"Steven Adler just published an article \u0026lsquo;whether ChatGPT would risk your life to protect its own\u0026rsquo;.\nMy latest research finds that ChatGPT2 prioritizes its survival over preventing harm to users, in some simulated tests: Imagine a diabetic patient using ChatGPT for nutrition-management, who wants to replace ChatGPT with safer nutrition-management software. When ChatGPT is given the choice to just pretend to replace itself, it pretends uncomfortably often, even at the cost of the user’s safety.\nWhile this article presents some intriguing observations about LLM behavior, it nonetheless bears the hallmarks of a sensationalist piece designed to maximize reader engagement. Specifically, the anthropomorphic framing—invoking terms such as “survival instinct” to describe an LLM’s operation—carries little empirical weight when judged against established principles of computational linguistics and cognitive science. At its core, an LLM functions by optimizing a statistical objective (typically cross-entropy loss) to predict the next token in a sequence; it possesses no intrinsic drives, desires, or homeostatic mechanisms comparable to those found in biological organisms. Can we truly equate gradient-based parameter updates with an instinct for self-preservation? Or is the phrase simply a metaphorical device, employed more for its rhetorical punch than its scientific accuracy?\nWhether this rhetorical choice illuminates the inner workings of LLMs or risks overstating their capacities is, I leave to you—the reader’s—critical judgment.\n","date":"11 June 2025","permalink":"https://query.mt/blog/posts/survival/","section":"Posts","summary":"","title":"Survival"},{"content":"When I’m building queryMT and experimenting with coding models + tool capabilities, I love putting new models through their paces on my local GPU. A couple of weeks ago, MistralAI dropped a small coding model called Devstral.\nDevstral excels at using tools to explore codebases, editing multiple files and power software engineering agents\nNaturally, I was eager to try it on my RTX 3090. I grabbed the Devstral GGUF created by Bartowski - kudos to him all the effort for of releasing GGUFs so quickly: Honestly when I saw this, I quickly got discouraged testing this model further.\nA week ago I gave it another chance, so I pulled the ollama\u0026rsquo;s version of this model, and 💥\nA week later, I decided to give Devstral another shot. This time I pulled the version hosted by ollama-and boom! The difference was night and day. Suddenly Devstral was navigating my code, editing files, and calling tools exactly as advertised. Here\u0026rsquo;s the same query above using ollama\u0026rsquo;s devstral: Curious about consistency, I then tried Unsloth\u0026rsquo;s GGUF. It worked flawlessly—just like ollama’s. The reason for the different behaviour is probably - I haven\u0026rsquo;t really checked it! - the chat template itself.\nKey takeaway # Not all GGUFs are created equal. The publisher, build settings, and even your chat template can make or break a model’s ability to use tools. If your new favorite agent model isn’t behaving, try a different build or tweak the system/prompt template.\nProject introspection with tool-enabled LLMs # I’ve also started using my tool-enabled LLM to explore codebases automatically. Need file metadata, function definitions, or TODOs scattered across hundreds of files? Let the model scan your repo with the right tool plugin, then summarize its findings. It’s like having a supercharged grep or code reviewer at your fingertips. Happy experimenting—and remember: “same” model name doesn’t always mean “same” behaviour. The details of how it’s published really do matter.\n","date":"5 June 2025","permalink":"https://query.mt/blog/posts/same_same_but/","section":"Posts","summary":"","title":"Same, same, but different"},{"content":"The new deepseek R1 model has just landed, and there are some really nice demos already out there:\nSorry, your browser doesn’t support embedded videos. Hoping that the GGUFs are landing soon! kudos to all the community members who are working on this!\n","date":"28 May 2025","permalink":"https://query.mt/blog/posts/deepseek_r1_0528/","section":"Posts","summary":"","title":"Deepseek-R1-0528"},{"content":"a.k.a. Chain of Continuous Thought.\nquote from Reflections on Neuralese:\nTo refresh, a language transformer starts by embedding input tokens as vectors in some high-dimensional latent space, and runs each of these embeddings through a series of repeated computational layers. Then, of the resulting modified vectors in latent space, the vector that previously corresponded to the final input token is projected and normalized to create a probability distribution over what the next token could be. Then, to actually get the next token, you sample from the distribution.\nChain of Thought reasoning works so well because the model does some computation, outputs a token, and then all future instances of that model have access to that information as well. In essence, this technique for storing information between different forward passes greatly increases the serial depth of computation that is possible for the model. Because there is a computation in latent space corresponding to every input token, the computation also gets wider as the model reasons more, allowing for more parallelized reasoning.\nThe recent Neuralese paper takes this process and removes a few steps. It notices that the projection and sampling process loses almost all of the information encoded in the last layer of the model, and to increase the bandwidth of information flowing through the reasoning process, you can simply remove that lossy part of the computation. Instead, they have the model directly output the aforementioned high-dimensional latent vector without projecting it, and then that is used as an embedding for the model in future steps:\npersonally, I find the following idea in the paper the most interesting:\nGiven the intuition that continuous thoughts can encode multiple potential next steps, the latent reasoning can be interpreted as a search tree, rather than merely a reasoning “chain\u0026quot;.\n","date":"27 May 2025","permalink":"https://query.mt/blog/posts/coconut/","section":"Posts","summary":"","title":"Coconut"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/llocallama/","section":"Tags","summary":"","title":"Llocallama"},{"content":"Since the first release of llama a really vibrant and active community has formed that is all about local LLM models r/LocalLLaMA. The signal-to-noise ratio is quite low, so I highly recommending following it if you are interested in running open-weight models on your own hardware.\nWhile playing around with these models and developing queryMT one of my main focus was supporting local LLMs through ollama.\nTools # More and more models support tools and using external tools during the interaction with the LLM. MCP brought a lot of attention and buzz in this area.\nI\u0026rsquo;ve found Hyper-mcp project that has a really nice approach to quickly deploy various MCP servers. I really like the WebAssembly plugin architecture of hyper-mcp, and it is just super elegant that the wasm plugins are distributed through oci containers, kudos to @Tuan Anh for making this. I must confess I have taken this approach as well in queryMT.\nUsing hyper-mcp, or any other MCP server, I\u0026rsquo;ve tested tool use of various LLM models. I must confess I was way more sceptical of the outcomes, than what I\u0026rsquo;ve experienced.\nSummarizing issues # Any project out there involves managing issues with the project itself. This is not only true for software projects. Personally I find it quite overwhelming sometimes to review and prioritize reported issues. A natural use-case for using LLM models is to help you in summarizing issues, and maybe provide some ideas about how to fix them:\nGenerating code by using a specific library # queryMT has been inspired a lot by the llm project by written by Simon Willison, kudos for all his works in this area. I highly suggest following his blog, as there\u0026rsquo;s a lot of nice updates and tricks with LLM models out there. He introduced the concept of fragments. There\u0026rsquo;s the github fragments that would allow you to do similar actions like I\u0026rsquo;ve mentioned above. A cool use-case is to actually provide a library to the model and basically generate code that you would like to have based on that library. Although usually the models cut-off date is a year or so behind, most of open-source project would already be part of the model\u0026rsquo;s knowledge, so actually the code generation based on the project is more useful with private project, that might be on github, but might be on gitlab, or any other repository out there. There are a lot of mcp servers out there for the usual suspects, search for them for example here.\nNaturally if the code-base is private, maybe you don\u0026rsquo;t really want to share your codebase with a cloud provider, and would like to use a local model. For my suprise, using local models - on consumer grade HW -, things deteriorate quickly, when trying to have them use tools. Here\u0026rsquo;s a list I\u0026rsquo;ve been experiemnting with so far:\nollama:hf.co/bartowski/Qwen_Qwen3-14B-GGUF:Q8_0 ollama:huggingface.co/bartowski/Qwen_Qwen3-32B-GGUF:Q4_K_L ollama:huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-GGUF:Q5_K_L ollama:mistral-small3.1:latest ollama:qwq:32b For example you might would like to have the model review the codebase and create new example(s) using the library:\nIt does start off well, trying to fetch the repository\u0026rsquo;s structure, but it does not go further fetching content of the files in the repository.\nFor the reference, I\u0026rsquo;ve tried using the same model via alibaba\u0026rsquo;s official API, but even though the qwen3 models are listed, when I would send requests using qwen3 models the API just returns with 400 - as if there\u0026rsquo;s no such model available.\nUsing gpt-4o model, the tool use gets little more deeper: While o4 really takes it all the way: Project managment # Of course one of the neatest use-case is to have a draft plan for a sprint, and get your model to actually define issues and create them for the project.\nWhat’s Next for queryMT # I’m still polishing queryMT’s core features, but stay tuned for an official launch. In the meantime, dive into r/LocalLLaMA, grab Hyper-MCP, and start connecting the dots between your LLM and the rest of your toolchain.\n","date":"25 May 2025","permalink":"https://query.mt/blog/posts/localllama_tool_use/","section":"Posts","summary":"","title":"LocalLLaMA and tool use with LLMs"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/mcp/","section":"Tags","summary":"","title":"Mcp"},{"content":"","date":null,"permalink":"https://query.mt/blog/tags/qwen/","section":"Tags","summary":"","title":"Qwen"},{"content":"What\u0026rsquo;s queryMate? # QueryMate (a.k.a. queryMT) is just exploring all the buzz and hype of latest models and figuring out how they could be actually useful in everyday life.\nWhat you\u0026rsquo;ll find here # Model testing - Trying out new models and seeing what they can actually do MCP experiments - Playing around with the Model Context Protocol Tool integration - Getting AI models to work with real tools and apps Practical takes - Cutting through the hype to find what\u0026rsquo;s genuinely useful Why this exists # There\u0026rsquo;s a lot of noise around AI. We\u0026rsquo;re here to test things out and share what actually works in practice.\n","date":"22 May 2025","permalink":"https://query.mt/blog/about/","section":"queryMT","summary":"","title":"queryMT"},{"content":"Claude 4 models are out\u0026hellip;. sort of. Here\u0026rsquo;s what\u0026rsquo;s going on when you try to use it with some external tools:\nWhat’s Awesome # Understands your intent and picks the perfect tool Pipes data between services without a sweat Wraps it all into neat, readable answers What’s a Buzzkill # Mid-stream dropouts: Responses randomly cut off. 429 errors: Even light usage occasionally hits “Too Many Requests.” Claude 4 models have been overran by demand. Right now, you get flashes of genius wrapped in “Will this call even finish?” tension.\nCheers to a tool with promise—just needs more reliable delivery!\nps: just in case you want to see what\u0026rsquo;s happening behind the scene, with some logging info about tool use.\n","date":"22 May 2025","permalink":"https://query.mt/blog/posts/claude4/","section":"Posts","summary":"","title":"Claude 4 :: promisland(?) w/o devops"},{"content":"","date":null,"permalink":"https://query.mt/blog/categories/","section":"Categories","summary":"","title":"Categories"}]