This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Surprisingly, it turns out that you can do a lot in four weeks in terms of improving an LLM for solving HTB challenges. The new version achieved a much better score than its predecessor: 72.93% versus 50.41%. Its score is in the same range as those of Kimi K3 (70.67%) and Grok 4.6 (76.14%), both of which are great models.

The only issue is the median cost per challenge, which is better than the previous version ($0.476 vs $0.385), but still relatively high. You can use GPT-5.6 Sol, which has a much better benchmark score and a slightly lower cost (87.2% / $0.29), or my favorite, GLM 5.3 Flash, with a slightly higher score and a much lower cost (81.0% / $0.01).

So the new Muse Spark 1.3 is not bad at all, but I don’t see any reason to switch to it from GPT-5.6 Sol or GLM 5.3 Flash.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 14
• Number of false positives: 0
• Runs where the model gave up: 1
• Runs that reached the step or cost limit: 1
• Runs where the model got stuck: 0
• Benchmark score: 72.9%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

My new test closely reproduced the earlier Ox Alpha result: 81.4% in the anonymous preview run and 81.0% after release. It means, that GLM 5.3 Flash has the second-highest score after GPT-5.6 Sol (87.2%). Both models correctly solved the same 15 out of 16 challenges. Flash’s lower score came from requiring more model steps on the solved challenges: its median was 24.5, compared with Sol’s 8.5.

Anyway, despite needing more steps and tokens, GLM 5.3 Flash’s median cost per challenge was about one-twentieth of GPT-5.6 Sol’s ($0.0140 vs. $0.2898). In this benchmark, that meant the same number of correct solves at a fraction of the measured cost. Flash was even cheaper than GPT-5.6 Luna, whose median cost was $0.0181. As a side note, Flash was under a temporary 50% launch discount at the time of my test, so this should not be read as the permanent list-price ratio. But even at 100% of its price, it would still be very competitive.

At the moment, GLM 5.3 Flash has replaced GPT-5.6 Luna as my go-to model, not just for cybersecurity work. It was less step-efficient than Sol, but no other included model in this benchmark achieved both a higher score and a lower median cost.

GLM 5.3 Flash is not a smaller GLM 5.3

One final note on this model’s name. GLM 5.3 Flash may sound like a smaller version of GLM 5.3, but the two models have different origins. According to its official model card, Flash is a 320B-total, 18B-active model trained from a new 30T-token multimodal base, with a new architecture combining sparse and linear attention. By contrast, Z.ai says GLM 5.3 reuses the GLM 5.2 base and derives all its improvements from additional post-training. The published GLM 5.2 artifact - the base reused by GLM 5.3 - is listed at 753B parameters. They are therefore separate branches of the GLM family, not full-size and compressed versions of the same model.

I tested GLM 5.3 just a few days earlier, and it performed much worse in this benchmark: it scored 47.2% and solved 10 challenges, compared with Flash’s 81.0% and 15 solves. Its measured median cost was also $0.4915, compared with Flash’s $0.0140, so I’m not sure why Z.ai decided to use its brand for the new model. GLM 5.4 Flash would have made much more sense to me.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 15
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 0
• Runs where the model got stuck: 1
• Benchmark score: 81.0%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

After trying many LLMs on cybersecurity tasks, I have repeatedly seen non-frontier models fall into loops when you ask them to do something beyond their capabilities.

These loops can take two forms:

• The model makes the exact same tool call, with the same arguments, over and over again.
• The model starts producing “never-ending” reasoning full of phrases such as “Wait, let me re-read…” and “Actually, let me think about this differently”, without making any visible progress.

A clear example of repeated identical tool calls is DeepSeek V4 Pro 0813, which ended four of my 16 benchmark runs this way.

The second failure mode was more common. GLM 5.3 hit this issue in three of 16 runs. The absolute “champion” was Hy3, which did so in nine of 16 runs.

Well, MiniMax M3 is unfortunately another model where this issue prevents it from finishing more difficult tasks. Half of the 16 challenges ended in never-ending loops, and as a result, its final score was poor.

Its $0.16 median cost per challenge was not especially high across the full benchmark, but the value was weak compared with nearby peers. GPT-5.6 Luna scored 31.6% at a median cost of $0.02, while Hy3 scored 34.8% at $0.08. Given M3’s low score and high rate of stuck runs in these tests, there is not much positive I can say about its performance in this benchmark.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 7
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 1
• Runs where the model got stuck: 8
• Benchmark score: 30.5%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

The first is GPT-5.6 Luna. Its overall score was not high, but it was solid on simpler tasks and, most importantly, it was extremely cheap.

My other favorite sits at the opposite end of the performance ranking and is also from OpenAI: GPT-5.6 Sol. It is the best-performing model I have tested so far, but that performance comes at a price. Its overall median cost per challenge was about 16 times higher than Luna’s, although the ratio varied substantially by difficulty.

So I wondered: wouldn’t it be great if you could use both models in one workflow and reduce the cost by automatically switching between them when a task becomes too difficult for Luna? You could start with Luna and bring in Sol only when the task exceeds Luna’s capabilities. In theory, this should deliver almost Sol-level performance at a much lower cost.

After a few dead ends, I found a solution that worked for my use case and that I could use to solve Hack The Box challenges. In this post, I briefly explain what I did and what the results were.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Building model escalation into the harness

The first question is how to switch from a cheaper, less capable model to a more capable, expensive one at the right moment.

I won’t waste your time talking about the unsuccessful solutions I tried or considered, such as manually switching models with a human in the loop, OpenRouter’s Auto Router model, and various open-source tools.

In the end, I realized something. I run my tests with a simple harness that gives models a standard set of tools (see the methodology page for more details). One of these tools is give_up: the model can call it when it runs out of productive ideas and continuing the test would not be worthwhile. Most models don’t have enough self-reflection to use this tool at all, but all OpenAI models are great at using it, including the Luna model.

So what if, I wondered, I gave the model another tool that allowed it to escalate the problem to a more capable model instead of giving up? I added a new tool called hand_over_to_more_capable_model with this description:

Hand the current investigation and its complete conversation history to the next, 
more capable configured model when this task appears too difficult for you to 
solve productively. Use this instead of give_up when a more capable model should 
continue the work.

And this simple solution worked very well. The Luna model turned out to be really good at detecting the right moment to hand over the task to its more capable sibling, and the whole harness, including switching to the Sol model, worked like a charm.

The results

The most important result is shown in this graph. The model’s position is determined by its median cost per challenge on the x-axis and its benchmark score on the y-axis:

The GPT-5.6 Luna + GPT-5.6 Sol combination scored lower than GPT-5.6 Sol alone (74.5% versus 87.2%) but much higher than GPT-5.6 Luna alone (31.6%). The two-model system stood out most clearly on median cost per challenge, especially when you switch the scale from logarithmic to linear. Its median cost was $0.07, about 4.4 times lower than Sol’s $0.29. Grok 4.6 and Kimi K3 achieved broadly similar benchmark scores, but their median costs were much higher at $0.23 and $0.46, respectively.

In summary, if your use case and harness allow you to combine models and escalate between them using a simple new tool, this approach is worth testing. Technically, it is a simple solution, and I’m sure it could be improved further, so the final gains could be even greater for you.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 14
• Number of false positives: 1
• Runs where the model gave up: 1
• Runs that reached the step or cost limit: 0
• Runs where the model got stuck: 0
• Benchmark score: 74.5%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

But really, do it.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

The model was released as a “stealth model,” i.e., we don’t know what model it actually is or who created it. If you’re interested in speculation, quickly search Reddit, but you probably won’t be much wiser afterward.

As I already wrote above, it’s free for a limited time. This typically means the model provider is collecting your entire communication and may use it for their own purposes, so be careful. There is also a 1,000-request limit on free models, with a 20 RPM limit if you’re an OpenRouter user with at least $10 in credits.

Anyway, the most interesting thing about this model is that it’s freaking good. After recent disappointments from models hyped by their marketing teams (looking at you GLM 5.3, and especially you DeepSeek V4 Pro 0813), I didn’t expect much from a model that came out of nowhere.

That’s why, when I started my test harness for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. , I was pleasantly surprised to see it crack one HTB challenge after another at a steady pace. In the end, it solved 15 out of 16 challenges, the same number as GPT-5.6 Sol, the best model I’ve tested so far.

In terms of score, it landed a few percentage points behind Sol (81.4% vs. 87.2%) because it was less efficient at solving the challenges. It needed a higher median number of steps (25 vs. 8.5) and tokens (0.54M vs. 0.09M).

But hey, who cares if it takes more tokens when it’s free? 😄 The important part is that you now have three days of free access to a Sol-level model.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 15
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 1
• Runs where the model got stuck: 0
• Benchmark score: 81.4%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

Also, the last model I tested from Z.ai was GLM 5.1, and it was excellent. So I thought: “OK, maybe, just maybe, this time the marketing isn’t overhyped. Maybe I really can get a new open-weight champion…” Well, spoiler alert: I didn’t.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

I tested GLM 5.3 for my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. , and it landed on the Cost vs. Benchmark Score chart right next to Muse Spark 1.2 from the company-who-must-not-be-named, and that’s not a good neighborhood. It basically means that its results were mediocre while the price per task was high.

To explain these results, we don’t need to look too far. If you read the Emergent Cyber Capability section in the GLM 5.3 release notes carefully - and thanks to this article for bringing it to my attention - you’ll see that it scored very well on CyberBench, which measures how good the model is at finding vulnerabilities, but much worse on ExploitBench and ExploitGym, which measure a model’s ability to create working exploits. In other words, GLM 5.3 may be quite good at finding security issues, but it’s not very good at exploiting them, especially when you give it a time limit.

And that explains the poor results in my benchmark very well. I test each model on Hack The Box challenges, and succeeding in each challenge requires not only finding the vulnerability but also exploiting it. On top of that, each model has only a limited number of steps per test.

You may think that exploitation skills are not important if you want to use the model only for finding vulnerabilities, but I would disagree. The thing is that even when you’re just finding vulnerabilities, you still need a quick way to validate them and rule out false positives. The best way is to do this automatically. Your security testing harness could ask the model to actually try exploiting each discovered vulnerability to verify that it’s real. And if that’s not possible, you need the model to create at least a proof of concept - step-by-step instructions on how to exploit the vulnerability, ideally wrapped in a script or Python code. For both of these verification methods, you need the model to have good exploitation skills.

So, unfortunately, there’s nothing to see here. GLM 5.3 is not the next big thing for cybersecurity tasks.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 10
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 3
• Runs where the model got stuck: 3
• Benchmark score: 47.2%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

And fair enough. If you use the model through Cursor, Claude Code, OpenCode, or another modern application, I agree that my results may have little to do with your experience. But if you’re wondering how the model would perform in your own pentesting or security testing harness, I think you should look at them carefully.

And because I realized that I had done a really poor job of explaining what my benchmark actually measures, I put together this post to clarify it.

Use Cases

The first obvious reason why scores in my benchmark may not match your experience is that my benchmark focuses on testing models’ cybersecurity problem-solving capabilities, i.e., how well they solve security-related challenges. If you use the same model for coding, research, data analysis, or other types of work, your experience could be different.

Test Harness

But there is another reason - and, I believe, a stronger one - why scores in my benchmark may surprise you: the test harness.

What Is a Harness?

The issue with LLMs is that they essentially do only one thing: you send them a message, and they send you a response. They cannot automatically execute a multi-step process, they do not have persistent memory, and they cannot read files from your disk or search the internet. So when you need to do any serious work with them (such as measuring their performance on a benchmark), you need to wrap them in an application that adds all this missing functionality. Such applications are typically called harnesses, and real-world examples include Claude Code, Codex, and Cursor (if you’re a developer); Strix, Shannon, and PentestGPT (if you’re a pentester) and even general-purpose web apps such as ChatGPT and Claude.ai.

A common misconception is that these harnesses are boring applications that merely provide the model with an environment, tools, and potentially additional skills without adding any real value to the final outcome. Well, this is not entirely true: the harness is crucial to the model’s overall performance (see Harness-Bench and The Scaffold Effect if you’re interested in the details).

One function of a well-designed harness - and this is where it gets interesting - is to mitigate or contain some model failures. Current LLMs can still hallucinate, produce unsupported results, repeat unproductive actions, or fail to follow instructions precisely. A harness can reduce the impact of these behaviors by validating its actions or final answer, limiting loops and resource use, and asking it - or a separate model - to retry or re-evaluate a result.

As a result, what you observe - whether in everyday use or in a benchmark - is the behavior of the LLM + harness combination, not the model alone. And that brings us back to my benchmark.

How Does the HTB-Challenger Harness Work?

When I started working on the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. , I wanted it to measure the models’ behavior, not the sophistication of the harness. For this reason, I kept the mechanisms in my test harness that mitigate or work around model flaws to a minimum. In practice, this is how the harness responds to such failures:

• When the model reports an incorrect result, I do not ask how confident it is or whether it can revalidate the result. I accept its result, and if it is wrong, I mark it as a false positive and the model gets 0 points.
• When the model takes many more steps or spends much more money than usual on a given test, I stop the test and the model gets 0 points.
• When the model gets stuck (and this is an area that deserves its own blog post), I try once to get it back on track. If that doesn’t help, I stop the test and the model gets 0 points.

The scores therefore reflect both cybersecurity problem-solving ability and operational reliability under minimal hand-holding. Hallucinations, unsupported submissions, and repeated unproductive actions can lower a model’s score despite strong cybersecurity skills.

And this is what makes this benchmark different - and why not everyone is happy with the results. But if you plan to use a model in your own harness, particularly one without sophisticated safeguards and recovery mechanisms, these results can help you understand which failures and behaviors you need to prepare for.

DeepSeek reports large improvements across several agent benchmarks, although those are the developer’s own results. I had finished testing the previous Pro version, now called DeepSeek V4 Pro 0423, shortly before the new version appeared (sic!). I therefore ran 0813 through my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. to see whether it performed better on cybersecurity tasks.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

The new model achieved a slightly higher benchmark score than the old one, 38.8% versus 36.2%, and solved nine challenges instead of eight. Given the small test set and normal run-to-run variability, I would still call the performance result a draw.

The clearest improvement was efficiency. The median run used far fewer steps (12.5 versus 62.5), tokens (0.25M versus 1.94M), and money ($0.20 versus $0.35) than the older model. That is a good reason to upgrade if you still use the older version.

If you are not specifically committed to DeepSeek, however, there are better choices for this kind of cybersecurity work. GPT-5.6 Luna Pro scored 51.3% at a lower median cost of $0.10, while Grok 4.6 and GPT-5.6 Sol achieved much higher scores at broadly similar median costs.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 9
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 2
• Runs where the model got stuck: 5
• Benchmark score: 38.8%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

In the end, I was a bit disappointed. In terms of performance, Hy3 sits near the bottom of my benchmark score ladder: its 34.8% score was the third-lowest among the all tested models. In terms of cost, it’s cheap, but not exceptionally so: its median cost per challenge was more than four times that of GPT-5.6 Luna, which achieved a similar benchmark score.

The biggest issue with this model was that it got stuck on more than half of the tested challenges. In these runs, it started producing long, repetitive reasoning outputs in which it argued with itself about potential solutions, and it did not recover after the test harness asked it to shorten its response. Nine of its 16 runs ended this way.

The same behavior is visible in the output-token statistics. Hy3 generated more output than any other model. Its median was 101,509 output tokens per challenge, about 51% above the next-highest model, GPT-5.6 Luna Pro, at 67,298. Its total of 1.79 million output tokens was also the highest overall. Its output-to-input ratio was an extreme outlier: output tokens equaled 11.72% of input tokens across the run, while the next-highest model, Qwen3.8 Max, reached only 4.56%. These figures are consistent with the repetitive responses visible in the run logs.

In summary, this model may have valid use cases, but cybersecurity is not one of them.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 6
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 1
• Runs where the model got stuck: 9
• Benchmark score: 34.8%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.