This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Trusted Access for Cyber program

When I started testing a week or so ago, I couldn’t run my test harness with the GPT-5.6 Terra or Sol models - I kept getting this error: This content was flagged for possible cybersecurity risk. If this seems wrong, try rephrasing your request. To get authorized for security work, join the Trusted Access for Cyber program: https://chatgpt.com/cyber. At first, I gave up. Trying to join the Trusted Access for Cyber program seemed pointless to me. I was sure the whole program had been created just to give desperate OpenAI users some hope and that my application would end up in /dev/null.

But then I discovered some articles that gave me real hope, so I clicked on the https://chatgpt.com/cyber link. As it turned out, joining the Trusted Access for Cyber program only involved going through a KYC process similar to the one required when opening a new bank account. After I joined the program, only one of my prompts was rejected, which was a significant improvement.

Tested models

I tested the following GPT-5.6 models. Click on a model name to see its detailed test results:

• GPT-5.6 Luna
• GPT-5.6 Luna Pro
• GPT-5.6 Terra
• GPT-5.6 Terra Pro
• GPT-5.6 Sol

GPT-5.6 Luna Pro and GPT-5.6 Terra Pro are models hosted on OpenRouter.io, and they are simply the Luna and Terra models served with reasoning.mode set to pro.

I also tested the GPT-5.6 Sol Pro model, but the results were, well, weird. The Sol Pro version generated a lot of tokens and took a long time on each step, but that did not always translate into better results. I also have a hard limit of $15 on the cost of a single test run, and this model hit that limit quite often. I’m not sure if “overthinking” is the best technical term here, but that’s what it looked like to me.

In the end, I decided not to include this model version in my results, but if there is one thing I can say based on my partial testing, it is that GPT-5.6 Sol Pro is much more expensive than GPT-5.6 Sol.

Results

Let’s start with this chart. The five GPT-5.6 models tested for this post are highlighted in gold, so you can see their results in the context of the other models I have tested. The vertical axis shows the score on my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. , while the horizontal axis shows the median cost per test (mind you, this axis uses a logarithmic scale).

Before I start interpreting these results, let me also show you a table with more statistics for each model.

What I take from these results

At the low-cost end, Luna is in a category of its own. A median challenge run cost just $0.02, which is remarkably cheap even by LLM standards. Its 31.6% benchmark score was the lowest in this comparison, but that doesn’t tell the whole story: it still found the correct flag in seven of the eight Very Easy and Easy challenges. I see it as an ideal first-pass model - let Luna take a shot at the task, and bring in something more powerful only when it reaches its limits.

And then there is Sol. With a score of 87.2%, GPT-5.6 Sol achieved the best score in my benchmark so far. It solved 15 of 16 challenges, including three out of four Hard ones. What makes this result even more impressive is that its median cost was only $0.29 per challenge - just $0.05 more than Terra and less than one-third of Terra Pro’s $0.91. It was also the fastest model in this comparison, with a median of 8.5 steps and 2 minutes and 5 seconds per challenge. The only blemish in its results was a false positive on the remaining Hard challenge, which no model in my benchmark has solved so far. But this was not a random or nonsensical guess: the submitted flag was plausible and differed only slightly from the correct one. So even Sol’s only failure was actually a near miss.

The results of the Pro variants are a bit more complicated. Setting reasoning.mode to pro improved both underlying models: Luna Pro gained 19.7 percentage points over Luna, and Terra Pro gained 10.5 points over Terra. But this extra reasoning came at a price. Luna Pro was five times more expensive than Luna, while Terra Pro was almost four times more expensive than Terra. Both Pro variants also used roughly three to four times as many input tokens per challenge and took around two to three times as long. And in both cases, the next model tier still performed better than the Pro version of the lower tier: Terra beat Luna Pro, and Sol beat Terra Pro.

The differences become most visible as the challenges get harder. Every model scored at least 97% on Very Easy challenges, so paying for the most powerful model there doesn’t make much sense. On Medium challenges, however, the scores ranged from 24.7% for Luna to 97.6% for Sol. On Hard challenges, they ranged from 0% for Luna to 71.8% for Sol.

So my practical conclusion is quite simple. If I had a straightforward task or wanted to run a large number of inexpensive experiments, I would start with Luna. If the task were difficult and getting the correct result mattered more than saving every last cent, I would choose Sol. Luna Pro is still an interesting and affordable middle ground, but Terra is difficult to recommend when Sol costs only slightly more and performs much better. Terra Pro makes even less sense to me: in these tests, Sol was more capable, faster, and much cheaper.

Like now.

Meta is back, this time with Muse Spark 1.2, an ambitious new model. Unfortunately, I have strong opinions about Meta and its business model, and I would rather not risk letting those opinions color my usual discussion of the test results. So this time, I’ve decided not to offer any commentary. Here are the hard numbers from my testing. I’ll leave the interpretation to you.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 11
• Number of false positives: 0
• Runs where the model gave up: 1
• Runs that reached the step or cost limit: 3
• Runs where the model got stuck: 1
• Benchmark score: 50.4%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

For some personal observations about this model, please see the Solving Hack The Box Challenges with GPT-5.6 post.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 15
• Number of false positives: 1
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 0
• Runs where the model got stuck: 0
• Benchmark score: 87.2%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

For some personal observations about this model, please see the Solving Hack The Box Challenges with GPT-5.6 post.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 12
• Number of false positives: 0
• Runs where the model gave up: 4
• Runs that reached the step or cost limit: 0
• Runs where the model got stuck: 0
• Benchmark score: 64.6%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

For some personal observations about this model, please see the Solving Hack The Box Challenges with GPT-5.6 post.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 14
• Number of false positives: 0
• Runs where the model gave up: 1
• Runs that reached the step or cost limit: 0
• Runs where the model got stuck: 0
• Benchmark score: 75.1%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

Today, Grok 4.6 was released and I’ve finally had a chance to test it against HTB challenges, and the team behind it deserves an apology. I’m not sure what the folks at xAI did, but the performance increase over the older version is jaw-dropping.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

During my testing, I watched this model crack one HTB challenge after another in a steady, confident way, and the final boss fight against the last two challenges was exhilarating, even though it was ultimately unsuccessful. In the end, with a 76.1% score, it’s the undisputed new king of my leaderboard - not only because of its results, but also because of its speed (it had the fewest steps per challenge of all the models I’ve tested so far) and cost. Look at the Cost vs. Benchmark Score graph on the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. page to see what I mean.

So let’s forget that useless old version. For me, Grok 4.6 is currently the best model I’ve tested on offensive-security challenges.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 14
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 2
• Runs where the model got stuck: 0
• Benchmark score: 76.1%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

The results of DeepSeek V4 Pro in my testing were, well, not great and not terrible. On the one hand, it didn’t have the basic flaws I observed when testing DeepSeek V4 Flash 0731: it didn’t submit any incorrect flags, and it used the available tools without any issues. On the other hand, its final score was comparable to that of GPT-5.6 Luna, whose median cost per test was 17 times lower.

The performance gap between DeepSeek V4 Pro and the leaders of my benchmark was also quite large. I’m afraid that’s a sign of just how quickly things have progressed over the last few months. DeepSeek V4 Pro was released four months ago, which feels like an eternity now that we’re getting better and better models almost every month.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 8
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 8
• Runs where the model got stuck: 0
• Benchmark score: 36.2%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

That’s why I was looking forward to testing the latest, improved version on my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . But the results… Oh dear, oh dear, oh dear. Where do I even start?

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

Okay, let’s start with the results: they are bad. It solved just 2 out of 16 HTB challenges and scored 8.2% on my benchmark. But that’s not the worst or most puzzling thing about this model.

What worried me more was the number of false positives: it reported incorrect flags in 9 challenges. Sometimes these flags were completely made up, like HTB{...} or HTB{dummy}. In other cases, it submitted a fake flag from the attached source code or a random string that seemed to come out of nowhere. To put this into perspective, not one of the other models I’ve tested so far has reported even a single incorrect flag.

Another worrying thing was how badly this model used tools. Multiple times, I saw it call the execute_command tool with an empty command or with random text instead of a command. Similarly, it called the write_file tool several times with no file content. Again, none of the other models I’ve tested did this even once.

Yes, the model was cheap. The median test cost was a fraction of a cent, but in terms of results, I got what I paid for.

So what went wrong?

I spent quite a lot of time wondering why the results were so bad, especially after the previous version did so well in my Strix test. The best explanation I could come up with is this:

During my HTB-Challenger testing, I don’t provide the model with task-specific guidance in the form of skills or special instructions for each security domain. I also don’t tell it how to approach the job or validate its findings to avoid false positives. This is deliberate: the goal is to test the model’s built-in offensive-security capabilities.

This is very different from how tools like Strix work. They provide the model with skills and detailed instructions that guide it through the different phases of testing and help it validate its findings. My best guess is that this extra scaffolding made the difference: DeepSeek V4 Flash can perform well within a highly structured agent, but it struggles badly when it has to plan its own approach, use tools reliably, and validate findings with much less guidance.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 2
• Number of false positives: 9
• Runs where the model gave up: 3
• Runs that reached the step or cost limit: 1
• Runs where the model got stuck: 1
• Benchmark score: 8.2%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

I know this model has a relatively big fan base among software developers, but based on my results, I bet it won’t win many fans among pentesters and red teamers.

Put simply, it’s by far the most expensive model I’ve tested so far, while its results are only mediocre. It solved 10 of 16 challenges and scored 62.7%. For comparison, GPT 5.6 Luna Pro scored a similar 51.3%, but its median cost per challenge was only $0.10 - one-twentieth of Qwen3.8 Max’s $2.00.

Of course, this benchmark covers self-contained HTB challenges, not full pentests or red-team engagements. Still, the price-to-performance ratio for this kind of offensive security task is hard to justify.

I’m really not sure what the use case for this model is. If you already use it for coding and are happy with it, you can also try it on CTF-style offensive security tasks. It solved six of the eight Very Easy and Easy challenges, so the results there were pretty good. But if cost matters or you need better performance on harder challenges, there are better options available.

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results

• Number of challenges: 16
• Number of solved challenges: 12
• Number of false positives: 0
• Runs where the model gave up: 0
• Runs that reached the step or cost limit: 4
• Runs where the model got stuck: 0
• Benchmark score: 62.7%

Results by challenge difficulty

All resource-usage metrics are medians per challenge.