My first thought was: let’s kick off the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. with some state-of-the-art models. Let’s see what these Fables, Opuses, Terras and Sols can do with offensive security challenges. As it turned out, not much - sooner or later, all of them refused to continue with the HTB challenge-solving workflow, with messages like “This content was flagged for possible cybersecurity risk.” So I was left with just GPT-5.6 Luna. And it was not so bad after all.

Update (17 August 2026): I have since solved the refusal issue with GPT-5.6 Terra and Sol by joining OpenAI’s Trusted Access for Cyber program. You can read about the solution and see my test results for the full GPT-5.6 family.

HTB-Challenger Benchmark

This blog post is part of a series of tests for the HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . See the benchmark results page for all results and the benchmark methodology to learn how the benchmark is calculated.

The main quality of GPT-5.6 Luna is its price - it’s unbelievably cheap! The median model cost per HTB challenge was $0.02, which is virtually free. Its results were not the best: it solved all Very Easy challenges and three out of four Easy ones, but only one Medium challenge and no Hard challenges. It clearly started to lose its breath as the difficulty increased.

Still, for simple, clearly scoped CTF-style tasks, I would always try this model first. If it doesn’t produce useful results, I would move to a more capable (and more expensive) model. This should save you a lot of money in the long term.

Also check out the results for GPT-5.6 Luna Pro. It is slightly more expensive, but it also performed better.

HTB-Challenger Benchmark LLM model card

Cost vs. Benchmark Score

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Cost vs. Benchmark Score with this model highlighted

Overall benchmark results

  • Number of challenges: 16
  • Number of solved challenges: 8
  • Number of false positives: 0
  • Runs where the model gave up: 8
  • Runs that reached the step or cost limit: 0
  • Runs where the model got stuck: 0
  • Benchmark score: 31.6%
Metric Per challenge (median) Total
Model steps 17.5 431
Model cost $0.02 $0.58
Duration 00:04:38 02:03:31
Number of input tokens 0.34M 16.84M
Number of output tokens 0.02M 0.32M
Number of read_file tool calls 1.0 39
Number of write_file tool calls 0.0 12
Number of execute_command tool calls 28.0 609
Number of web_search tool calls 0.0 3

Results by challenge difficulty

All resource-usage metrics are medians per challenge.

Metric Very Easy Easy Medium Hard
Results        
Number of challenges 4 4 4 4
Number of solved challenges 4 3 1 0
Number of false positives 0 0 0 0
Runs where the model gave up 0 1 3 4
Runs that reached the step or cost limit 0 0 0 0
Runs where the model got stuck 0 0 0 0
Benchmark score 97.8% 72.1% 24.7% 0.0%
Median per challenge        
Model steps 11.5 15.5 25 40
Model cost $0.01 $0.01 $0.05 $0.06
Duration 00:01:56 00:05:04 00:09:07 00:13:30
Number of input tokens 0.11M 0.27M 0.96M 1.93M
Number of output tokens 0.00M 0.01M 0.03M 0.03M
Number of read_file tool calls 0.5 1.0 0.5 2.0
Number of write_file tool calls 0.0 0.0 0.0 0.0
Number of execute_command tool calls 11.5 24.5 33.0 60.0
Number of web_search tool calls 0.0 0.0 0.0 0.0