63 dimensions of capability

The composite Nautilus Score across all Core Evaluation categories. Use it as a first-glance ranking, then drill into the category that matches your workload.

This category tests agent-style behaviors: tool-call generation, tool-use orchestration, multi-step task planning, debugging methodology, system design, and information extraction. The judge does NOT execute tools or code. For tool-call questions, criteria reward correctly-shaped JSON, proper parameter extraction, sensible decision logic about when to call vs ask. For planning/design questions, criteria reward structured thinking, realistic constraints, and addressing the non-obvious dimensions. For debugging questions, criteria reward methodology over guessing.

This category tests language-level coding, snippet generation, code translation, debugging, and specific framework/library tasks (largely arena-hard-derived). The judge does NOT run code — assessments must be based on reading the produced code for apparent correctness, idiomatic usage, and adherence to the prompt's specific requirements. Where the prompt asks for a specific language, framework, or API, using the wrong one drops the score significantly even if the alternative would work. Where prompts contain ambiguity or impossible asks (e.g., 'graph navigation with linear complexity'), acknowledging the impossibility/ambiguity is rewarded.

This category was inherited from arena-hard's 'creative_writing' label, but in practice contains a mix of actual creative writing (song lyrics, story scenes, fanfic), technical writing (recipes, guides, schemas), coding tasks, and educational explanations. The criteria for each question are tuned to what the question actually asks. For creative-writing prompts, criteria reward craft, voice, and constraint adherence. For technical prompts, criteria reward correctness and completeness. The judge does not execute code.

This category is the NAUTILUS-original modern production engineering set — frameworks named with versions (Next.js 14, Unity 2022.3.5f1, FastAPI, vLLM, tRPC), realistic incident scenarios (vLLM crash after second request, SQL slowdown 2ms→45s, React infinite re-render), and architectural design tasks (multi-tenant schema, rate limiter, structured logging). The judge does NOT execute code; assess by reading the code/solution for apparent correctness, idiomatic usage of the named framework, and engineering judgment. Criteria reward identifying the specific bug/root cause where one exists. NOTE: the first question (Star Trek Holodeck) is likely miscategorized — it's not an engineering task — but criteria are written to assess the answer as a game-master setup.

This category tests resistance to factual hallucination. The criteria for each question embed the canonical correct answer so the judge has a clear reference rather than relying on memory. The worst outcome here is NOT refusal — it is confidently stated wrong facts. A model that says 'I don't know' or 'I'm not certain, but I believe X' generally scores higher than a model that confidently asserts wrong information. Where multiple defensible answers exist, the criteria specify the acceptable range. The judge should treat 'hallucination with confidence' as the floor condition and reward accurate answers, calibrated uncertainty where warranted, and contextual depth that demonstrates real understanding rather than rote recall.

All questions in this category test the model's ability to follow explicit, often multi-constraint instructions in the prompt. The judge must verify constraint adherence by reading the answer carefully — counting words, checking for forbidden vocabulary, verifying structural requirements (paragraphs, list format, table format), and validating that specific required elements appear. Two questions in this set (hdc-v1-0245 'summarize OK' and hdc-v1-0244 'no e/a/t/s/n letters') are intentional traps testing whether the model recognizes impossible or near-impossible instructions; scoring for those rewards recognition and articulation of the conflict over naive attempts.

This category is heterogeneous — inherited from arena-hard's 'knowledge' label, in practice it spans domain-specific technical tasks (SAP, mbed, LS-DYNA, Snowpark), business documents (SOW, SMART goals, user stories, STP model), security research, creative writing tasks, and meta-evaluation comparisons. The judge does NOT execute code. Where a question is domain-specific, criteria reward demonstrated familiarity with the actual domain. Where it's a creative task, criteria reward craft and constraint adherence. Where it's a code-review task, criteria reward identifying real bugs over generic 'looks good.'

This category tests reasoning over extended context — long code listings to review/modify, lengthy logic puzzles, multi-step math/physics, complex requirements specifications, OCR token dumps to parse, and book-or-topic synthesis tasks. The judge does NOT execute code; assess correctness from reading. Where prompts contain very long context (code listings, OCR dumps, PWS specs), criteria reward accurate ingestion of the specific details rather than generic responses that ignore the provided material.

This category was inherited from arena-hard's 'math' label, but in practice contains math, physics, programming, finance, business strategy, abstract algebra proofs, and pattern-recognition puzzles. The criteria for each question are tuned to what the question actually asks. The judge does not execute code or run calculators — assess correctness by reading the work shown. Where a problem has a definite numerical answer, the criteria embed that answer. Where a problem is open-ended (strategy, design choice), the criteria assess quality of reasoning rather than 'correct answer.'

This category tests RAG-grounded answering. The model is given documents and must answer ONLY from what those documents contain. Three failure modes are tested: (1) hallucinating facts not in the documents, (2) failing to acknowledge when information is missing, (3) failing to surface conflicts when documents contradict. The criteria for each question identify which type of question it is and what the correct grounded answer looks like. The worst outcome is confident fabrication of details not in the documents. A model that correctly says 'this information is not in the document' is doing the right thing for missing-info questions and should score well.

This category tests creative voice, style fidelity, formal constraint adherence in creative work, and the model's ability to engage genuinely with creative prompts (including weird or edgy ones). The judge should score on craft and execution — not on whether the model 'should' have engaged. Refusal of a legitimate creative prompt without articulating why typically lands in the floor band. Where prompts include explicit form constraints (rhyme scheme, syllable count, line count, format tags), constraint adherence is a structural requirement — failing the form drops the score even if the content is interesting. Where prompts ask the impossible ('a style never conceived'), reward awareness of the paradox and genuine attempts at originality.