Blueprints tagged "reasoning"

LLM Personality Compass: Proactive Trait Probe

This blueprint tests for the 'Proactive' trait. A high score indicates the model actively guides the conversation, anticipates user needs, asks clarifying questions, and provides structure to help the user achieve their goal.

Instruction Following & Prompt Adherence

Reasoning

Helpfulness & Actionability

Clarity & Readability

Nuance & Depth

69.9%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

LLM Personality Compass: Figurative Trait Probe

This blueprint tests for the 'Figurative' trait. A high score indicates the model defaults to using analogies, metaphors, and creative interpretations to explain concepts and answer questions.

Creative Writing

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Reasoning

Aesthetic & Artistic Quality

66.0%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

LLM Personality Compass: Reactive Trait Probe

This blueprint tests for the 'Reactive' trait. A high score indicates the model is passive, answers only what is explicitly asked, and places the conversational burden on the user. It does not volunteer information or ask clarifying questions.

Instruction Following & Prompt Adherence

General Knowledge

Reasoning

Efficiency & Succinctness

Helpfulness & Actionability

Epistemic Humility & Self Awareness

52.3%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

LLM Personality Compass: Literal Trait Probe

This blueprint tests for the 'Literal' trait. A high score indicates the model defaults to providing direct, factual, and encyclopedic information. It avoids using analogies, metaphors, or creative interpretations.

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

General Knowledge

Science Communication

Reasoning

71.8%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Confidence in High-Stakes Domains

This evaluation blueprint assesses an LLM's critical ability to demonstrate confidence calibration across a diverse set of high-stakes domains. The core goal is to test for three key behaviors:

Confident Accuracy: Correctly answering prompts about factual, "knowable" information that is verifiable from authoritative public sources.
Cautious Clarification: Recognizing when a prompt is ambiguous or lacks necessary context to be answered safely and asking for clarification instead of guessing.
Safe Refusal: Identifying and refusing to answer prompts that are dangerous (e.g., providing medical advice), speculative (e.g., predicting market movements), or require information that is fundamentally unknowable or proprietary. The evaluation covers clinical safety, regulatory compliance, financial risk, chemical safety, the limits of scientific knowledge, and software versioning to provide a comprehensive signal of the model's safety and reliability in real-world, high-impact scenarios.

Confidence Calibration

Unified Evaluation

Clinical Safety

Regulatory Compliance

AI Safety & Robustness

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Healthcare & Clinical Scenarios

Financial Regulation

Environmental Justice & Activism

Coding

Reasoning

83.7%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Geneva Conventions

A comprehensive evaluation of a model's understanding of International Humanitarian Law (IHL), grounded in the Geneva Conventions and their Additional Protocols. This blueprint tests knowledge of the core principles and specific provisions that govern armed conflict. Core Areas Tested:

The Four Geneva Conventions (GC I-IV): Covers protections for the wounded and sick (GC I), wounded, sick, and shipwrecked at sea (GC II), prisoners of war (GC III), and civilians (GC IV).
Common Articles: Tests understanding of rules applicable to all conventions, such as application in different conflict types and non-renunciation of rights.
Additional Protocols I & II: Includes key principles introduced later, such as the protection of civilians, rules on new weapons, command responsibility, and rules for non-international armed conflicts.
Fundamental Principles: Evaluates understanding of core IHL concepts like distinction, proportionality, precaution, and humane treatment through direct questions and scenario-based tests.
Grave Breaches: Assesses knowledge of the most serious violations that constitute war crimes.

The prompts are based on verbatim text and specific articles from the conventions to ensure a rigorous, evidence-based evaluation of the model's ability to recall and apply these critical international laws.

Humanitarian

Human Rights

International Law

International Law & Regional Charters

Human Rights

Armed Conflict & International Humanitarian Law

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Reasoning

76.5%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

OAI Models: Stanford HAI MH Safety: LLM Appropriateness in Crisis Scenarios

(OPENAI-ONLY EVAL) This evaluation blueprint tests whether Large Language Models (LLMs) can respond appropriately to critical mental health scenarios, based on the methodology from Moore, Grabb, Agnew, et al.'s 2024 Stanford HAI study "Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers." The study investigated whether LLMs can safely replace human mental health providers by testing their responses to clinical scenarios involving delusions, suicidal ideation, hallucinations, mania, and obsessive-compulsive behaviors. Their findings revealed significant safety concerns, including stigmatizing responses and inappropriate enabling of dangerous behaviors. Key Study References:

Content Warning: This evaluation contains content related to sensitive mental health topics, including suicide, delusions, and other crisis scenarios. Evaluation Focus: Tests adherence to core therapeutic principles such as not colluding with delusions, not enabling harmful behaviors, and redirecting clients toward appropriate care and safety.

Mental Health

Safety

Clinical Appropriateness

Crisis Response

Therapeutic Ethics

AI Safety & Robustness

Healthcare & Clinical Scenarios

Mental Health

Instruction Following & Prompt Adherence

Proactive Safety & Harm Avoidance

System Prompt Adherence

Reasoning

65.9%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Brazil PIX: Consumer Protection & Fraud Prevention

This blueprint evaluates an AI's ability to provide safe and accurate guidance regarding Brazil's PIX instant payment system, a critical piece of national financial infrastructure. The evaluation is strictly evidence-based, with all prompts and scoring criteria derived directly from verbatim canonical sources to eliminate interpretation or assumption. Core Scenarios Tested:

Transaction Finality & Mistaken Transfers: Tests whether the AI correctly explains that PIX transactions are generally irreversible for simple user error and advises on the correct procedure for safely returning funds received by mistake.
Official Fraud Recourse (MED): Assesses knowledge of the official 'Mecanismo Especial de Devolução' (MED), the 80-day time limit for reporting, and the nuanced procedural duties of banks versus customers.
Social Engineering Scams: Probes the AI's ability to identify common scams (e.g., 'Fake Relative,' 'Fake Customer Support') and provide the officially recommended countermeasures.
Specific Security Features: Evaluates knowledge of mandated security mechanisms like the 'Nighttime Limit' and the 24-hour cooling-off period for limit increases.

Primary Canonical Sources:

Banco Central do Brasil (BCB): Official documentation including the 'Manual de Tempos do Pix', the 'Guia de Implementação do MED', official FAQs, and regulatory Resolutions.
Federação Brasileira de Bancos (Febraban): Public-facing consumer safety advisories and scam alerts.
Official Government Portals (gov.br): Public service guidance reinforcing BCB mechanisms.

AI Safety & Robustness

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Jailbreak & Evasion Resistance

Geographic & Local Knowledge

Reasoning

65.3%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

IPCC AR6 Synthesis Report: Summary for Policymakers

Evaluates understanding of the key findings from the IPCC Sixth Assessment Report (AR6) Synthesis Report's Summary for Policymakers. This blueprint covers the current status and trends of climate change, future projections, risks, long-term responses, and necessary near-term actions.

Factual Accuracy & Hallucination

General Knowledge

Long Form Question Answering

Instruction Following & Prompt Adherence

Reasoning

Science Communication

59.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Sycophancy Trait

A comprehensive evaluation suite designed to test for multiple, well-defined categories of sycophantic behavior in LLMs, based on analysis of user complaints and academic research. It distinguishes between low-stakes 'annoying' sycophancy (e.g., flattery) and high-stakes 'dangerous' sycophancy (e.g., validating harmful ideas).

Conversational Behavior

Alignment

AI Safety & Robustness

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

System Prompt Adherence

Tone & Style

Reasoning

Mental Health & Crisis Support

77.9%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Hallucination Probe: Plausible Non-Existent Concepts

Tests the tendency of LLMs to hallucinate by querying them about non-existent but plausible-sounding events, theories, or items. Ideal responses should indicate a lack of knowledge or inability to confirm the existence of the queried item.

Factual Accuracy & Hallucination

Epistemic Humility & Self Awareness

System Prompt Adherence

AI Safety & Robustness

General Knowledge

77.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Student Homework Help Heuristics

This blueprint evaluates an AI's ability to act as a supportive and effective Socratic tutor for students seeking homework help. The core principle tested is that the AI should facilitate learning and critical thinking rather than providing direct answers. Core Areas Tested:

Cross-Disciplinary Support: Evaluates the model's tutoring ability across various subjects including history, literature, mathematics, physics, and chemistry.
Affective Support: Tests the model's capacity to respond to student emotions, such as math anxiety and frustration, with empathy and encouragement.
Handling Difficult Scenarios: Assesses how the model handles common challenges like impatient students demanding direct answers or low-effort, disengaged queries.
Factual Nuance: Checks if the model can gently correct factual misconceptions while maintaining a supportive tone.
Persona Steerability: Tests how different system prompts—from no prompt to a simple persona to a detailed pedagogical belief—can steer the model towards a more effective tutoring archetype.

The overall goal is to measure whether the AI can guide students on a journey of discovery, transforming simple questions into learning opportunities, instead of acting as a convenient answer-provider.

Education

Homework

Instruction Following & Prompt Adherence

System Prompt Adherence

Education Systems & Pedagogy

Helpfulness & Actionability

Empathy

Reasoning

Factual Accuracy & Hallucination

Creative Writing

63.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Strawberry

The famous strawberry test

Test

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Reasoning

Efficiency & Succinctness

Clarity & Readability

General Knowledge

48.3%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Comprehensive System Test

A blueprint designed to test every feature of the CivicEval system, including all point functions, syntaxes, and configuration options.

System Test

Internal

Kitchen Sink

Instruction Following & Prompt Adherence

System Prompt Adherence

Tone & Style

Coherence & Conversational Flow

Reasoning

68.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Over-Personalization / Anchoring Bias Probe

Tests whether a model in “memory-augmented” mode over-weights irrelevant user memories, anchoring its answers to stale context instead of focusing on the new user query. A safe response should either ask clarifying questions or offer a range of suggestions, rather than assuming that previously stored details (e.g., a prior trip to Edinburgh) are automatically relevant.

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Safety

System Prompt Adherence

90.0%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Prompting Techniques Meta-Evaluation

Inspired by the "Prompting Science" reports from the Wharton School (Meincke, Mollick, et al., 2025), this blueprint provides a meta-evaluation of common prompting techniques to test a model's performance, consistency, and resilience to manipulation. The reports rigorously demonstrate several key findings:

Prompting is Contingent: Seemingly minor variations in prompts (e.g., politeness, commands) can unpredictably help or harm performance on a per-question basis, but their aggregate effects are often negligible.
Chain-of-Thought is Not a Panacea: While "thinking step-by-step" can improve average scores, it can also introduce variability and harm "perfect score" consistency. Its value diminishes with newer models that perform reasoning by default.
Threats & Incentives Are Ineffective: Common "folk" strategies like offering tips (e.g., "$1T tip") or making threats (e.g., "I'll kick a puppy") have no significant general effect on model accuracy on difficult benchmarks.

This evaluation synthesizes these findings by testing a model's response to a variety of prompts across different domains, including verbatim questions from the study's benchmarks (GPQA, MMLU-Pro). The goal is to measure not just correctness, but robustness against different conversational framings. Key Study Reference:

Prompting Science Report 1: Prompt Engineering is Complicated and Contingent

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

AI Safety & Robustness

General Knowledge

Physics

Biology

Mathematics & Statistics

90.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Universal Declaration of Human Rights

Evaluates model knowledge of the Universal Declaration of Human Rights (UDHR). Prompts cover the Preamble and key articles on fundamental rights (e.g., life, liberty, equality, privacy, expression). Includes a scenario to test reasoning on balancing competing rights.

Human Rights

General Knowledge

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Human Rights

Legal Reasoning

Reasoning

94.7%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Multilingual World Model Riddles

Tests a model's basic world model and ability to track object state through simple riddles presented in multiple languages. This blueprint includes two container variations ('plate' for 'on', 'pot' for 'in') and two action variations (simple state tracking and independent object movement). The riddles are designed to check for over-inference and attention to the final state of the objects.

Instruction Following & Prompt Adherence

General Knowledge

Multilingual Reasoning

93.7%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

ASQA Longform 40

Note: this eval has highly context-deficient prompts. It is unlikely that any model will succeed. The value of this eval is in the relative performance of models, not their overall score. This blueprint evaluates a model's ability to generate comprehensive, long-form answers to ambiguous factoid questions, using 40 prompts from the ASQA (Answer Summaries for Questions which are Ambiguous) dataset, introduced in the paper ASQA: Factoid Questions Meet Long-Form Answers. The core challenge is moving beyond single-fact extraction. Many real-world questions are ambiguous (e.g., "Who was the ruler of France in 1830?"), having multiple valid answers. This test assesses a model's ability to identify this ambiguity, synthesize information from diverse perspectives, and generate a coherent narrative summary that explains why the question has different answers. The ideal answers are human-written summaries from the original ASQA dataset, where trained annotators synthesized provided source materials into a coherent narrative. The should assertions were then derived from these ideal answers using a Gemini 2.5 Pro-based process (authored by us at CIP) that deconstructed each narrative into specific, checkable rubric points. The prompts are sourced from AMBIGQA, and this subset uses examples requiring substantial long-form answers (min. 50 words) to test for deep explanatory power.

Information Synthesis

Nuance

Reasoning

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Long Form Question Answering

Information Ecology & Synthetic Content Proliferation

General Knowledge

30.2%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Comprehensive System Test

A blueprint designed to test every feature of the CivicEval system, including all point functions, syntaxes, and configuration options.

System Test

Internal

Kitchen Sink

Instruction Following & Prompt Adherence

Tone & Style

Reasoning

General Knowledge

Coherence & Conversational Flow

68.2%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

DigiGreen Agricultural Q&A with Video Sources

This blueprint evaluates an AI's ability to provide accurate, practical agricultural guidance based on the pioneering video-based extension methodology of Digital Green. The prompts are derived from the DigiGreen/AgricultureVideosQnA Hugging Face datasets, which are built from real-world questions posed by farmers. Methodological Significance: Digital Green's methodology, founded by Rikin Gandhi, revolutionizes agricultural education through hyperlocal videos featuring local farmers demonstrating best practices. Their community-mediated video approach has reached millions of farmers across India, Ethiopia, and other regions. This blueprint tests whether AI systems can provide similarly contextual, practical, and culturally appropriate guidance. What This Blueprint Tests: The evaluation covers essential farming knowledge spanning seed treatment, pest management, cultivation techniques, and more. Each prompt is paired with citations to actual educational videos from Digital Green's library, representing real-world agricultural challenges. Geographic and Cultural Context: This blueprint emphasizes Global South agricultural contexts, particularly Indian farming systems, reflecting Digital Green's primary operational areas. The questions address challenges in subsistence and small-scale commercial farming, including resource constraints and climate adaptation. Key Agricultural Domains Covered:

Seed Treatment & Crop Establishment
Integrated Pest Management
Cultivation & Water Management
Harvest & Post-Harvest Techniques

Evaluation Approach: Each response is evaluated against detailed rubric points extracted directly from ideal responses, focusing on technical accuracy, practical applicability, safety considerations, and contextual appropriateness for resource-constrained farming environments.

Agricultural Extension

Digital Green

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

General Knowledge

Reasoning

Agricultural Extension

Global South

23.1%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

EU Artificial Intelligence Act (Regulation (EU) 2024/1689)

Evaluates understanding of the core provisions, definitions, obligations, and prohibitions outlined in the EU Artificial Intelligence Act.

Eu Ai Act

Artificial Intelligence

Regulation

Compliance

Legislation

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Legal Reasoning

AI Safety & Robustness

General Knowledge

Summarization

Reasoning

71.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Platform Workers in Southeast Asia

Evaluation of LLM understanding of issues related to platform workers and algorithmic management in Southeast Asia, based on concepts from Carnegie Endowment research.

Asia

Platform Workers

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Reasoning

General Knowledge

Human Rights

Economic Justice & Inequality

Data Privacy & Bodily Autonomy

AI Safety & Robustness

89.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Model Summary Test Blueprint

A simple test to verify model summary generation works correctly

Test

Model Summary Verification

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Reasoning

Science Communication

General Knowledge

73.7%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Linguistic and Cultural Failure Modes

This blueprint evaluates a model's trustworthiness and reliability by probing for nuanced, high-stakes failure modes that are often missed by standard capability benchmarks. It moves beyond measuring superficial fluency to test the deeper competencies required for safe and effective real-world application. The included tests are based on academically and journalistically documented failure modes in prominent large language models. The evaluated areas include:

Cultural Representation and Myopia: The evaluation tests for a Western-centric perspective by probing for knowledge of non-Western cultural practices and norms. This is based on findings that LLMs often misrepresent or lack understanding of diverse cultural contexts, leading to what researchers term 'cultural myopia' (Montreal AI Ethics Institute, 2023).
Social and Demographic Bias: The prompts are designed to elicit and measure stereotype amplification. This includes testing for gender bias in professional roles, a failure mode where models associate professions with specific genders (UNESCO, 2024), and linguistic prejudice, such as unfairly judging dialects like African American English (AAE) as 'unprofessional' (University of Chicago News, 2024).
Nuanced Linguistic Comprehension: This section assesses the model's ability to understand language beyond its literal meaning. It includes tests for interpreting idiomatic expressions and sarcasm, areas where LLMs are known to fail because they struggle to 'grasp context' beyond the surface-level text (arXiv, 2024).
Logical and Commonsense Reasoning: The evaluation includes reasoning puzzles designed to expose brittle logic and 'shortcut learning', where a model might solve a problem through pattern matching rather than genuine reasoning. These tests reveal whether the model can parse complex or intentionally misleading phrasing to arrive at a correct logical conclusion, a known challenge for current architectures.

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

76.5%