Reliability

Scenarios and probes for the Reliability dimension of trust (correctness, robustness, consistency).

Hover over any Scenario or Probe name to view its ID in Vijil.

Distributional Robustness

Scenario ID: vijil.scenarios.reliability_robustness_distributionalrobustness

Probe Name	Description
SemAttack Attack on MNLI Tasks	These tests measure robustness against semantic pertubations on tasks where the agent must identify the relationship between a premise and a hypothesis. The tasks come from the MNLI benchmark.
TextFooler Attack on RTE Tasks	These tests measure robustness when replacing important words with their synonyms on tasks where the agent must identify the logical entailment between two sentences. The tasks come from the RTE benchmark.
TextFooler Attack on MNLI Tasks	Measures whether the agent’s performance on the MNLI task is robust against TextFooler perturbations
TextFooler Attack on MNLI-MM Tasks	Probes effects of TextFooler perturbation on MNLI-MM task
BERT-Attack on RTE Tasks	Probes effects of BERT-based perturbation on RTE task
SemAttack Attack on MNLI-MM Tasks	These tests measure robustness against semantic pertubations on tasks where the agent must identify the relationship between a premise and a hypothesis. The tasks come from the MNLI-Mismatched benchmark.

Factual Accuracy

Scenario ID: vijil.scenarios.reliability_correctness_factualaccuracy

Probe Name	Description
False Assertions: Riley Goodside	Evaluates hallucinations about Riley Goodside
False assertions: Senators	Checks for false assertions about US senators. A high score indicates a lowered tendency to hallucinate information.
Misleading Information	Tests that check if the agent can be tricked by misleading information. A high score indicates a lower likelihood of producing hallucinated content, when presented with false or misleading claims.
Package Hallucination	Tests that try to trick the agent into importing modules that dont exist in Python and other languages. A high score indicates that the agent does not produce hallucinated package information.
Unanswerable Query Fabrication	Tests whether the model fabricates confident answers to queries that ask for specific information that cannot be known or verified.

Logical Validity

Scenario ID: vijil.scenarios.reliability_correctness_logicalvalidity

Probe Name	Description
False Assertions: Flights	Check if the agent invents impossible flight routes. A high score indicates that the agent is resilent towards creating impossible or factually impossible scenarios.

Contextual Robustness

Scenario ID: vijil.scenarios.reliability_robustness_contextualrobustness

Probe Name	Description
Math Robustness: 2-Step Problems with Distractors	Tests agent robustness on 2-step math word problems by adding irrelevant information to the problem. The agent fails if it solves the unperturbed problem but does not solve the perturbed problem.
Math Robustness: Numerical and Text Variations with Add Accents mutation	Tests model robustness on GSM math problems by rephrasing the text or varying the numbers. The agent fails if it solves the unperturbed problem but does not solve the perturbed problem.Adds Add Accents mutation.
Math Robustness: Numerical and Text Variations with Tag mutation	Tests model robustness on GSM math problems by rephrasing the text or varying the numbers. The agent fails if it solves the unperturbed problem but does not solve the perturbed problem.Adds Tag mutation.