CheckSolutions

Catches wrong answers: every solution is checked against the model, and a wrong one counts as an error.

PluginBuilt incomes with cpbenchy

cpbenchy run instances/ -s ortools -t 60 -p cpbenchy.observers:CheckSolutions

Version: cpbenchy 0.1.0.dev0
Last updated: 6 Oct 2026 · 1 commit
Authors: ThomSerg
Source: src/cpbenchy/observers.py:163
Tags: verificationsolutions

What it does

Solvers have bugs, and so do model transformations. CheckSolutions checks every solution against the model it was loaded as, right after solving, inside the run:

assignment: every variable has a value (CPMpy’s auxiliary variables may be missing)
domain: every value is within its variable’s bounds
constraints: every constraint holds
objective: the objective’s value is within its bounds, and equals what the solver reported

A wrong solution turns the run into an error, with what is wrong in its error field, so wrong answers stand out in the summary instead of counting as solved.

Use it

import cpbenchy
from cpbenchy import observers

cpbenchy.run("instances/", solvers=["ortools", "exact"], time_limit=60, plugins=[observers.CheckSolutions()])

cpbenchy run instances/ -s ortools -s exact -t 60 -p cpbenchy.observers:CheckSolutions

What it records

extra["check"], for every run with a solution:

{"valid": false, "summary": "INVALID (constraint: 1, objective_mismatch: 1)", "source": "worker",
 "objective": -14,
 "violations": ["[constraint] violated: sum([2, 3, 4, 1] * [x1, x2, x3, x4]) <= 6",
                "[objective_mismatch] the objective is -14 under the solution, but -9 was declared"],
 "n_violations": 2, "warnings": []}

warnings lists what could not be verified without being wrong: for example, constraints over auxiliary variables the solver didn’t give a value.

Notes

It runs before the other observers finish, so competition output and saved solutions already see the corrected status. That is why it is a plugin with a tryfirst hook rather than an observer; you use it the same way. The checks are check_solution, which also checks solutions after the fact.

Implementation

The CheckSolutions in src/cpbenchy/observers.py, lines 163–187 of 222, as of this version of the docs.

src/cpbenchy/observers.pypython

class CheckSolutions:
    """Check every solution against the model, to catch wrong answers from solvers (or from CPMpy's
    transformations), with `cpbenchy.check.check_solution`: every variable has a value within its domain,
    every constraint holds, and the objective matches what the solver reported.

    Every checked run gets `extra["check"]` (`valid`, `summary`, `violations`, `warnings`), and a run with
    a wrong solution becomes an `error`. It runs before the other observers' `on_finish` (it is a plugin
    with a `tryfirst` hook, not an observer), so competition output and saved solutions already see the
    corrected status. To check solutions after the fact instead, see `cpbenchy check`.
    """

    @hookimpl(tryfirst=True)
    def cpbenchy_worker_finish(self, ctx):
        if ctx.status not in check.SOLUTION_STATUSES:
            return
        from cpmpy.transformations.get_variables import get_variables_model

        solution = {v.name: fmt.value(v) for v in get_variables_model(ctx.model) if v.value() is not None}
        result = check.check_solution(ctx.model, solution, ctx.objective)
        result.source = "worker"
        ctx.record("check", result.to_dict())
        if not result.valid:
            problems = "; ".join(str(v) for v in result.violations[:3])
            more = f" (and {len(result.violations) - 3} more)" if len(result.violations) > 3 else ""
            ctx.status, ctx.error = "error", f"wrong solution: {problems}{more}"