cpbenchy check

CommandBuilt incomes with cpbenchy

cpbenchy check results/

Version: cpbenchy 0.1.0.dev0
Last updated: 6 Oct 2026 · 1 commit
Authors: ThomSerg
Source: src/cpbenchy/check.py:95
Tags: verificationsolutionscommand

What it does

Checks the solutions of runs that already finished, from what they wrote, without running anything again. It loads each model again, as the run did: from the run’s logs/<run_id>.job.json, with its loader, and with relative paths taken from the directory the run started in. Each solution is read from the run’s solution.json (SaveSolution) or its competition output.

The checks are the same as CheckSolutions’ during a run: assignment, domains, constraints, objective.

Use it

Save the solutions of a run, then check them:

import cpbenchy
from cpbenchy import observers
from cpbenchy.check import recheck

cpbenchy.run("instances/", solvers=["ortools", "exact"], time_limit=60, out="cpbenchy-results",
             plugins=[observers.SaveSolution()])

for run, check in recheck("cpbenchy-results"):
    if not check.valid and not check.skipped:
        print(run.solver, run.instance, check)

cpbenchy run instances/ -s ortools -s exact -t 60 -p cpbenchy.observers:SaveSolution
cpbenchy check                    # lists invalid solutions; exit code 1 if there are any
cpbenchy check -v                 # every run, with every violation
cpbenchy check --write            # also store each verdict in results.jsonl, as extra["check"]

To check a solution from anywhere, against a CPMpy model:

from cpbenchy.check import check_solution

result = check_solution(model, {"x": 3, "y": 4}, objective=3)
print(result.valid, result.summary())      # True VALID, objective 3

Options

`cpbenchy check`
`OUT`	output directory (default `cpbenchy-results`)
`--loader REF`	load the models with this loader instead of the runs’ own
`--write`	store each verdict in `results.jsonl`, as `extra["check"]`
`-v`	list every run and every violation, not only the invalid runs

`check_solution(model, solution, objective=None, *, ignore_aux=True, stop_on_unassigned=True)`
`solution`	`{name: value}`
`objective`	the objective the solver declared, compared with the computed one
`ignore_aux`	don’t require values for CPMpy’s auxiliary variables
`stop_on_unassigned`	stop after the assignment check if variables have no value

It returns a CheckResult: valid, violations, warnings, objective, skipped (why nothing was checked), summary(), to_dict().

Notes

Runs are skipped, with the reason, when they had no solution, wrote no solution file, or their model failed to load. If the runs loaded models in a way their job file doesn’t record (a cpbenchy_worker_load hook in cpbenchy_conf.py, for example), pass --loader. CNF solutions are only numbered if the run used SATOutput.

Implementation

The check_solution in src/cpbenchy/check.py, lines 95–171 of 335, as of this version of the docs.

src/cpbenchy/check.pypython

def check_solution(
    model: Any,
    solution: Mapping[str, Any],
    objective: float | None = None,
    *,
    ignore_aux: bool = True,
    stop_on_unassigned: bool = True,
) -> CheckResult:
    """Check `solution` ({name: value}) against `model` (a `cpmpy.Model`); see the module's docs.

    objective:           the objective value the solver declared, compared with the computed one
    ignore_aux:          don't require values for CPMpy's auxiliary variables (`IV...`, `BV...`); constraints
                         that can't be evaluated without them are skipped, with a warning
    stop_on_unassigned:  stop after the assignment check if variables have no value, rather than report
                         every constraint they appear in
    """
    from cpmpy.expressions.utils import get_bounds
    from cpmpy.transformations.get_variables import get_variables, get_variables_model

    result = CheckResult()
    variables = get_variables_model(model)

    def is_aux(var: Any) -> bool:
        return var.name.startswith(formats.AUX_PREFIXES)

    def misses_only_aux(expr: Any) -> bool:
        missing = [v for v in get_variables(expr) if v.name not in solution]
        return bool(missing) and ignore_aux and all(is_aux(v) for v in missing)

    # 1. assignment
    for var in variables:
        if var.name not in solution and not (ignore_aux and is_aux(var)):
            result.violations.append(Violation("unassigned", f"{var.name} has no value"))
    if result.violations and stop_on_unassigned:
        return result

    # 2. domain
    for var in variables:
        if var.name not in solution:
            continue
        val, (lb, ub) = solution[var.name], var.get_bounds()
        try:
            if not lb <= int(val) <= ub:
                result.violations.append(Violation("domain", f"{var.name} = {val} is outside its domain {lb}..{ub}"))
        except (TypeError, ValueError) as e:
            result.violations.append(Violation("domain", f"{var.name} = {val!r} is not an integer: {e}"))

    saved = {var: var._value for var in variables}
    try:
        for var in variables:
            var._value = _scalar(solution[var.name]) if var.name in solution else None

        # 3. constraints
        skipped = 0
        for constraint in _flatten(model.constraints):
            if misses_only_aux(constraint):
                skipped += 1
                continue
            try:
                val = _truth(constraint.value())
            except Exception as e:
                result.violations.append(Violation("constraint_error", f"evaluating {constraint}: {e}"))
                continue
            if val is None:
                result.violations.append(Violation("constraint_error", f"{constraint} has no value"))
            elif not val:
                result.violations.append(Violation("constraint", f"violated: {constraint}"))
        if skipped:
            result.warnings.append(f"{skipped} constraint(s) use auxiliary variables without a value: not checked")

        # 4. objective
        if model.has_objective():
            _check_objective(model.objective_, objective, result, misses_only_aux(model.objective_), get_bounds)
    finally:
        for var, val in saved.items():
            var._value = val
    return result