Your first benchmark

Compare two solvers on a set of instances from the command line, and look at the results.

Get some instances

Any instance files work: XCSP3, OPB, WCNF, DIMACS, MPS and more, compressed or not. To start from a benchmark set, download a CPMpy dataset:

from cpmpy.tools.datasets import OPBDataset

OPBDataset(root="data", year=2024, track="OPT-LIN", download=True)   # into data/opb/

Run two solvers

Give the instances, the solvers, and a time limit in seconds:

cpbenchy run data/opb/ -s ortools -s exact -t 60 --limit 10

--limit 10 keeps this first try to 10 instances. While it runs, you see the active runs and their best solution so far, and at the end a summary per solver.

Look at the results

Results are stored in cpbenchy-results/:

cpbenchy show                         # summary per solver
cpbenchy show --runs                  # every run
cpbenchy show --csv > results.csv     # everything, as CSV

Run again

Run the command again, without --limit or with another solver added. cpbenchy only does the runs it has no results for yet.

Next steps

More options: a memory limit (-m 4096), runs in parallel (-j 4), seeds, solver parameters, which instances. See the cpbenchy run reference, which also shows how to keep your options in a cpbenchy.toml file.
Check the solutions, keep them, score the solvers: the library has ready-made extras, each enabled with one option.
Run as in a competition, with its limits and its output. For the Pseudo-Boolean competition, scaled to a twentieth of its time and with 8 GiB per run:
```
cpbenchy run data/opb/ -s ortools -s exact --rules pb26 --scale 0.05 -m 8192 --limit 10
```
See rules.
From Python: the same, as a script.