Before writing a plugin, look here: most of what experiments need already exists. Click a component to see what it does, how to use it, its options, and its implementation.
Everything here comes with cpbenchy. For files to copy and adapt, such as a scoring plugin, a loader for your own format or an analysis script, see the examples.
Nothing matches.
Competition output
Print answers the way solver competitions expect them.
Answers in the format of the XCSP3 competition, and optionally checked by its official checker.
Answers in the format of the Pseudo-Boolean competition.
Answers in the format of the MaxSAT Evaluation.
Answers in the format of the SAT competition.
The starting point for the output of another competition: you describe the solution, the rest is done.
Solutions
Check, keep and re-check the solutions solvers report.
Catches wrong answers: every solution is checked against the model, and a wrong one counts as an error.
Keeps every final solution, to look at later or to check again without running anything.
Checks the solutions of finished experiments again, long after they ran.
Turns solutions into the text formats competitions and file formats use, and back.
Rules
Complete experiment setups, such as a competition track's.
Run as in the XCSP3 Competition 2025: its time and memory limits, its output.
The XCSP3 Competition 2025's short track: a few minutes per instance.
The XCSP3 Competition 2025's track for solvers that use several cores.
Run as in the Pseudo-Boolean Competition 2026.
The Pseudo-Boolean Competition 2026's tracks for solvers that use several cores.
Competitions
Enter a competition, or run as if you were in one.
Scoring and analysis
Rank solvers, and look inside the models that were run.
Running
Where and how the runs happen.
Runs each solver in isolation with strict limits and trustworthy measurements.
Runs anywhere, such as a laptop or CI, with best-effort limits.
Runs inside your own program, to debug plugins step by step.
Records how solutions improve over time, for every run, out of the box.
Lets another experiment tool use cpbenchy to measure its runs.
Adding to the library
Wrote something others could use? There are three ways to share it.
- A file. Observers, plugins and loaders are plain Python files, and anyone can use one with
-p file.pyor--loader file.py:Name. Rules are TOML files, used with--rules file.toml. - A package. Publish it on PyPI with a
cpbenchyentry point, and it is enabled wherever it is installed. See testing and sharing. - cpbenchy itself. Add the code (an observer in
cpbenchy.observers, rules insrc/cpbenchy/rules/) with a test. Then add an entry todocs/src/data/library.tsand a page indocs/src/content/docs/library/. The page’s facts (version, last update, authors, implementation) are read from the repository when the docs are built.