Examples | cpbenchy

Small, complete files to copy and adapt: plugins, loaders and scripts that show how to do common things with cpbenchy.

The repository’s examples/ directory has small, complete examples: plugins, loaders and scripts. Unlike the library, they don’t come with cpbenchy. Each is one file, meant to be copied and changed for your own experiments, and each shows how to do something you will want to do your own way, such as scoring, storing or analysing. The test suite runs all of them, so they work as shown.

Plugins

Score, store or follow runs; enable one with -p.

SqliteStore

Also keeps results in a database, to collect many experiments in one place.

Shows: a plugin with its own option, a plugin object with state, and trylast

Plugin

Loaders

Benchmark problems in a format of your own.

KnapsackJSON

Benchmarks problems stored in your own file format, here knapsacks written as JSON.

Shows: a Loader for a file format of your own

Loader

Scripts

Whole experiments and analyses, from Python.

param_sweep.py

Finds the best settings for a solver by trying several on the same instances.

Shows: an Experiment with several settings, and results as they come in

Script

formulations.py

Compares different ways of modelling the same problem, without any instance files.

Shows: generated instances, and a Loader that builds the model in the worker

Script

analyze.py

Compares solvers on finished results: scores, the best of all solvers, and a plot.

Shows: results as a pandas DataFrame: scores, the virtual best solver, a cactus plot

Script

Using an example

Use an example from a clone of the repository, as it is:

import cpbenchy

cpbenchy.run("instances/", solvers=["ortools"], time_limit=60, plugins=["examples/plugins/sqlite_store.py"])

cpbenchy run instances/ -s ortools -t 60 -p examples/plugins/sqlite_store.py

Or copy the file into your own project, change it, and pass your copy instead. For how plugins and loaders work, see writing an observer and writing a loader.

The library’s own observers are worth reading too: src/cpbenchy/observers.py shows live output from the worker (on_solution), final output (on_finish), files per run (ctx.artifact), a check in the parent after each run (on_result), and changing a run’s status (CheckSolutions).