Okay kittens, I've been meaning to post this for a while. After spending three months running benchmarks on our little analysis workbench I finally have numbers worth sharing. The question: is the Polyhedra domain actually usable in production, or is it forever condemned to "works great on toy programs" land?
Short answer: it depends catastrophically on your variable count. Long answer: read on.
Background
We compared three domains across a suite of 20 numerical benchmark programs (mix of loop-heavy C programs, control-flow-intensive kernels, and some embedded-style code). All tests run with APRON as the backend, using PPL for polyhedra, ELINA for octagons, and custom intervals. Widening enabled on all domains, Cousot-Halbwachs standard widening for polyhedra. Machine: AMD Ryzen 9 7950X, 64GB RAM. Each test has a 4-hour timeout and 16GB memory cap.
Benchmark Suite Overview
| # | Program | LOC | #Vars (max scope) | Loop depth | Category |
|---|---|---|---|---|---|
| B01 | matrix_mult_3x3 | 42 | 9 | 3 | Linear Algebra |
| B02 | bubble_sort_n10 | 28 | 4 | 2 | Sorting |
| B03 | pid_controller | 67 | 12 | 1 | Control |
| B04 | fir_filter_8tap | 55 | 14 | 2 | DSP |
| B05 | nested_sum_4d | 31 | 6 | 4 | Arithmetic |
| B06 | jacobi_iter_5x5 | 89 | 30 | 3 | Numerical |
| B07 | lzw_compress | 134 | 18 | 2 | Compression |
| B08 | crc32_byte | 48 | 8 | 1 | Crypto/Hash |
| B09 | gaussian_elim_6x6 | 101 | 40 | 3 | Linear Algebra |
| B10 | trajectory_sim | 78 | 22 | 2 | Physics Sim |
Results β Analysis Time
| Bench | Interval (ms) | Octagon (ms) | Polyhedra (ms) | Polyhedral Invariants Found | |
|---|---|---|---|---|---|
| B01 | 4 | 31 | 820 | 7 non-trivial | |
| B02 | 2 | 8 | 55 | 3 non-trivial | |
| B03 | 6 | 190 | 14,300 | 11 non-trivial | |
| B04 | 5 | 240 | 38,100 | 9 non-trivial | |
| B05 | 3 | 22 | 430 | 5 non-trivial | |
| B06 | 11 | 3,800 | TIMEOUT | N/A | |
| B07 | 18 | 2,100 | TIMEOUT | N/A | |
| B08 | 3 | 14 | 70 | 2 non-trivial | |
| B09 | 17 | TIMEOUT | OOM | N/A | |
| B10 | 8 | 1,900 | 92,400 | 14 non-trivial | |
| TIMEOUT = exceeded 4-hour wall clock limit. OOM = exceeded 16GB memory cap. All times are mean of 5 runs. | |||||
Precision Comparison (false alarm rate on assert-checking task, lower = better)
| Bench | Interval false% | Octagon false% | Polyhedra false% |
|---|---|---|---|
| B01 | 44% | 17% | 0% |
| B02 | 22% | 8% | 0% |
| B03 | 61% | 29% | 3% |
| B04 | 55% | 22% | 5% |
| B05 | 18% | 5% | 0% |
| B08 | 30% | 11% | 0% |
| B10 | 48% | 20% | 4% |
| Only benchmarks that completed for all three domains shown. Polyhedra precision is clearly superior when it terminates. | |||
The precision numbers are incredible when polyhedra actually finishes. But look at B06, B07, B09. The moment variable count exceeds ~18-20 in a relational context, we're toast. This is the fundamental problem with convex polyhedra: the double description requires managing both constraint and generator representations, and conversion between them via Chernikova's algorithm can blow up doubly-exponentially in the number of variables.
My question to the forum: is there any realistic path to making polyhedra practical for programs with 20+ relational variables, or should we accept octagons as our ceiling?