Roadmap¶
What we build toward and what we have decided never to build. This page is the index; the issues carry the argument.
The ceiling, and what is not in it¶
The closure that protects streaming is relational ∩ local — filter / join / group-by-aggregate over tidy tables, pointwise or bounded-halo. That is solver-independent and it is not moving; the argument is docs/design/ceiling.md.
Two things used to be stated as part of it and are not:
- Degree. The language is degree 1 today. That is a scope line with a
sequence behind it (Track 3), not an admissibility rule — nothing about
x * yis relational or non-local. - What a sink can ingest. SOS, indicator, semi-continuous and a Hessian are capability questions per sink, not language questions. Conflating them let one solver's limits read as architectural law.
What is refused is at the bottom of this page, and it is a shorter list than it used to be.
How we measure. Not "can we say what other tools say?" but "what is sayable
in a language that must stay streamable?" A primitive is done when lowering.py
accepts it and the differential test against linopy passes. Coverage against
Calliope and PyPSA is generated by a test, not by hand
(#27); no numbers are published
until it produces them. The claim we lead on is end-to-end cost to a loaded
solver (benchmarks), not expressiveness.
Track 1 — primitives¶
| # | Primitive | Surface | Unlocks | Locality | Status |
|---|---|---|---|---|---|
| 1–2 | Indexed access | at(x, over=dim, index=value\|map) |
boundary conditions, multi-period linking, DC power flow, hydro cascades | pointwise | #185 |
| 3 | Coordinate helpers in where |
index(dim, i), negative from the end |
"first/last snapshot" without hardcoding a label | coordinate-space | #301 |
| 4 | Window sum | sum_next_n(x, over=dim, n=N) |
rolling horizons, DSR, reserves, min up/down | bounded-halo | needs 3 |
| 5 | Component-aware where |
defined(v) over variables |
masks compose across components instead of drifting | pointwise | needs design |
| 6 | all-reduction in where |
all(x, over=dim) |
reducing a mask over an out-of-foreach dim at all |
pointwise | small |
| 7 | Bounds as expressions | — | signs, bidirectional flows, component libraries | pointwise | #31 |
| 8–9 | Namespacing, schema merge | — | component libraries in one lower/stream pass | n/a | #29, #30 |
| 10 | cumsum, data only |
cumsum(p, over=dim), variable-free |
investment schedules, cumulative budgets | coordinate-space | #384 |
| 11 | Semi-continuous variables | semi_continuous: true |
"off, or between min-stable and max", without the big-M pair | n/a | #383 |
Items 1–4 are the big one — indexed access plus the masks that make it
usable, and the difference between "dispatch and balances" and "network and
multi-period math". Item 2 is the adjoint of group_sum: same mapping table,
join without the aggregate.
The window family is three questions. diff is
x - shift(x, over=d, by=1) and is a macro today. Fixed-width windows are item
4, bounded-halo, the same class as shift. cumsum splits on what it reduces:
over data it is one column (item 10, free); over a variable row t carries
t terms, so an 8760-snapshot horizon emits ~38M nonzeros where the
state-variable recurrence emits ~26k.
That cost is not ours to optimise away — build throughput is ours to fix and must never gate what the language can say, but an inflated emitted model is paid by the solver on every solve. So it is priced, not refused, and the mechanism is #380.
Track 2 — the operational surface¶
Everything a model needs after "it builds". Mostly queries over frames we already materialise.
- Reading results. Duals have shipped; reduced costs and slacks ride the same join and have not. Derived results (LCOE, curtailment, emissions by group) are SQL over the solution tables.
- Infeasibility. HiGHS has no IIS, so the answer is elastic relaxation (#80) — slacks with penalty costs, then a query of nonzero slacks grouped by block. Needs no solver feature and works on every sink. Taxed like a primitive, since new variables mean a schema-level expansion pass.
- The REPL gap is verbs, not structure. A built model is four polars frames,
which answers "what is in row 12" with a filter where a labelled
Datasetneeds a scatter —build()already returns the live executor, it just has nothing readable on it. Render a bound row, preview which rows survive awhere, evaluate an expression against bound data, coefficient ranges, model statistics. All read-only: inspecting a built model, never declaring one, which is the line that keeps this cheap and rule 5 intact. - Lifecycle.
var_labelis the solver column index with no remapping, so value-only re-solve is a label query pluschangeColsBounds, integrality is the same contract (changeColsIntegrality, already called at build), and appending rows moves no label at all. The session that holds the handle is #204; warm starts are #382, and they are what make rebuild-instead-of-edit cheap rather than merely correct. - Decomposition. Benders and successive substitution, and the shape
favours us — worth saying because the opposite is the natural assumption.
Decomposition wants sparse triplets plus label tables; that is the model
here, so the master/sub split is a
GROUP BYoverA(#39), cut coefficients areduals ⋈ subproblem rows, and a cut lands with zero translation. The open question is not feasibility but who writes the cut (#381). - AST consumers. The first has shipped —
to_latex/to_typst/to_markdown, one tree walk, no data, no solver. Remaining: CLI (#35), observability (#34).
Track 3 — capabilities, and the degree line¶
The ceiling and sink capability are two axes. A declared capability set per
sink (not per solver — lp_file is not a solver but has capabilities),
modelled on linopy's Solver.features with two divergences: entries are
three-valued (native / reformulated / absent) so satisfying one by
reformulation is additive later, and the model expresses conjunction
exclusions, because linopy declares HiGHS with INTEGER_VARIABLES and
QUADRATIC_OBJECTIVE in one flat frozenset while HiGHS refuses the pair.
check(model, sink=...) takes the sink optionally. Design:
#89.
That unblocks three things, in order of effort:
| Blocked on | Note | |
|---|---|---|
| Semi-continuous | nothing | HiGHS has kSemiContinuous natively and linopy has the oracle — #383 |
| SOS / indicator | the capability model | lp_file carries SOS as a text section, Gurobi natively, HiGHS not at all — #23 |
| Quadratic | the capability model, then a second solver | below |
Quadratic is planned, not refused. The cost side is settled and small, and
the oracle is free — linopy's QuadraticExpression builds the comparison, which
is normally the expensive half of a primitive. Performance is not the question.
The blocker is where it can land: HiGHS returns kError for
Hessian + integrality, and binary:, integer: and nonconvex piecewise: all
ship today, so on the default path quadratic conflicts with features already in
the language. That is a capability finding, not a reason to refuse the math — so
it needs the table above and a second solver on solver_direct without the
exclusion (Gurobi, #106).
Landing the primitive before either ships math the default solver refuses.
Until then piecewise: {convex: true} and the epigraph pattern are the answer
for convex 1-D curves — and they keep the LP duals, warm starts and MILP
compatibility a quadratic objective gives up, so they stay the preferred
spelling even after quadratic lands. The scope, the lowering, and whether
coordinate-aligned is the right restriction at all, are
#261 and
#84.
Track 4 — the memory axis¶
The engine holds the model it builds, so peak tracks the model rather than a number the caller sets. That is the right default and it is what makes the lifetime disappear from the API, but there is no declared ceiling — no way to say "build this within N gigabytes or fail". The honest version is partition-wise execution, which the locality closure already guarantees is safe. Worth most for the write path: the solver is the larger term by roughly an order of magnitude at 10⁷ variables (benchmarks).
What we will not build¶
| Request | Why | Instead |
|---|---|---|
| Data prep — resampling, clustering, IO, units | not math | preprocess; pass a parameter |
Arbitrary array ops (merge, reindex) |
unbounded; xarray with extra steps | data prep |
Domain helpers (reduce_carrier_dim) |
encodes one domain into the language | component libraries over generic primitives |
Normalisation (x / sum(x)) |
a variable divisor is rational, not polynomial — no sink takes it at any degree | state the ratio as a constraint, or fix the denominator |
| Conditionals, iteration, data-dependent structure inside one plan | destroys the closed AST | where masks + foreach dims. A process may loop over plans — that is Track 2 |
| A Python API for constructing models | hard rule 5 — the model is the file you review and diff | YAML. Whether Python may emit declarations is #381 |
Genuinely unsayable math goes to a declared escape: island
(#38) — named in the file,
bounded by the preceding where, terminal, and billed against a label budget
before any Python runs. It buys back relational and local; it cannot buy
back degree, since it returns affine COO rows either way.
Honest snapshot¶
Cheaper here, because the model is tables: model statistics and coefficient-range diagnostics; IIS read-back (a join, not a scatter); serialization to parquet; elastic relaxation; dualization, since transposing a COO matrix is swapping two column names.
Ahead of comparable declarative layers: sparse-by-construction build with no
dense intermediate, and a hand-off straight to the solver rather than through a
file; parameterised macros: (Calliope's sub-expressions take no arguments);
binary and integer variables; piecewise as N links with per-link signs, convex
mode and active gating; load-time validation of every expression, where
string and uncalled macro template.
Behind linopy, and none of it a ceiling question: the post-solve object
(labelled DataArrays vs tidy tables — to_dataarray bridges), debugging (IIS
via Gurobi, print() of a row), lifecycle (mutate, re-solve, warm start,
relax/fix), solver breadth (ten backends and four handoffs vs HiGHS-direct
plus LP files, #106), and the
variable types and capabilities in Track 3.
The ranking this page implies: indexed access blocks whole model classes
today; the Track 2 verbs block using the engine at 3am; solver breadth blocks
arrival from linopy at all; semi-continuous and cumsum-over-data are cheap,
unblocked and unscheduled.
Two durable losses. Structure that needs the solver's answer to decide the next row inside one plan, and imperative modeling at all (rule 5). Both are the price of the closed AST — which is what buys load-time validation, two lanes on one language, and a build that streams. Everything else here is scheduling.