GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — fix sugar expression lifetimes, aliased assignment, and operand lengths by kevinushey · Pull Request #1513 · RcppCore/Rcpp Conversation
|
| code | master | this PR |
|---|---|---|
x + y (1e7) |
24.8 ms | 24.9 ms |
x * y + 2.0 * x - y (1e7) |
52.2 ms | 50.2 ms |
ifelse(x > 0.5, x, -x) (1e7) |
86.1 ms | 82.2 ms |
x = x * c + d in place, 10 times (1e7) |
61.6 ms | 41.2 ms |
z = x * y + x, n = 10, 1e6 times |
29.0 ms | 29.6 ms |
m(_, j) = m(_, j) * 2.0 for each column (1000 x 1000) |
0.20 ms | 0.20 ms |
Two things were needed to keep these at parity:
Expressions that aren't elementwise (e.g. x = rev(x)) now pay for a temporary when assigned in place, which is what makes them correct.
SugarBlock_3_VVV declared its third operand with the second operand's type; fixed.MatrixBase gains a const get_ref(), matching VectorBase.lapply() indexes its operand through VectorBase again, so a vector yields its stored type (e.g. SEXP for a CharacterVector) rather than a proxy. Holding the operand as the derived type had broken lapply(x, f) where f takes a const CharacterVector& (seen as a compile failure in the touch package).Sapply example uses sugar_operand. I edited only vignettes/rmd/Rcpp-sugar.Rmd; the pre-built PDF hasn't been regenerated.New tests are in inst/tinytest/cpp/sugar_expressions.cpp, run from test_sugar.R. They cover:
!, && and || on single logical results, against R's own operators for every combination of TRUE, FALSE and NA (including the specializations for operands known not to be NA, and the bool overloads).On master, the lifetime tests return wrong values or crash R, and the aliasing tests fail. The full suite passes locally with RunAllRcppTests=yes (1732 tests, 0 failures). All test sources also compile with -std=c++11, and introduce no new warnings under -Wall -Wextra -Wconversion other than two -Wsign-conversion notes (clang-only; gcc's -Wconversion doesn't enable those for C++) where rowSums()/diag() now call Matrix::operator()(size_t, size_t) directly.
Tracking issue for the affected packages: #1517.
The 2026-10-06 reverse-dependency run (against a development version, so with the length checks on) flags 12 packages with the new length errors, plus the touch compile failure fixed above. Against a release build with the checks off, all 12 pass unchanged. Each was traced to a single site and a fix was sent upstream (linked below); in all 12, released Rcpp sized the result by the left operand or target and silently truncated the longer side, so none crashed before. In 8 the truncated result happened to be correct; in binspp, CARBayesST, PP and HistDAWass it was wrong or read out of bounds.
| Package | Site | Problem |
|---|---|---|
| greeks (PR) | make_BM.cpp:15 |
inclusive Range(s, s+paths) is one element too long |
| FDX (PR) | kernel_weight.cpp:111 |
pmin(probs, NumericVector(numTests, 1.0)) with shorter probs |
| EpiLPS (PR) | KerRpostmap.cpp:69-73 |
BBtemp built with length t instead of simax |
| clustAnalytics (PR) | scoring_functions.cpp:136 |
com_size allocated with n instead of n_com |
| binspp (issue) | Binspp_core.cpp:10 |
adds images on different pixel grids |
| PP (PR) | GPCM4PL_stuff.cpp:244 and siblings |
R side subsets respm but not theta_start |
| CARBayesST (issue) | CARBayesST.cpp:5189 via matN(rep(mode.phi,N),K,N) |
length K*N assigned to each length-N row |
| dbscan (PR) | kNN.cpp:59 |
k+1 duplicate neighbours assigned to a width-k row |
| FVDDPpkg (patch sent by email) | C++FVDDP.cpp:220 |
reps updated before use, RHS len(v) times too long |
| HistDAWass (patch sent by email) | Utility_fun.cpp:1289 |
NumericVector tmp(k, ...) assigned into a row of width vars |
| tsfgrnn (PR), tsfknn (PR) | code.cpp:42/69/96, code.cpp:41/70/99 |
Range(ind-nt+1, ind+1) assumes a half-open range |
The revdbayes failure in the same run is unrelated to this PR; it comes from the object layout change in #1511.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Sugar nodes held every operand by const reference, so an expression kept past the full-expression that built it (e.g. in an auto variable) referred to destroyed temporaries. Nodes now hold nested expressions and views by value, and only objects that own an R object (vectors, matrices) by reference. Part of #1512.
Assigning a same-length sugar expression to a vector, or any expression to a range, matrix row or matrix column, writes into existing storage while the expression is still being read. When the expression read from that storage at other positions (e.g. `x = rev(x)`, `x[Range(1, n - 1)] = head(x, n - 1)`), the result was silently wrong. Elementwise nodes now declare themselves via a nested `rcpp_elementwise` type; those are still written in place, and everything else is evaluated into a temporary first. Part of #1512.
Sugar operations on two or more vectors took their length from one operand and never compared it with the others, so a shorter operand was read past its end and a longer one was silently truncated. Likewise for vectors assigned into a range, matrix row or matrix column. These now signal an error, matching the documented requirement that operands have the same size. Part of #1512.
Evaluating every expression that reads a view into a temporary made updates like `m(_, j) = m(_, j) * 2` several times slower. Whether reading a view in place is safe depends on where the result is written: a column can read other columns of the same matrix (they coincide or are disjoint), a row other rows, and a whole vector any view, but a range can't read another range. Sugar expressions now list their elementwise operands, views declare what kind of view they are, and each write site checks the expression against its own kind of target. Part of #1512.
With the error path inlined, the compiler could no longer propagate constants into the loops following these checks (e.g. `m(_, j) = m(_, j) * 2.0` lost its constant multiplier and vectorized half as wide).
None of `!all(x)`, `all(x) && any(y)`, `all(x) || any(y)`, `all(x) && true` or `all(x) || true` compiled: the combinators called the non-const get() through const references, the && and || overloads for two results declared their second parameter with the first one's template arguments (so they could never be deduced), and Or_SingleLogicalResult_bool named the And class as its CRTP base. The combinators now hold their operands by value, like other sugar expressions, and the general && and || follow R's three-valued logic (NA && FALSE is FALSE, NA || TRUE is TRUE).
Also mark Range as elementwise, include sugar_operand.h from is_elementwise.h, and drop stale FIXME comments in ifelse.h.
…d type Holding the operand as the derived vector type made lapply() pass a proxy (e.g. const_string_proxy for character vectors) to the function, which no longer converted to a const CharacterVector& argument. Seen as a compile failure in the touch package.
Twelve CRAN packages rely on a longer sugar operand being cut short, so an unconditional error would block the release until all of them update. Derive RCPP_SUGAR_LENGTH_CHECKS from the version in config.h: on when the fourth component is non-zero (development snapshots, which is what reverse-dependency checks run against) and off in releases. Defining the macro before including Rcpp.h overrides the default. The tests check for the errors only when enabled, and for the truncation otherwise.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Reviewing an 80 files PR is always tricky but given the reverse dependency pass (after the extra conditional was added allowing a run with the suppressed extra check) this is likely fine.
Thanks also for opening #1517 to see if we can get downstream to update where needed.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
sugar expressions give wrong results under aliasing, length mismatch, and captured temporaries
There was an error while loading. Please reload this page.