GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix sugar expression lifetimes, aliased assignment, and operand lengths by kevinushey · Pull Request #1513 · RcppCore/Rcpp

Conversation

@kevinushey

@kevinushey kevinushey commented Sep 30, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Closes #1512.

This fixes the ways sugar expressions could silently give wrong results (or crash) described in #1512. It's split into commits by problem; each commit's message has the details.

1. Stored expressions no longer dangle

Sugar nodes held every operand by const&, including temporary sub-expressions, so an expression kept past the statement that built it referred to destroyed objects. auto e = x + x * 2.0; return e; returned garbage, and auto e = -(x * 2.0); segfaulted R.

Nodes now hold their operands through a new traits::sugar_operand<T>: objects that own an R object (vectors, matrices and classes derived from them) by reference, and everything else (nested expressions, views such as MatrixColumn, scalars in rep()) by value. auto e = x * 2.0 + y; is now safe. An expression built on a temporary vector, e.g. auto e = clone(x) + 1.0;, can still dangle; that is documented in the sugar vignette.

2. Aliased in-place assignment is evaluated correctly

Assigning a same-length expression to a vector, or any expression to a range, row or column, writes into existing storage while the expression is still reading. Expressions that read other positions of that storage gave wrong results: x = rev(x) gave 5 4 3 4 5, and the shift x[Range(1, n - 1)] = head(x, n - 1) gave 1 1 1 1 1.

Assignment still happens in place (so attributes are kept and the caller's object is still modified, as before), but only expressions known to be safe are written directly:

  • Nodes whose element i depends only on element i of their operands declare them with typedef traits::elementwise_operands<...> rcpp_elementwise. That covers arithmetic, comparisons, !, &, |, math functions, ifelse(), pmin()/pmax(), is_na() and friends, and the d/p/q functions.
  • Views declare what kind of view they are (rcpp_view). Whether reading one in place is safe depends on the target:
    • a whole vector can read any view;
    • a column can read other columns;
    • a row can read other rows;
    • a range can't read any view.
  • Anything else (e.g. rev(), head(), sapply(), whose function may read the target) is evaluated into a temporary first.

3. Operand lengths are checked in development versions

Operations on two or more vectors took their length from one operand, so a shorter operand was read past its end (with only the bounds warning) and a longer one was silently truncated. These nodes now check their operands' lengths once, when they are constructed, and signal an error on a mismatch, matching the vignette's documented requirement that operands have the same size. Vectors assigned into a range, row or column are checked the same way. Several of these constructors already carried FIXME/TODO notes asking for exactly this check.

The checks are compiled in only for development versions. config.h derives RCPP_SUGAR_LENGTH_CHECKS from the version: on when the fourth component is non-zero (e.g. 1.1.2.5, which is what reverse-dependency runs test), off for releases (three components, fourth set to zero). Defining the macro as 1 or 0 before including Rcpp.h overrides the default. With the checks off, check_sizes() and check_assign_size() are empty inline functions and behaviour is exactly as before: the result is sized by the first operand or the target, and a longer operand is cut short.

The reason is in the "Reverse dependencies" section below: twelve CRAN packages rely on the truncation, and an unconditional error would hold the release until all twelve update. This way the reverse-dependency runs keep catching the mismatches (and the fixes have been sent upstream), while the CRAN release doesn't break anything. The tests check for the errors when enabled and for the truncation otherwise, and the sugar vignette documents when the lengths are checked.

4. !, && and || on single logical results compile, and handle NA like R

None of !all(x), all(x) && any(y), all(x) || any(y), all(x) && true or all(x) || true compiled on master:

  • The combinators called the non-const get() through const references.
  • The &&/|| overloads for two results declared their second parameter with the first one's template arguments, so they could never be deduced (leaving an ambiguity between the bool overloads).
  • Or_SingleLogicalResult_bool named the And class as its CRTP base.

The combinators now hold their operands by value like other sugar nodes, so auto b = !(all(x) && any(y)); is also safe. Since this code had never run, its logic had a bug too: the general &&/|| treated an NA on the left as final, so NA && FALSE gave NA and NA || TRUE gave NA. They now follow R's three-valued logic.

Performance

Same functions compiled against master vs this branch (R 4.6.1, arm64 macOS); results are identical.

code master this PR
x + y (1e7) 24.8 ms 24.9 ms
x * y + 2.0 * x - y (1e7) 52.2 ms 50.2 ms
ifelse(x > 0.5, x, -x) (1e7) 86.1 ms 82.2 ms
x = x * c + d in place, 10 times (1e7) 61.6 ms 41.2 ms
z = x * y + x, n = 10, 1e6 times 29.0 ms 29.6 ms
m(_, j) = m(_, j) * 2.0 for each column (1000 x 1000) 0.20 ms 0.20 ms

Two things were needed to keep these at parity:

  • Target-aware view rule. Without the per-target rule for views, the column update above went through a temporary and was ~4.75x slower.
  • Out-of-line error path. With the length check's error path inlined, the compiler stopped propagating the constant multiplier into that loop, and it vectorized half as wide.

Expressions that aren't elementwise (e.g. x = rev(x)) now pay for a temporary when assigned in place, which is what makes them correct.

Other changes

  • SugarBlock_3_VVV declared its third operand with the second operand's type; fixed.
  • MatrixBase gains a const get_ref(), matching VectorBase.
  • lapply() indexes its operand through VectorBase again, so a vector yields its stored type (e.g. SEXP for a CharacterVector) rather than a proxy. Holding the operand as the derived type had broken lapply(x, f) where f takes a const CharacterVector& (seen as a compile failure in the touch package).
  • The sugar vignette gains a section on operand lengths, storing expressions and assignment, and its Sapply example uses sugar_operand. I edited only vignettes/rmd/Rcpp-sugar.Rmd; the pre-built PDF hasn't been regenerated.

Testing

New tests are in inst/tinytest/cpp/sugar_expressions.cpp, run from test_sugar.R. They cover:

  • stored expressions of each kind of node, after overwriting the stack;
  • each aliasing case above, including row/column cross-reads that must go through a temporary;
  • which expressions are written in place for each kind of target;
  • length mismatches in both directions;
  • !, && and || on single logical results, against R's own operators for every combination of TRUE, FALSE and NA (including the specializations for operands known not to be NA, and the bool overloads).

On master, the lifetime tests return wrong values or crash R, and the aliasing tests fail. The full suite passes locally with RunAllRcppTests=yes (1732 tests, 0 failures). All test sources also compile with -std=c++11, and introduce no new warnings under -Wall -Wextra -Wconversion other than two -Wsign-conversion notes (clang-only; gcc's -Wconversion doesn't enable those for C++) where rowSums()/diag() now call Matrix::operator()(size_t, size_t) directly.

Reverse dependencies

Tracking issue for the affected packages: #1517.

The 2026-10-06 reverse-dependency run (against a development version, so with the length checks on) flags 12 packages with the new length errors, plus the touch compile failure fixed above. Against a release build with the checks off, all 12 pass unchanged. Each was traced to a single site and a fix was sent upstream (linked below); in all 12, released Rcpp sized the result by the left operand or target and silently truncated the longer side, so none crashed before. In 8 the truncated result happened to be correct; in binspp, CARBayesST, PP and HistDAWass it was wrong or read out of bounds.

Package Site Problem
greeks (PR) make_BM.cpp:15 inclusive Range(s, s+paths) is one element too long
FDX (PR) kernel_weight.cpp:111 pmin(probs, NumericVector(numTests, 1.0)) with shorter probs
EpiLPS (PR) KerRpostmap.cpp:69-73 BBtemp built with length t instead of simax
clustAnalytics (PR) scoring_functions.cpp:136 com_size allocated with n instead of n_com
binspp (issue) Binspp_core.cpp:10 adds images on different pixel grids
PP (PR) GPCM4PL_stuff.cpp:244 and siblings R side subsets respm but not theta_start
CARBayesST (issue) CARBayesST.cpp:5189 via matN(rep(mode.phi,N),K,N) length K*N assigned to each length-N row
dbscan (PR) kNN.cpp:59 k+1 duplicate neighbours assigned to a width-k row
FVDDPpkg (patch sent by email) C++FVDDP.cpp:220 reps updated before use, RHS len(v) times too long
HistDAWass (patch sent by email) Utility_fun.cpp:1289 NumericVector tmp(k, ...) assigned into a row of width vars
tsfgrnn (PR), tsfknn (PR) code.cpp:42/69/96, code.cpp:41/70/99 Range(ind-nt+1, ind+1) assumes a half-open range

The revdbayes failure in the same run is unrelated to this PR; it comes from the object layout change in #1511.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eddelbuettel reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
kevinushey and others added 13 commits September 30, 2026 11:40
@kevinushey
hold nested sugar expressions and views by value
14c88c5
Sugar nodes held every operand by const reference, so an expression kept past the full-expression that built it (e.g. in an auto variable) referred to destroyed temporaries. Nodes now hold nested expressions and views by value, and only objects that own an R object (vectors, matrices) by reference.

Part of #1512.
@kevinushey
evaluate non-elementwise sugar before writing it in place
ecd157f
Assigning a same-length sugar expression to a vector, or any expression to a range, matrix row or matrix column, writes into existing storage while the expression is still being read. When the expression read from that storage at other positions (e.g. `x = rev(x)`, `x[Range(1, n - 1)] = head(x, n - 1)`), the result was silently wrong.

Elementwise nodes now declare themselves via a nested `rcpp_elementwise` type; those are still written in place, and everything else is evaluated into a temporary first.

Part of #1512.
@kevinushey
check that sugar operands have matching lengths
dd71cd5
Sugar operations on two or more vectors took their length from one operand and never compared it with the others, so a shorter operand was read past its end and a longer one was silently truncated. Likewise for vectors assigned into a range, matrix row or matrix column. These now signal an error, matching the documented requirement that operands have the same size.

Part of #1512.
@kevinushey
let rows and columns read matching views in place
617bb03
Evaluating every expression that reads a view into a temporary made updates like `m(_, j) = m(_, j) * 2` several times slower. Whether reading a view in place is safe depends on where the result is written: a column can read other columns of the same matrix (they coincide or are disjoint), a row other rows, and a whole vector any view, but a range can't read another range. Sugar expressions now list their elementwise operands, views declare what kind of view they are, and each write site checks the expression against its own kind of target.

Part of #1512.
@kevinushey
keep sugar length-check errors out of line
60f24be
With the error path inlined, the compiler could no longer propagate constants into the loops following these checks (e.g. `m(_, j) = m(_, j) * 2.0` lost its constant multiplier and vectorized half as wide).
@kevinushey
document sugar operand lengths, lifetimes, and assignment
878bea9
@kevinushey
update NEWS
3292c48
@kevinushey
fix !, &&, and || on single logical results
dcca2a1
None of `!all(x)`, `all(x) && any(y)`, `all(x) || any(y)`, `all(x) && true` or `all(x) || true` compiled: the combinators called the non-const get() through const references, the && and || overloads for two results declared their second parameter with the first one's template arguments (so they could never be deduced), and Or_SingleLogicalResult_bool named the And class as its CRTP base.

The combinators now hold their operands by value, like other sugar expressions, and the general && and || follow R's three-valued logic (NA && FALSE is FALSE, NA || TRUE is TRUE).
@kevinushey
update NEWS
e03c2ed
@kevinushey
route range-to-range assignment through the aliasing checks
32b70a4
Also mark Range as elementwise, include sugar_operand.h from
is_elementwise.h, and drop stale FIXME comments in ifelse.h.
@kevinushey
read subset elements before writing when both index the same vector
d4cbe58
@kevinushey
index lapply operands through VectorBase so vectors yield their store…
e6a267b
…d type

Holding the operand as the derived vector type made lapply() pass a
proxy (e.g. const_string_proxy for character vectors) to the function,
which no longer converted to a const CharacterVector& argument. Seen as
a compile failure in the touch package.
@kevinushey
update ChangeLog
2e6ab0b
This was referenced Oct 9, 2026
fix off-by-one range in make_BM ahudde/greeks#23
Open
cap probabilities with a scalar in kernel_wPB_fast DISOhda/FDX#1
Merged
use a window of length simax for BBtemp in Rcpp_KerRpostmap oswaldogressani/EpiLPS#4
Merged
allocate com_size per community in FOMD_Rcpp martirm/clustAnalytics#2
Merged
covariate images are added to Wpix on a different pixel grid tomasmrkvicka/binspp#1
Open
phi.mat in the CARclustrends models is built from a vector of the wrong length duncanplee/CARBayesST#13
Open
keep the k nearest hits in kNN_int when the query point is not among them mhahsler/dbscan#99
Merged
use an inclusive Range of length nt for the targets franciscomartinezdelrio/tsfgrnn#2
Open
use an inclusive Range of length nt for the targets franciscomartinezdelrio/tsfknn#5
Open
keep theta_start, mu and sigma2 aligned with respm after dropping persons jansteinfeld/PP#3
Open
Transition to sugar length checks (#1513): affected packages #1517
Open
@kevinushey
enable the sugar length checks in development versions only
9682f05
Twelve CRAN packages rely on a longer sugar operand being cut short, so
an unconditional error would block the release until all of them
update. Derive RCPP_SUGAR_LENGTH_CHECKS from the version in config.h:
on when the fourth component is non-zero (development snapshots, which
is what reverse-dependency checks run against) and off in releases.
Defining the macro before including Rcpp.h overrides the default. The
tests check for the errors only when enabled, and for the truncation
otherwise.
eddelbuettel
eddelbuettel approved these changes Oct 12, 2026
View reviewed changes

@eddelbuettel eddelbuettel left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing an 80 files PR is always tricky but given the reverse dependency pass (after the extra conditional was added allowing a run with the suppressed extra check) this is likely fine.

Thanks also for opening #1517 to see if we can get downstream to update where needed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@eddelbuettel
Merge branch 'master' into bugfix/sugar-soundness
8088d21
@eddelbuettel
eddelbuettel merged commit 0f3e317 into master Oct 13, 2026
30 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@eddelbuettel
eddelbuettel deleted the bugfix/sugar-soundness branch October 13, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eddelbuettel eddelbuettel eddelbuettel approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

sugar expressions give wrong results under aliasing, length mismatch, and captured temporaries

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@kevinushey @eddelbuettel
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.