GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Add more unicode functions to the c-api by bschoenmaeckers · Pull Request #8272 · RustPython/RustPython

Conversation

@bschoenmaeckers

@bschoenmaeckers bschoenmaeckers commented Jul 13, 2026 •
edited by coderabbitai Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features

    • Extended Unicode C-API with new constructors for UTF-8 strings, objects, and ordinals, plus default encoding reporting and string interning.
    • Added Unicode encoding helpers (ASCII, Latin-1, escape variants, UTF-16/UTF-32) and decoding APIs (general and encoding-specific wrappers).
    • Added Unicode operations and comparisons (concat, length, ordering/equality, containment, formatting, identifier checks, partition/rpartition, translate).
  • Bug Fixes

    • Improved decoding validation and edge-case handling (e.g., negative sizes and null input behavior) for more predictable failures.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bschoenmaeckers bschoenmaeckers changed the title Add more unicode functions Add more unicode functions to the c-api Jul 13, 2026
@coderabbitai

coderabbitai Bot commented Jul 13, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: acd21ceb-a8ba-402f-acfe-0b13a4608a71

📥 Commits

Reviewing files that changed from the base of the PR and between acbcaaf and 37fd030.

📒 Files selected for processing (2)
  • crates/capi/src/unicodeobject.rs
  • crates/vm/src/builtins/str.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/vm/src/builtins/str.rs
  • crates/capi/src/unicodeobject.rs

📝 Walkthrough

Walkthrough

The Unicode C API gains constructors, codec-based encoding and decoding helpers, comparisons, interning, containment, formatting, partitioning, translation, and identifier checks. Four PyStr methods become publicly callable for these APIs.

Changes

Unicode C API

Layer / File(s) Summary
Unicode construction and encoding
crates/capi/src/unicodeobject.rs
Adds string, object, and ordinal constructors, shared encoding support, and multiple Unicode encoding helpers.
Unicode decoding
crates/capi/src/unicodeobject.rs
Adds general and encoding-specific decoding functions with size and pointer validation and codec-registry delegation.
Unicode operations and VM method access
crates/capi/src/unicodeobject.rs, crates/vm/src/builtins/str.rs
Adds concatenation, length, interning, comparisons, equality, containment, formatting, partitioning, translation, and identifier checks while exposing the required PyStr methods.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CCaller
  participant UnicodeAPI
  participant PyStr
  participant codec_registry
  CCaller->>UnicodeAPI: provide Unicode object or encoded bytes
  UnicodeAPI->>PyStr: downcast and obtain string text
  UnicodeAPI->>codec_registry: encode_text or decode_text
  codec_registry-->>UnicodeAPI: encoded or decoded object
  UnicodeAPI-->>CCaller: return result
Loading

Possibly related PRs

  • RustPython/RustPython#7904: Extends the same Unicode C-API module with related constructors and encoding or comparison entrypoints.
  • RustPython/RustPython#8044: Adds related Unicode C-API encoding and decoding integration with the VM codec registry.

Suggested reviewers: youknowone

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding more Unicode C-API functions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
  • X
  • Mastodon
  • Reddit
  • LinkedIn

Comment @coderabbitai help to get the list of available commands.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

coderabbitai[bot]
coderabbitai Bot reviewed Jul 13, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
crates/capi/src/unicodeobject.rs (1)

98-171: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Collapse the six near-identical PyUnicode_As*String helpers onto a shared inner function.

Each of the six exported helpers has an identical body differing only by the encoding-name literal. Keep the individual #[no_mangle] symbols (required for the ABI) but delegate to one private helper to avoid drift.

♻️ Suggested shared helper
fn encode_as(unicode: *mut PyObject, encoding: &str) -> *mut PyObject {
    with_vm(|vm| {
        let unicode = unsafe { &*unicode }.try_downcast_ref::<PyStr>(vm)?.to_owned();
        vm.state.codec_registry.encode_text(unicode, encoding, None, vm)
    })
}

#[unsafe(no_mangle)]
pub unsafe extern "C" fn PyUnicode_AsASCIIString(unicode: *mut PyObject) -> *mut PyObject {
    encode_as(unicode, "ascii")
}
// ...and so on for latin-1, raw-unicode-escape, utf-16, utf-32, unicode-escape

As per coding guidelines: "When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once to avoid duplicate code."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/capi/src/unicodeobject.rs` around lines 98 - 171, Extract the
duplicated with_vm/downcast/encode_text logic from PyUnicode_AsASCIIString,
PyUnicode_AsLatin1String, PyUnicode_AsRawUnicodeEscapeString,
PyUnicode_AsUTF16String, PyUnicode_AsUTF32String, and
PyUnicode_AsUnicodeEscapeString into one private encode_as helper accepting the
encoding name, then have each exported ABI function delegate to it with its
existing literal while preserving all #[unsafe(no_mangle)] symbols.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/capi/src/unicodeobject.rs`:
- Around line 50-60: Update PyUnicode_FromObject to reject objects that are not
PyStr or PyStr subclasses instead of calling obj.to_owned().str(vm). Return the
existing C-API error representation for a TypeError, while preserving the
successful conversion path for accepted string objects.
- Around line 277-293: Update PyUnicode_CompareWithASCIIString to avoid calling
PyStr::expect_str(), which can panic for surrogate-containing strings. Compare
the PyStr value using its WTF-8 view, matching the behavior of
PyUnicode_Compare, while preserving the existing ordering-to-integer mapping and
UTF-8 validation for the C string.

---

Nitpick comments:
In `@crates/capi/src/unicodeobject.rs`:
- Around line 98-171: Extract the duplicated with_vm/downcast/encode_text logic
from PyUnicode_AsASCIIString, PyUnicode_AsLatin1String,
PyUnicode_AsRawUnicodeEscapeString, PyUnicode_AsUTF16String,
PyUnicode_AsUTF32String, and PyUnicode_AsUnicodeEscapeString into one private
encode_as helper accepting the encoding name, then have each exported ABI
function delegate to it with its existing literal while preserving all
#[unsafe(no_mangle)] symbols.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: a08d9e91-1068-4efd-8524-7620fca90951

📥 Commits

Reviewing files that changed from the base of the PR and between a9c2c52 and ecde4ab.

📒 Files selected for processing (1)
  • crates/capi/src/unicodeobject.rs

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread crates/capi/src/unicodeobject.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread crates/capi/src/unicodeobject.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bschoenmaeckers
bschoenmaeckers marked this pull request as draft July 13, 2026 14:15
@bschoenmaeckers

bschoenmaeckers commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bschoenmaeckers
bschoenmaeckers force-pushed the c-api-simple-unicode branch from 12667c2 to 1aa1771 Compare July 14, 2026 15:07
@bschoenmaeckers
bschoenmaeckers marked this pull request as ready for review July 14, 2026 15:07
coderabbitai[bot]
coderabbitai Bot reviewed Jul 14, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/capi/src/unicodeobject.rs`:
- Around line 578-591: Update PyUnicode_Translate to honor the _errors parameter
by converting and passing it through to PyStr::translate, preserving the
requested codec error-handler behavior; if PyStr::translate cannot support it,
explicitly document the intentional API divergence instead.
- Around line 258-331: Update PyUnicode_DecodeUTF16 and PyUnicode_DecodeUTF32 to
honor the _byteorder pointer: when it is non-NULL, select the corresponding
forced little-endian or big-endian codec based on its input value, while
preserving BOM/native-order decoding when it is NULL. After decoding, write the
detected byte order back through the pointer as required by the API.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: c01ec87c-2d24-4be3-b3ab-de829f116906

📥 Commits

Reviewing files that changed from the base of the PR and between 12667c2 and 1aa1771.

📒 Files selected for processing (2)
  • crates/capi/src/unicodeobject.rs
  • crates/vm/src/builtins/str.rs

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread crates/capi/src/unicodeobject.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread crates/capi/src/unicodeobject.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bschoenmaeckers
bschoenmaeckers marked this pull request as draft July 14, 2026 21:25
@coderabbitai coderabbitai Bot mentioned this pull request Jul 16, 2026
Add codecs support to c-api #8267
Merged
@bschoenmaeckers
bschoenmaeckers force-pushed the c-api-simple-unicode branch from 1aa1771 to 3a63323 Compare July 17, 2026 07:56
@bschoenmaeckers
bschoenmaeckers marked this pull request as ready for review July 17, 2026 17:09
@bschoenmaeckers

bschoenmaeckers commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

youknowone
youknowone requested changes Jul 18, 2026
View reviewed changes

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread crates/capi/src/unicodeobject.rs Resolved

Uh oh!

There was an error while loading. Please reload this page.

bschoenmaeckers added 3 commits July 20, 2026 15:59
@bschoenmaeckers
Add more unicode functions
ea6bd45
Review

Add unicode decode functions

Add more functions
@bschoenmaeckers
Remove
8a845b8
@bschoenmaeckers
Add helper
37fd030
@bschoenmaeckers
bschoenmaeckers force-pushed the c-api-simple-unicode branch from acbcaaf to 37fd030 Compare July 20, 2026 14:06
youknowone
youknowone approved these changes Jul 22, 2026
View reviewed changes

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@youknowone
youknowone merged commit 72daf7c into RustPython:main Jul 22, 2026
26 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bschoenmaeckers
bschoenmaeckers deleted the c-api-simple-unicode branch July 22, 2026 11:48
youknowone pushed a commit that referenced this pull request Sep 16, 2026
@bschoenmaeckers
Add more unicode functions to the c-api (#8272)
455d22b
* Add more unicode functions

Review

Add unicode decode functions

Add more functions

* Remove

* Add helper
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments
@youknowone youknowone youknowone approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@bschoenmaeckers @youknowone
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.