GeezerVison

So, a few quick comments:

1. I basically don't know rust at all (I believe I've used it for a combined twelve minutes), so I can't speak to any of the code itself.
2. Rust 1.67 was released literally two days ago. Compared to us sticking to C++11 that's a staggering difference in version support. Is it that much easier to get a new rust on old systems? Our typical targets are old CentOS and Debian.
3. Following on from that, how easy is this to package from a distro perspective? We've expended quite a bit of effort on making that easy, and I'd hate to drop that aspect.

As of yet unknown compatibility story for Tier 2+ platforms (Cygwin, etc).

Cygwin is mostly a major PITA because of its wchar_t type. If we can get away from using libc that could make some things easier.

get away from using libc

Rust uses libc under the hood.

Rust uses libc under the hood.

Of course technically that's true, but that's not what I'm talking about.

What I mean is that we wouldn't use e.g. libc's "iswdigit" and friends, like we're forced to do now because C++ never got around to replacing them with ones that work with its strings.

And because we use those functions, we're beholden to the libc wchar_t type where we use them, which is everywhere, which means the difference between cygwin (2 byte wchar_t) and as best as I can tell any other system (4 byte wchar_t) can appear everywhere.

And we're beholden to platform differences like "does this provide a uselocale" or "does this have wcscasecmp or std::wcscasecmp", because those directly appear in libc.

In effect what I'm saying is that my hope is rust would not rely on libc for everything and abstract some of the other libc gunk away instead of throwing it in our face like C++ does. And so this historical wchar_t difference could just go away.

The progress to date is impressive, and the motivation to allow concurrent mode is compelling.

I agree that packaging is important but I'd be happy to tackle it. RHEL 7 is only supported for another 18 months, and I have managed to backport GCC to RHEL in the past so it might be possible to do the same. There's a couple of ways to generate Debian packages, which are worth exploring.

I don't know any rust, but I didn't know any C++ before I started work on fish, so I'd be keen to learn.

(Maybe a way forward to #972 etc?)

Just throwing an idea out there: Given that this is a big underlying technical change and needs some larger changes in packaging, would it make sense to:

1. Call fish-in-rust "fish 4"
2. Support the last C++ series (3.6.x?) for a bit longer with bugfixes, added completions and such

This is, of course, a change in policy and more work, but I believe it may be warranted. These releases could be source-only (just tag a 3.6.3 and send out a mail about it), for those who are unable to install rust, especially distro packagers who find themselves looking at an incompatible rustc version.

It's of course possible I'm overestimating the difficulty here, maybe it's not needed because nobody has a problem switching to 4.0 immediately.

Building on that old NetBSD VM I have lying around currently fails with

running: "ar" "s" "/home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out/libfish-rust.a"
  exit status: 0
  cargo:rustc-link-lib=static=fish-rust
  cargo:rustc-link-search=native=/home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out
  cargo:rustc-env=AUTOCXX_RS=/home/alfa/fish-shell/build-rust/fish-autocxx-gen/rs

  --- stderr

  CXX include path:
    /home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out/cxxbridge/include
    /home/alfa/fish-shell/build-rust/./cargo/build/x86_64-unknown-netbsd/release/build/fish-rust-6df62e1ec51440cc/out/cxxbridge/crate
  /home/alfa/fish-shell/fish-rust/../src/proc.h:103:23: error: static_assert expression is not an integral constant expression
  /home/alfa/fish-shell/fish-rust/../src/proc.h:103:23: note: cast from 'void *' is not allowed in a constant expression
  Error: 
    × the include_cpp! macro couldn't be expanded into Rust bindings to C++:
    │ Bindgen was unable to generate the initial .rs bindings for this file.
    │ This may indicate a parsing problem with the C++ headers.

--- CMakeFiles/_cargo-build_fish-rust ---
*** [CMakeFiles/_cargo-build_fish-rust] Error code 101

Removing that static_assert (which is fine) makes it go on until it eventually ends in

[ 98%] Linking CXX executable fish
ld: libfish_rust.a(nix-e8ff18d973978598.nix.35ee0c1d-cgu.15.rcgu.o): in function `nix::sys::timer::Timer::get':
nix.35ee0c1d-cgu.15:(.text._ZN3nix3sys5timer5Timer3get17h045b7bbd1ff7f87dE+0x1b): warning: warning: reference to compatibility timer_gettime(); include <time.h> to generate correct reference
ld: libfish_rust.a(nix-e8ff18d973978598.nix.35ee0c1d-cgu.15.rcgu.o): in function `nix::sys::timer::Timer::set':
nix.35ee0c1d-cgu.15:(.text._ZN3nix3sys5timer5Timer3set17h47eed997322493a6E+0x4e): warning: warning: reference to compatibility timer_settime(); include <time.h> to generate correct reference

(and a bunch more nix::sys::timer and nix::sys::aio functions)

It also absolutely requires libclang, and will fail rather late in the build process.

Rust 1.67 was released literally two days ago. Compared to us sticking to C++11 that's a staggering difference in version support. Is it that much easier to get a new rust on old systems? Our typical targets are old CentOS and Debian.

If you want to use the distro version of Rust then you'll need to use a much older rust than latest stable. You can look it up for a particular distro, but Debian's rustc (for example) is often 6 months or more behind the latest stable. If you want to ship your program in a distro you usually need to build the program using that distro's version of the compiler, so I suspect you'd need to push your "minimum supported rust verison" (MSRV) farther into the past if your project is supposed to be part of distros.

If you don't need to be shipped in a distro, it's as easy as rustup update to update to the latest compiler. It's so simple to update the compiler that it's honestly hard to convince people in the rust community to have a library support anything other than the latest stable.

You can look it up for a particular distro, but Debian's rustc (for example) is often 6 months or more behind the latest stable.

AFAIK Debian updates both rustc and other packages for a while, and then freezes all of them. So it does have the latest rustc for the duration of the development cycle, and so this should not be an issue for packages shipped by the distro itself.

Third-party repos, however, will need either their own newer rustc for the build, or have all the code written in Rust support an older compiler version, including all dependencies.

Just taking a quick look at the versions in https://packages.debian.org/sid/rustc:

• buster 1.41 (2020-01-30)
• bullseye 1.48 (2020-11-19)
• bookworm 1.63 (2022-08-11)
• sid 1.63

I'm not sure what you mean by "for the duration of the development cycle", but even sid falls behind "latest stable" on a regular basis until someone goes and updates it.

so I suspect you'd need to push your "minimum supported rust verison" (MSRV) farther into the past if your project is supposed to be part of distros.

You do not. See: BurntSushi/ripgrep#1019

Essentially, distros like Debian will just ship an older version of your program, just like they do for almost everything else.

I've been alternating between obsessed with and simply always entertaining in the back of my mind the idea of a rust-powered fish for many years now (and have discussed that with other team members), so I'm approaching this with an open mind, but also with concerns. I must say that I knew this was coming for many months now from following @ridiculousfish on GitHub (❤️), and have been wondering how it would the topic would be broached.

I passionately believe that rust is the way forward for writing any new code, especially in a cross-platform context. Quite apart from all the memory and concurrency safety it brings to the table, the general approach of the language and the community to always put correctness first and foremost has built a strong culture of correct, well-designed, and fairly maintainable software engineering. The package management is great, the team is fantastic, and support in the broader software world has been extremely positive and welcoming.

To go further, I have written my own fish-compatible rewrite (not port) of fish's tokenizer and was making progress on the AST and parser when I last had my "burning desire" to see something like fish but in rust. I was maintaining it on my private git server and not GitHub, but I've pushed it to GH in case anyone cares to take a look: https://github.com/mqudsi/velvet

Having said all that, I worry that rushing any part of this could be a death knell for the project. A quality rewrite (or port) takes time, on the magnitude of years rather than days. One good example to take inspiration from is tectonic, a xelatex port that started off with machine-rewritten code and then manually made progress on actual, idiomatic rust conversions... but even that was a greenfield project and not a first-party effort and so not saddled with some of the concerns we would be saddled by. The recommended module-by-module approach is definitely a great suggestion for a starting place, and I think it has great potential.

Off the top of my head, some concerns or points:

• Compatibility, obviously. Fish runs on archaic hardware for fun. This would be the end of that. That's not a reason that should stand in the way of progress, but it's something we can't turn a blind eye to and would have to quite clearly signal that we're ok giving up all platforms that don't run (modern versions of) rust.
• Idiomatic rust is too high-level for fish. You don't fork in rust, using the standard library interface doesn't give you access to poll/select/epoll/kqueue/whatever natively. async io is a horrible fake cludge with tokio/async_std faking it with thread pools or whatnot and performing far worse than truly native async implementations. We'd have to go that mostly alone to keep the same code we have (thinking "port" and not "rewrite" and aiming to preserve what we have).
• Despite whether we call it a "port" or a "rewrite" there's going to be a heck of a lot of rewriting taking place. Subtle bugs will creep in. Subtle compatibility fixes that weren't properly documented or annotated will be lost.
• We will be in a state of partial limbo until this completes one way or the other. We won't want to make any major changes to the C++ codebase while we're trying to get to feature parity in the rust one. Depending on how things unfold this could greatly hinder further progress in the project.
• Not all core team members are fluent in rust. This seems unfair. (This isn't just something I'm saying in passing but rather something I've given a lot of thought to and broached with others before.)
• C++ isn't really a "legacy" language (yet), no matter how much we wish the world would start treating it like that given all its warts and shortcomings.
• I understand what @ridiculousfish meant by "Rust is what we need to turn on concurrent function execution" but respectfully disagree. A complete rewrite (or port, if you prefer) to another language is a great time to facilitate that, but it's hardly a requirement, though I guess it could functionally be considered one if you want to take into account correctness, maintainability, etc.
• (Really minor compared to everything else, but if we're ultimately going to be a rust project, then I think it makes sense for all types coming from the rust codebase to use idiomatic rust naming conventions, but that would cause churn cpp-side. I suggest using idiomatic rust naming conventions and then using statements to alias them with cpp-style names in the cpp headers.)

Idiomatic rust is too high-level for fish. You don't fork in rust, using the standard library interface doesn't give you access to poll/select/epoll/kqueue/whatever natively. async io is a horrible fake cludge with tokio/async_std faking it with thread pools or whatnot and performing far worse than truly native async implementations. We'd have to go that mostly alone to keep the same code we have (thinking "port" and not "rewrite" and aiming to preserve what we have).

Note that in the Approach section of the documentation there is this:

As a general comment to anyone reading this, it's worth reading the entire fish-riir-plan.md document before commenting.

https://github.com/ridiculousfish/fish-shell/blob/riir/doc_internal/fish-riir-plan.md

Just leaving a comment here, based on your planning doc:

types cannot represent non-UTF8 filenames

This crate might be what you want to use instead.

Any update on when/how riir will be merged? Right now, master is steadily diverging from it.

I'll tag the "Last C++11" and merge in maybe 12-16 hours, assuming there are no objections from fish shell committers. (It will be a real merge, not a rebase).

Merged as 27f5490 Here we go! 🤞

I've deleted the riir branch; please redirect any outstanding PRs to master

I've deleted the riir branch; please redirect any outstanding PRs to master

Doesn't look like that's possible once they've been force-closed, so they'll have to be recreated.

If it's not too much trouble, some people over at r/rust are curious how the rewrite is going? ❤️

Is it at a point where people unfamiliar with the codebase might be able to help with drive-by contributions?

@PoignardAzur I've written something up about the state of the port in #10123.

I would ask you to leave any replies there, this PR is quite big and replying here might notify a lot of people.

In short: We're "mostly" done and I don't believe the rest is well-suited for drive-by contributions.