For the runtime breaking scenarios, I think the "project template" solution might be a solution.
Existing projects unaffected, expert users can define it manually, new projects get the relevant setting by default via a template.

A few thoughts:

• I would like to see the design for .Compat.* to be more of a family of assemblies, and not just 1. That way, pieces of it could be imported without taking the whole dependency as a whole

How would it be distributed? We don't really want a new NuGet, or more "special" assemblies/projects at this point, it might become a maintenance nightmare very soon (since FSharp.Core is special and has features enabled only for it, and we will need to extend it for number of libraries).

Idea of "layered" core library is a good one, but probably a bit orthogonal to this one (thought related to it).

• There should be 0 work needed for brand new projects, this should ideally be a hidden detail. Solvable either via dependency on lang version (as Vlad wrote above), or by modified project templates.

This might work, however, we should keep in mind that it will require bringing in some msbuild magic to when we invoke compiler, to let it know which libraries/modules/namespaces to autoload. Logic of "default" behaviour will be spread thin between project file, compiler and core library (parts). This will also become very polluted very soon, if we will chose this approach.

• The following scenario should be considered: "I have an existing project. I update it to new NET+ new langversion. I do assume that if I do not see any new warnings/errors and the code compiles, I should not run into runtime issues".

  • This point is especially relevant for breaking changes which only happen on edge cases and do not have diagnostics, the whole nan equality semantics comes to mind.
  • I do not have an answer, but it is a problem area which should be kept in mind. Could custom code analyzers help here? A lot more aggressive "migration notes" about the changes which can exhibit such behavior?

It's also good to keep in mind that it's not how it currently works. Right now, one opts-in implicitly into new language features every time SDK is updated for the project (i.e. we always use latest language version), which inherently might bring some breaking changes (mostly, compile-time).

Problem is nobody reads those, and if they do, it shows that people don't pay enough attention.

Understanding how release notes interacts with your own code base can be hard. I might read it and think this should be good we don't rely on this but then there is one part of our project that relies on it directly or perhaps transiently through external dependencies which we have no insights in.

Could still be the reasonable way forward. F# is rather small language and has a pretty up to date community so could still be worth it in a risk/reward scenario.

Can an Intellisense refactor save us?

Can an Intellisense refactor save us?

You mean like showing potential braking changes to people?

It sure can, but I am not certain how to make them predictably detectable.

Ah - that makes sense. I echoed Mark Seeman's missive about linters earlier today and I kinda felt sheepish about making this suggestion as a cheap way out.

That was a point in recent discussion I had as well. Doesn't Fable have its own core library? Or do they have FSharp.Core fork and use it as a source?

I'd imagine they'll need to have a per-language mapping from FSharp.Core to target?

I imagine if we choose to do shadowing, no "binary" breaking changes will take place.

Yes, Fable reimplements the core library for all languages. Any breaking change should eventually be a breaking change in Fable as well

Yes, Fable reimplements the core library for all languages. Any breaking change should eventually be a breaking change in Fable as well

But only binary breaking changes, not runtime ones, correct? So, shadowing wouldn't affect anything.

I know there's the equality issue, but what else?

Vectorized collections operations, for example. Or if we'll want to migrate async to state machines, there might be some breaking changes.

I remember a huge PR a few years ago by @manofstick to replace Seq with something significantly faster that I seem to remember wasn't accepted due to risk of regressions. That would have been great to get in! I am one the weird ones that have a hard time using Seq due to being aware of the slowness and memory consumption of it even when it doesn't matter.

The Option type being a ref type when it should be a value type is a breaking change needed by the high-perf, real-time devs. Having to use OptionValue is a PITA. Many libraries only accept/return Option<'T>, which generates a lot of GC activity in a hot loop.

Yeah, versions are up for discussion, not sure if it's the prettiest solution or not. In regards of syntax changes - also need to think more about it, what would it imply and how can we communicate this.

I think it could get pretty unwieldy to maintain differing versions here :/

I think it could get pretty unwieldy to maintain differing versions here :/

That's my concern as well. Also opens up potential rabbit hole of "try to figure out what implementation is used based on sdk, language version and project flags"

I see three cases

1. I upgrade sdk and fsharp core (which is the most often) - the compat will be not used, so it won't help avoid issues

2. I use old sdk and upgrade only fsharp core - compat will be not used, so it won't help avoid issues

3. I upgrade sdk, but keep old fsharp core - compat will be used, so it will help avoid issues

It won't, just an old fsharp.core will be used, depending on language version.

I've ever used only 1 and 2 in my experience, option 3 where compat makes sense seems to be the least often used

That's the whole point, introduce a breaking change, with ability to quickly opt out (either via language version or some property). Alternatively it can fully be opt in via the same mechanism.

Can't just make a breaking change and not let people to chose to fallback to the old behaviour.

with ability to quickly opt out

This doesn't have any importance to me. Let's say I have a silent bug sitting in my code which relies on buggy NaN comparison. It's 99% chance that I don't know about it (otherwise I would fix it myself). As I don't know about it - I upgrade FSharp.Core to get new features and/or performance (why do I upgrade otherwise). And it appears that upgrade breaks in production. At the moment when I investigate and issue, it would be clear that the issue is that I relied on NaN in the first place, so I'll rather fix it than enable compat.

Can't just make a breaking change and not let people to chose to fallback to the old behaviour.

If people are that afraid of doing upgrade that after upgrade they want to enable compat, then they shouldn't upgrade at all. Because the unintentional breaking changes can still happen and nobody is immune of those.

This doesn't have any importance to me.

Exactly, you're only describing your experience, which is not necessary mapping to others'.

I talked to some of F# folks who use it in bioinformatics or telecoms, and they will appreciate if we don't break much of runtime.

appreciate if we don't break much of runtime

I'll definitely appreciate this as well! :) But whenever I upgrade major version of any library or framework, I do acknowledge that runtime can be broken. Also, if I really care about the minimizing the impact of upgrade, I'll very carefully read release note and possible breaking changes of major version change.

I agree with @Lanayx . Any runtime bugfix is "breaking" since when fixed it is a runtime change in behaviour and people may be relying on the bug. We should not be scared about bugfixes!

Any approach that unblocks F# progress is good, especially in an area like the canonical example here which both fixes a bug and improves performance. That includes the proposal here. However the proposal adds additional complexity in communication, and may add additional complexity in the compiler, which will then result in making it harder to make improvements in future. So I believe the best approach would be to do nothing except fix the bug. If anyone for historical reasons wants to preserve runtime behaviour then they can specify an FSharp.Core version and a dotnet sdk.

That sounds no different to just a nuget with some shadowing modules.

Yep, that is.

Ok, done, what should I include next?

https://www.nuget.org/packages/FSharp.Core.Faster
https://github.com/vzarytovskii/FSharp.Core.Faster

We can test things on this package version and iterate faster than normal FSharp.Core.

Nice! When shadowing you might run into #13535 though, as we created such a library internally as well

Ok, done, what should I include next?

https://www.nuget.org/packages/FSharp.Core.Faster

https://github.com/vzarytovskii/FSharp.Core.Faster

We can test things on this package version and iterate faster than normal FSharp.Core.

Just to be clear, I unironically intend to use it as platform for experiments and benchmarks. I also might try and use it in our VS extension (since it's more or less independent) and maybe in future LSP layer, since it'll be a new project with small to no risks.

Been thinking about it as well. Will experiment with it.

Don't forget tuples, basically they're wrappers, should've been ValueType

If newer core is loaded, then yes (given its runtime change and nothing was inlined).

That is a non-behavioural breaking change. It will result in huge compile-time and runtime breaking changes, and will likely never happen in such form. I can suggest shadowing for all functions used to work with value options.

Shadowing is a great idea that solves the problem. Throw in a nice compiler warning too and we'd be good to go!

If we don't break things other people will, it's part of growing up, (none ValueType wrappers) needs byebye

Shadowing doesn't guard against code that uses reflection.

Shadowing doesn't guard against code that uses reflection.

Yeah, but reflection should then depend not only on the structure but also on the types of entities (structs vs ref types, etc), which is surely a "deep" implementation detail, and people should not do that (I know they will anyway).

Leave behind people who wants to use the old version, by introducing change inside next version of F# / .NET and telling people if upgraded to newest version there is breaking change, start documenting the behavior of old version and the new version of the language is different and how, and making these changes a moduler part of the language, so people who wants to use the old version of a specific part of the language can specify so (some how), other parts of the language can be using latest F# features

Yeah, this isn't happening any time soon. Especially if the change is "silent" or runtime.

People building their project in .NET 6 is still going to use the old version, you can achieve the adaptive compatibility because people has to change their project to use the newer version of Fsharp, as we are moving on to .NET 9 if they use new version of course they get new features and breaking changes and what comes with it

Interesting problem. Seems manageable to migrate by defining a serializer that adds a new serialization method in addition to pickler and a deserializer that deserializes either, then deserialize and reserialize the data, then delete the pickler, then update fsharp.core.

We are doing well with cbor serialization, we find that manual serialization is great for making changes needing backwards compatibility including occasional tricky sequential changes like the one above, and that github copilot is great at writing serializer code so you might find that the process goes faster now than it did a few years ago.

Yeah, it's all solvable in principle, but the correct choice of serialisation format is different on a case-by-case basis - it's not the kind of thing one can trivially automate, or else you end up trying to reserialise a petabyte of floating point data as JSON :)

That may be, but 1. it's possible to choose something which dominates pickler in terms of speed and size, and 2. JSON is heavily dominated in both dimensions so never the correct choice.

(By the way, another thing that appears to break FsPickler is DictImpl between 8.0.100 and 9.0.100; I just reverted an FSharp.Core upgrade that was intended to work around #18298. Again, we are performing a completely unsupported operation; just wanted to record somewhere for posterity because I sometimes need to recall these things months later.)

@Smaug123: Could you please point me to a break in DictImpl? I was not aware of that one.

This a good approach.

Those are the good breaking changes - if their breakiness can be detected by analyzing the source code.
Where this does not work and those are the riskiest changes - break cannot be detected by looking at source code only, and it changes runtime behavior.

Perfect example is the generic equality and how it works with floating point nan.

This is not proposal for any specific changes. It's is a discussion of possible approaches if library breaking changes make sense and we want to introduce them, how it should be done.

Any arguments made against breaking compatibility are pretty much defeated by the fact that we've already successfully navigated possibly the biggest transition in the .NET Framework -> .NET Core move.

If we could break compatibility with complete disregard for legacy code, imagine what we could do:

What you're suggesting is pretty much "Hey, let's not think of users with legacy code bases, who just update SDK and want their code just work, and instead do all sorts of breaking changes, for reasons.". It's not right approach here. There are a lot of F# projects out there which can't account such breaking changes, we for sure need to think about those.

• Just make Option<T> a struct

We can't "just" make it a struct, it will break all existing libraries which are using options and not rebuilt with latest FSharp.Core. If we ever to do that, it will need a bunch of work in compiler to account for that (pretty much make (v)option an intrinsic and deal with conversions during typechecking).

• Totally remove Async<T> in favour of Task<T>

They are not the interchangeable, they have different approach, cold vs hot, implicit vs explicit cancellation, composable vs not, etc.

• Make all tuples structs/C# compatible by default.

Not sure what that means, but probably less breaking than others.

• Tie nullability directly into C#-style nullability.

Also not sure what that means.

• Fix a lot of the syntactic warts from ML compatibility days

That was discussed many times, we wanted to hide a bunch of mlcompat under separate lexer/parser.

I would venture to suggest, that while keeping compatibility is a noble goal, in the sufficiently long term it's highly detrimental to the adoption of F#. Trying to explain to new developers how/why all this stuff exists that we shouldn't use and have to work around is really annoying.

if we do not actually do this, F# will continue to languish in its warty state, filled with thing that seemed like a good idea at the time.

Is it really is the intention of Microsoft to literally keep all of these problems around forever? that seems like an engineering approach that I and many others would fundamentally disagree with.

It's not, we're not literally keeping all of these problems around forever. This is exactly why this discussion is started.

I really don't want to have to be dealing with these legacy elements still in 10 years time - we might as well try to get rid of them sooner rather than later.

There are people with different projects and use-cases, you described yours - when you're fine if every major version language does breaking changes, both compile-time and runtime, and you're dealing with those. The other side of the coin is people who can't and won't deal with them, as it simply doesn't work for them.

Is is important to find a middle ground (which this discussion aims to do), and not just say "hey, let's just change things" or "hey, let's keep this issue foreve", as it's not productive.

Any arguments made against breaking compatibility are pretty much defeated by the fact that we've already successfully navigated possibly the biggest transition in the .NET Framework -> .NET Core move.

Just to note that the large majority of .NET Core work was in maintaining compatibility with most of .NET Framework, and especially in ensuring most libraries can get ported over to .NET Standard fairly painlessly. Of course there were some big breaks, but the reason why most people have been able to move their code over is explicitly because of legacy compatibility.

Here I meant the hot path which shouldn't be allocating on the heap at all to avoid any GC pauses (probably used the wrong term)

The first three points of that list would be quite easy to implement in a fork-of-fsharp.
Doing that in a way that would not rip the library ecosystem in half (see also: Python, Scala) is the hard part.

@EricM81, F# is only slowly dying from having passed the threshold of immortality, the slowness corresponds to that of the heat death of the universe and beyond, as it has been inscribed in the source of conscience itself.

For all your low level needs, you don't have to rely on FSharp.Core library, and you can contribute to the practices that makes F# suitable.

You can take the compiler frontend, and make a C backend for Fable, with all the things you mention, or look at the existing Rust backend.

For dotnet users, F# users enjoy productivity parties in high value codebases, that are all slowly dying at a slower rate than code which is not null safe, full of OO hierarchies, bloated with virtual dispatch, etc.

The community is welcoming the non breaking suggestions, and looking forward for your PRs 🙂.

I'd like a whole program optimization that elides option and allocations for closures, etc., as much as you do, but however this may or may not exist, the mighty CLR team is doing the ground breaking work so it is less of a concern, to 99.99% of F# users (I know, this is not so many of them).

For most existing F# users, I think compiler & tooling performance is above priority than beating rust at lower level code (C# isn't there, why expect F# to be?).

For what it's worth, recent improvements in the JIT can already make option allocations disappear altogether in some situations. For example:

open BenchmarkDotNet.Attributes
open BenchmarkDotNet.Configs
open BenchmarkDotNet.Environments
open BenchmarkDotNet.Jobs
open BenchmarkDotNet.Running

[<MemoryDiagnoser(displayGenColumns=false)>]
[<HideColumns("Job", "Error", "StdDev", "Median", "RatioSD")>]
type Benchmarks () =
    let x = Some 3

    [<Benchmark>]
    member _.DoSomeStuffWithOptions () =
        x
        |> Option.map ((+) 1)
        |> Option.map ((+) -1)
        |> Option.defaultValue 0

ignore <| BenchmarkRunner.Run<Benchmarks>
    (DefaultConfig.Instance
        .AddJob(Job.Default.AsBaseline().WithRuntime CoreRuntime.Core80)
        .AddJob(Job.Default.WithRuntime CoreRuntime.Core90))

| Method                 | Runtime  | Mean      | Ratio | Allocated | Alloc Ratio |
|----------------------- |--------- |----------:|------:|----------:|------------:|
| DoSomeStuffWithOptions | .NET 8.0 | 4.0503 ns |  1.01 |      48 B |        1.00 |
| DoSomeStuffWithOptions | .NET 9.0 | 0.4249 ns |  0.11 |         - |        0.00 |

It looks likely that we'll get more improvements along those lines in .NET 10. (Of course the JIT cannot solve everything, but it's not nothing.)

This is great, but then if you are storing objects in objects or passing them to non-trivial functions this goodness disappears, and then you end up with a mix of ValueOption and Option and it's just not pretty.

Personally I would recommend to just use Option and not be worried about garbage. It's prettier and more supported.

Like most of the time it just does not matter. Machines are amazingly fast. If you do care about performance though, maybe don't write your code in fsharp (but really we're talking about the 1% cases here - most performance is going to be algorithmically driven, not low level details)

I'd be very surprised if collection module functions are slower than Linq esp for a type like FSharpList. Linq adds I believe two layers of overhead. It uses reflection doesn't it? Plus it doesn't know about FSharpList and will need to access it via seq and then convert back for something like map or filter. Benchmarks needed. Frankly if the collection module functions are slow then anyone can make a PR to make them faster.

That said, if you care about performance you don't want to be using FSharpList in particular except in rare situations. Out of hundreds of thousands of places in our codebase where collections are created I believe there is only one place where a linked list is useful for performance.

Option is a key component of the F# language. Have you tried programming without it?

Yes. We use it everywhere except optional arguments and ValueOption optional arguments are in preview. F# has made a lot of steady progress towards having ValueOption usable everywhere and allowing struct representations of things in general, so these comments are out of date. Also you created 2 new threads about Option when there was one already in this discussion @EricM81 .

I'd be very surprised if collection module functions are slower than Linq esp for a type like FSharpList. Linq adds I believe two layers of overhead. It uses reflection doesn't it? Plus it doesn't know about FSharpList and will need to access it via seq and then convert back for something like map or filter. Benchmarks needed. Frankly if the collection module functions are slow then anyone can make a PR to make them faster.

The most obvious places are where LINQ uses SIMD but FSharp.Core can't (because it's netstandard). I tried this years ago and failed for that reason. Orders of magnitude speedups are possible for things like Array.sum, but not in FSharp.Core.

I'm curious about this work @Smaug123 . My understanding from playing around with Vector and with your code is that it applies only for a very short list of functions (like max, multiply, add, sin, etc.) on a very short list of types (including int, float, float32). It has an appalingly type-unsafe API. So I don't think it affects overall performance considerations of collection functions but is rather something specialist for a library which can be used explicitly, e.g. VectorOperations.max<'a when 'a is (int or float or float32?)>(span: ReadOnlySpan<'a>) (where the (int or float or float32?) resolves into either a type constraint which may be impossible to define, or multiple separate methods like maxFloat32). Perhaps can be added to Math.Net Numerics?

I'd be very surprised if collection module functions are slower than Linq esp for a type like FSharpList.

Be surprised then, because they are. System.Linq wraps functions like Select (Seq.map) and Where ((Seq.filter) with an internal type so that when you have a combination you just combine the two function calls, rather than accessing through the MoveNext(), Current mechanisms. It also carries things like the Count of the collection through calls to Select so that the size is available for allocations to calls like ToArray, hence requiring no temporary buffers. Also When temporary buffers are needed, the first few elements are stored in a stack allocated object, which also removes the need for temporary buffers.

Now, if you're talking about List.map then yes, it would be faster than Select() |> List.ofSeq, but you are kind of talking apples an oranges here, as all the List functions are eager, and will allocate the complete list before moving to the next step in the pipeline as opposed to the lazy evaluation of Seq/System.Linq.

Linq adds I believe two layers of overhead. It uses reflection doesn't it?

It does not. Unless you are using IQueryable, in which case you are doing somethingn completely different. Linq-to-objects is the Seq equivalent. Linq-to-database uses runtime reflection/tree manipulation to do things such as converting to SQL.

Plus it doesn't know about FSharpList and will need to access it via seq and then convert back for something like map or filter. Benchmarks needed. Frankly if the collection module functions are slow then anyone can make a PR to make them faster.

Go for your life. I have done many in the past, you can probably hunt some of them down in other github comments that I have written. Although the changes in platform's code generation has changed results.

That said, if you care about performance you don't want to be using FSharpList in particular except in rare situations.

Meh, the reason to use FSharp is FSharpList. Really. The joy in FSharp is pattern matching with lists. Just don't let performance concern you. I mean for the vast majority of cases it just doesn't matter. And if it does matter, I would highly recommend not using FSharp. You just end up fighting the language. So go ahead an embrace list, along with all the garbage it creates.

Out of hundreds of thousands of places in our codebase where collections are created I believe there is only one place where a linked list is useful for performance.

You don't use linked-lists for performance - or if you do, you don't use an immutable linked-list for performance... (OK, I lie, there can be some edge cases - if you are leaning heavily into into sharing roots of the lists.)

Look, FSharp is great. I love the language, It taught me heaps, and my c# is better for it, and I'm a better mentor to other developers because of it. I highly recommend that people spend 6 months coding it exclusively so that you really get to appreciate the beauty. BUT, I would not recommend it for anything that I care about performance for. Yes you can carve that bits for performance out and do them in another language, or bastardize the usage of fsharp to make it work (I know, I have), but its just harder to then maintain. c# is till missing discriminated unions, but

abstract record Base { 
    private Base() {}
    public record A...) : Base(...);
    public record B(...) : Base(...);
}  

thing switch
{
    Base.A a => ...,
    Base.B b => ...
    _ => throw ...
}

does a faily good job, and presumably will be easy to refactor into whatever final form that they will eventually choose.

(c# pattern matching via switch statement also work fine, although ending each block with break just makes me sad)

My understanding from playing around with Vector and with your code is that it applies only for a very short list of functions (like max, multiply, add, sin, etc.) on a very short list of types (including int, float, float32). It has an appalingly type-unsafe API. So I don't think it affects overall performance considerations of collection functions but is rather something specialist for a library which can be used explicitly...

LINQ just does it all behind the scenes, doing whatever type tests are required to ensure the dispatch is safe. You can use the low level Vector APIs, but you're expected just to let LINQ work out when it can speed things up.