GeezerVison is showing github.com — Lambdas, Closures and everything in between · Issue #1048 · ziglang/zig
Lambdas, Closures and everything in between #1048DescriptionI've been thinking about this topic for a few weeks now, and after quite a bit of research I think I have a solution that fits Zig's needs :).
This is building on #229 (and similar issues), since this is talking about implementation specifically and the issue is old I felt it was worth creating a new one.
Step 1: Make all functions anonymous; I.e. you can define a function like; const Foo = (bar: i32) i32 => { ... }; however of course we could support using fn Foo(bar: i32) i32 { ... } as the short form. In this way you can define functions inline, these aren't closure functions and when compiled will be placed outside the scope of course.
// i.e. to sort
sort(array, (a: i32, b: i32) bool => { return a < b; });Step 2: Lambdas; Building onto anonymous functions you can define 'inline' lambda functions which are just a single statement that returns like;
sort(array, ($) => $0 < $1); The $ just acts as a wild card effectively matching whatever the types the input requires, if the input is a 'var' type then it makes a function def that is fn X(a: var, b: var) var, perhaps? Or maybe that is just a compile error, I'm not sold either way.
Step 3: Closures; In the case where you actually want a closure you would define it like any other function but indicate type of function input;
var x = someRuntimeValue();
// 'Long form'
where(array, (a: i32, x = x) bool => { return a < x });
// And as a lambda (implicit 'var' return)
where(array, ($, x = x) => $0 < x));
// The above cases are by value, if you wanted by reference you would just do
where(array, ($, x = &x) => $0 < x));The above is synonymous to the following Zig code if we allow some kind of implicit cast;
const Closure = struct {
x: i32, // note: if doing the by value it would be `x: *i32`
f: fn(i32) bool,
};
where(array, Closure { .x = x, .f = (a: i32, closure: &const Closure) bool => { return a < closure.x } });HOWEVER, this is where the problem occurs, you require this pointer to exist in the definition, and so we need someway to get around this call and it's been suggested in the past that you can pass around some kind of 'closure' type that allows you to call it like a function but is really just this struct, personally this hides information from the coder and I feel goes against Zig's core, and furthermore would you allow the above 'closure' type to be passed into a function with definition (a: i32) bool?
Instead I propose that we can use LLVM Trampolining in quite a few cases to 'excise' a parameter from the call, which would be the closure information and the call would rather become something like;
const Closure = struct {
x: i32, // note: if doing the by value it would be `x: *i32`
// Note: no f
};
fn Foo(env: &Closure, a: i32) bool {
return a < env.x;
}
var x = runtimeValue();
const c = Closure { .x = x };
where(array, @trampoline(Foo, c));Note: of course in this case I'm using trampoline as if it was an inbuilt, I'm not actually sure if we want it as an inbuilt, but in reality it is more like generating the LLVM code that trampolines the function. A trampoline (said that word too many times now) just basically stores the value on the stack in advance to the call this would be much more efficient then a typical closure as it would avoid the need for a function pointer and would avoid the ugliness of indirection. HOWEVER, there is not much information on how this effects things like 'arrays of closures' which may instead require a different approach. Note: this is how GCC implements nested functions this is a relevant section.
So basically I propose that we approach this in the order I've given, and perhaps implement that builtin before integrating the closure x = x syntax. Reacted by Zetelias Reacted by bronze1man, RG, Spencer Brower and Marius Aasan Activitybnoordhuis commented on Jun 4, 2018 Trampolines require that the stack is marked executable. That's not the default on most systems and it reduces security, it makes injecting shellcode much easier.
I have some ideas on how to tackle closures but I don't want to derail this issue with counterproposals. Reacted by bronze1man, bb010g, adontz, The Potato Chronicler and Karl Hedberg bronze1man commented on Jun 4, 2018 How the closures is alloc?
How to free this alloc? BraedonWooding commented on Jun 4, 2018 ContributorAuthor More actions @bnoordhuis Huh, you got me there, completely forgot that damn. I guess the big challenge with this problem is trying to figure out an efficient solution that isn't just 'every closure gets a struct with a function ptr' since that just deliberately hides information from the coder. Counter proposals are fine since its more an implementation detail. The other idea I had was a 'fat ptr' like solution however that is just bloating a lot of calls, and forces us to use C style function calls (that allow extra parameters). BraedonWooding commented on Jun 4, 2018 ContributorAuthor More actions @bronze1man sorry I don't understand your questions? No need to free closures as we are talking about stack allocations, you can read up on trampolining if you really want the nitty gritty implementation detail. bheads commented on Jun 4, 2018 Could we use @newStackCall as part of the allocation.
var needle : i32 = 10;
var lambda = fn [needle] (x: var) bool { return x >needle; }; // yes cpp capture syntaxWhat I image is lambda is a struct, that has captured init stack instance and a function pointer:
struct {
stack: Struct { needle : i32 },
func: (x: var) bool,
}Calling the lambda would use newStackCall, I am just not sure how to put the stack into scope for the call. BraedonWooding commented on Jun 4, 2018 ContributorAuthor More actions
Could we use @newstackcall as part of the allocation.
Actually I think you are onto something, originally I dismissed your idea since I was like "nah that can't work", however upon considering it I definitely think it could work. What if you could bind a stack to a function, regardless of the functions purpose such as;
This is what the lambda would decompose to basically.
var stack : [8]u8;
var x : i32 = &stack[0]; // skipping cast to make it easier to understand
x.* = 10;
var lambda = fn (a: var) bool { return a < @stack(0, i32); }; // Longer syntax for clearness, @stack giving you the stackptr, then asking for an 'i32' integer
doWhatever(@bindstack(stack[4..], lambda, i32)); // @bindstack returns a function ptr, 4... to indicate stack begins from the '4'th positionBasically the language would have the concept of allowing you to bind a stack to a function, so there is no need to 'implicitly' cast. In effect yes it would look like that struct you gave (kinda) but instead of requiring some weird obscure cast or changing how functions work, we just allow functions to carry a different stack then the one we give it. Of course this requires the function to have a scope within the caller, if the scope is outside perhaps we allocate at start of program then deallocate at end?
As a side note I'm also not too sure about cpp syntax, it looks clunky; and is the cause of much confusion and bugs due to having some finer details that are a little odd. I prefer the more common style that is prevalent in languages like C#, Go, and Python for example since it is much clearer. Reacted by Igor Ferreira Reacted by lost22git bheads commented on Jun 4, 2018 I get the capture syntax is tricky when your learning but it was added to c++ for good reason. One of the good things about the capture syntax is it gives you control over what goes into the closure. This fits nicely with the zen and my personal rule of the principle of least surprise (ie, I have no idea what is getting capture in the closure and I have run into performance issues and bugs in D where it was capturing and trying to make deep copies of things I did not want/need.) Reacted by Stevie Hryciw, logicchains, Cezar Petriuc, Delyan Angelov and serid BraedonWooding commented on Jun 4, 2018 ContributorAuthor More actions Of course, however I still feel that the syntax I originally proposed is clearer i.e. (a: i32, x = x) => { ... } where x = x is by value and x = &x would be by reference; of course you could change the name of the variable to better match its context i.e. upperbound = x, and maybe could even just not include the new name if it is going to be the same therefore being (a: i32, =x, =&y). I guess it also comes down to not having the [] at the beginning of each one which kinda makes it look uglier.
Honestly a personal preference and we both agree that a clear copy/by ref is really needed which is important, which way it goes I think is not important right now; more important is figuring out how the lambda will be implemented. bheads commented on Jun 4, 2018 Ah sorry, I was not trying to defend the c++ syntax, just the concept of capturing. So yeah I am on the same page as you. bjornpagen commented on Jul 14, 2018 @BraedonWooding
I really like how neatly this proposal fits into Zig! Making all functions anonymous is a great way to prevent the language from becoming more complex, especially in implementing closures. It would probably be the best way to implement closures in Zig.
When I say that more discussion is needed before Zig gets closures, it has nothing to do with your proposal, only with the concept in general. I just don't think that this anonymous functions and closures are a good idea in Zig.
Zig aims to be a C alternative, not a C++ alternative. C does not have anonymous functions. Lambdas have not been added to C because over 45+ years, no one has ever felt they they needed them. Anonymous functions are simply not useful in imperative languages such as C, and Zig.
I love Zig because it aims to be an alternative for C, and it is the only language with this goal that is on the right track. As of now, Zig is small and simple, and fit for low-level/embedded/os development. Rust is horrible for this purpose: it is endlessly bloated with features, lang items, and 3 standard libraries (core, alloc, std). But individually, these features are very small, like Rust closures. But boy, do they add up, and make for a hell on earth.
So, unless you can come up with some real-life code examples of why we need lambdas, rather than just some quick convenience in sort(), I remain unconvinced.
Everyone, please give your own perspectives and ideas! I don't want to be "that guy", just spouting disapproval in every Zig github issue. ;) Reacted by Komu Wairagu, Davorin Šego, Davi, Giulio Phykos, xxxbxxx, 8176e68a, Yavuz Günay, RoyalDonkey, DerTee, skelewir and 4 more Reacted by Mattia Maldini, Igor Ferreira, jibal, valentin richard, Conscat, Nelson Lopez, Danny, Max Kamps, Sean Dewar, effinsky and 9 more Reacted by C. Tyler H. bjornpagen commented on Jul 14, 2018
that is obviously false, just cause its not in c does in no way mean no one needed it
Here are your examples written in current zig.
First:
fn foo(a : i32, b : i32) bool {
return a < b;
}
sort(some_arr, foo);
Second:
fn add(a: i32, b: i32) i32 {
return a+b;
}
assert(add(40,2) == 42);
I don't see the benefit of these functional programming features here. I'm not saying I hate closures: I just want to see an example of them being legitimately useful.
Zig zen bullet 4:
Only one obvious way to do things.
Reacted by bronze1man, Davorin Šego and Nicholas Montaño jarble commented on Oct 11, 2020 It's already possible to define a "closure" using a function within a struct. This program compiles successfully, for example:
pub fn main() void {
const j = 1;
var b = struct{
fn function(x: i32) i32 {
return x+j;
}
}.function;
var c = b(1);
}
ikskuh commented on Oct 11, 2020 @jarble Note that this only works as j is comptime known and thus can be captured at compile time. When you initialize j with a runtime-known value, this code will fail Reacted by jibal, JSmith AR, Oscar Pernia and qouteall phykos commented on Nov 29, 2020 Reacted by Mattia Maldini, hexaflex-zz, Adam Vandolder, Dag Ågren, Ehtesh Choudhury, Patrick George Wyndham Smith, Sean Dewar, Mirsario, ulzu, Alejandro Juan Pérez and 8 more leira commented on Feb 13, 2021
@jarble Note that this only works as j is comptime known and thus can be captured at compile time. When you initialize j with a runtime-known value, this code will fail
This works at runtime:
test "closure" {
var a: i32 = 1;
a += 1;
const addX = (struct {
a: i32,
fn call(self: @This(), x: i32) i32 {
return x + self.a;
}
} { .a = a }).call;
testing.expect(addX(2) == 4);
}
Reacted by Zack Kollar, JSmith AR and Martín Schonaker kevinswiber commented on Mar 5, 2021 @leira This works because of first-class function support in the language. For an example of where it falls short, notice that addX can't be passed as a parameter, as it's a bound function.
First-class functions without closure support can be a bit frustrating, which leads me to this issue in the first place. I was hoping to build parser combinators with function composition, and lack of closure support is kinda biting me, mostly because I was really determined to try to find a workaround but have so far been unsuccessful. 😅 Reacted by Helin Wang, Pradeep Verghese, Ken Okabe, Zzzen, IceSentry, JustAnotherCodemonkey, Anton Erdman, Timothy Harding, The Potato Chronicler, Jackie Li and 4 more iacore commented on Aug 21, 2021 JustAnotherCodemonkey commented on Sep 8, 2023 I don't know why Andrew marked as completed.
Maybe a little unrelated but it's a sticking point where someone mentions "I like to be able to create code dynamically like via macros. It dramatically eases some workflows" and points to Rust macros or even the C preprocessor and then you point out Zig's compiletime system and say "no need" but they say "for structs; what about functions?" and I really don't have much of an answer. I know being able to construct functions at compile time may be a bit much but to me it does seem very much possible with my understanding even without macros.
I agree with those that say that using structs to define anonymous functions is a major weakness and easily needlessly complicated. To those who say "why add a language feature when it's already possible", I point to the entire language and say "why add a language when it's already possible via C". Having a common feature be achieved via a workaround (even if small), especially if users of all levels will have to see, do, and deal with it is, in fact, a major weakness. We shouldn't need to use the struct workaround.
Honestly, when I first came to this language, I was extremely confused by the fact that functions aren't just all consts bound to anonymous functions / closures. It's very bizarre for a language seemingly on the surface as fluid and reflective as the ocean, be full of these hard, inflexible poles sticking out. mlugg commented on Sep 8, 2023
I don't know why Andrew marked as completed.
The issue close reasons is a pretty recent GitHub feature, and didn't exist at the time this was closed. (GitHub marked all old closures as "completed" when they added this feature.)
Functions returning functions is absolutely possible: comptime values can be captured into a nested struct scope, and logic in the function can use those comptime-known values to do conditional compilation.
fn isMultipleOfFn(comptime n: u64) fn (u64) bool {
return struct {
fn f(x: u64) bool {
return x % n == 0;
}
}.f;
}This is a little bit clunky, but that's kind of intentional. Zig to an extent discourages this kind of metaprogramming pattern, because logic is almost always significantly easier to write, understand, and debug when functions contain "direct" logic, even if it's a little longer. If you're using callbacks, it's almost always more correct to use a context struct with methods, to avoid requiring globals. But this functionality exists and works, and you can use it if it's genuinely the right tool.
For more complex code generation, the Zig build system makes it really easy to add custom build steps that generate Zig source code if required. Again, this isn't a thing you should have to do often, but if it feels like the right choice then there's nothing wrong with doing it!
As for the rest of your comment, it sounds a lot like you're discussing #1717: I recommend taking a look at the last comment on that issue for a bit more context as to why functions retain the syntax they have today. JustAnotherCodemonkey commented on Sep 8, 2023 Hm yes that does help to elaborate. I disagree but I can see a bit more. I still think that if we have try expressions, the anonymous function syntax is entirely reasonable.
Although I adore functional programming, I completely understand sticking to your imperative guns but there are many times where using callbacks is not only very warranted but necessary so I feel like punishing users for doing something they may have to do frequently if they're working with certain C libraries and many low-level APIs is not good. I'm specifically thinking about windowing and audio which tends to rely heavily on you supplying callbacks. Of course that's all very much possible with Zig and furthermore, naming those callbacks is a good idea but it makes the point that there are plenty of times that a user would have to supply a small lambda / callback and anonymous functions would make that so much easier.
I also don't like how they said that "anonymous functions don't show up in call stacks" because Rust proves this (like a lot of anti-anon-func arguments) not necessarily correct. Anonymous functions may not have names but that doesn't mean they can't have ID. In rust, the name and type of closures (closure are also unique anonymous types) is something like [closure@1e48baf89faaufdhsfundsu]. Also, having all functions be named is actually a drawback in of itself as small callback functions can pollute the namespace when they're only used for a single little thing.
Oh well I suppose mlugg commented on Sep 8, 2023
...but there are many times where using callbacks is not only very warranted but necessary...
In pretty much any scenario, plain callbacks actually aren't what you want: instead you want a context type. The core issue here is that Zig doesn't have closures, so you can't easily refer to locals scoped outside your callback. So let's say you're using some library which streams data to you in a callback, and you want to buffer all that data into an ArrayList. With simple function callbacks, that would look like this (in a fictional world with anon functions):
test {
var data_stream = try beginDataStream();
var bytes = std.ArrayList(u8).init(std.testing.allocator);
defer bytes.deinit();
try data_stream.stream(fn (data: []const u8) !void {
// Hmm...
});
}What do we put in the function body? We can't easily refer to bytes, because it's from an outer scope. We'd have to put a pointer to bytes into a global - this means the logic can't be reentrant and you also have to mark the global thread-local to avoid nasty bugs. Yuck!
C libraries generally handle this by allowing the callback to receive an arbitrary user-supplied pointer as a parameter, generally called the "user pointer" or "user data" or similar. This might be e.g. an integer you've cast to a pointer, or maybe it's an actual pointer to some bigger structure in memory. This solution totally works, but it comes at the cost of type-safety and may add unnecessary pointer indirections. It makes sense for C, but in Zig we can do better!
This is where context structs come in. Instead of taking a single function as our callback, we instead take a type with a method on it (we would usually put all the callbacks we need on that one type). We then take a value of that type, and call the methods on it. So to implement the above example, you do this:
var data_stream = try beginDataStream();
var bytes = std.ArrayList(u8).init(std.testing.allocator);
defer bytes.deinit();
const Context = struct {
out_bytes: *std.ArrayList(u8),
pub fn process(ctx: @This(), data: []const u8) !void {
try ctx.out_bytes.appendSlice(data);
}
};
try data_stream.stream(Context{ .out_bytes = &bytes });This solution gives us more type safety, improves clarity, and allows the context to be stored and passed directly, potentially avoiding an unnecessary pointer indirection.
I don't want to assert without evidence that this pattern is always applicable where you want to use a callback, but it definitely makes sense in every case I've seen. The fact is, plain anonymous functions in a language without closures aren't actually overly useful. They sometimes make sense for, say, sorting functions (to which you are providing a comparison function), but sorting can still make use of contexts sometimes, so it still makes sense for us to provide this more powerful (and still safe) API. Reacted by Frank Denis, JustAnotherCodemonkey, Mitchell Kember, Jon Åslund, InKryption, bronze1man, UnchartedSystems, Anton Chekulaev, Ian Ray, Austin Dworaczyk Wiltshire and 3 more ianprime0509 commented on Nov 5, 2023 @expikr container-level variables have static lifetimes, so unfortunately that pattern doesn't have the intended effect: all the "closures" created using that method share the same hidden_variable, as in the following example:
const print = @import("std").debug.print;
pub fn main() void {
const counter1 = counter(0);
counter1();
const counter2 = counter(5);
counter2();
counter1();
}
fn counter(j: usize) *const fn () void {
return (opaque {
var hidden_variable: usize = 0;
pub fn init(state: usize) *const @TypeOf(run) {
hidden_variable = state;
return &run;
}
fn run() void {
print("{}\n", .{hidden_variable});
hidden_variable += 1;
}
}).init(j);
}This prints 0, 5, 6, while it would be expected to print 0, 5, 1 if these were true closures. Using a context struct avoids this by leaving the caller in control of the memory for the state data. htqx commented on Apr 22, 2024 Key points of anonymous functions:
- Used as an expression
- The essence of closure is structure
- However, var members have side effects and are a poor design that should be avoided. (At least it shouldn't be too easy)
- Either a const member
- Either a field. Fields are passed as parameters
To summarize I recommend:
- Simple and easy-to-read syntax (for use as expressions)
- Don’t capture variables
// 1
const f : fn(i32,i32) i32 = (a,b) => a+b;
// 2
const Func = fn(i32,i32)i32; // Arbitrarily complex function types
const f: Func = (a,b)=>a+b;
// 3
const f = (a,b)=> a+b; // fn(a:anytype, b:anytype) @TypeOf(a,b) {return a+b;}
// 4
var a:i32 = 123;
var b:i32 = 456;
const f = (self)=> self.a + self.b;
_ = f(.{.a = a, .b = a}); // capture var
// 5
const a = 123;
const b = 456;
const f = ()=> a + b; // capture comptime const
// 6
const f = (a,b)=>{return a+b;} // Similar to below
//const f = struct { fn anonymous(a:anytype, b:anytype) @TypeOf(a,b) {return a + b;}}.anonymous;justin330 commented on Jun 4, 2024 Would this work?
fn someFunction() void {
// some local vars to capture
var x: u32 = 0;
const y: i32 = 10;
// proposed syntax A, like C++'s auto x = [&]() -> void {...}
const lambda = .{&x, &y}() void {
...
};
// or (preferably) proposed syntax B
const lambda = @lambda(.{&x, &y}, inline fn () void {
// pointer
capture[0].* = ...
// copy
capture[1] = ...
...
});
// which expands into this, for a single @lambda() call with unique captures: (works in 0.12)
const lambda = blk:{
const _capture = .{&x, y};
const Lmb_Ptr_x_Val_y = struct {
var capture: @TypeOf(_capture) = undefined;
inline fn func() void {
...
// pointer
capture[0].* = ...
// this is a copy
capture[1] = ...
}
};
Lmb_Ptr_x_Val_y.captures = _capture;
break:blk Lmb_Ptr_x_Val_y.func; // or &Lmb_Ptr_x_Val_y.func
};
lambda();Otherwise if more than a single lambda uses the same set of captures (and all of them being pointers), this could be expanded/optimised into:
var x: u32 = 0;
var y: i32 = 10;
// inserted directly below the last capture
const Lambda_Ptr_x_Ptr_y = struct {
var capture: @TypeOf(.{&x, &y}) = undefined;
inline fn funcA() void {
...
// pointer
capture[0].* = ...
// this is a copy
capture[1] = ...
}
inline fn funcB() void {
...
}
};
Lambda_Ptr_x_Ptr_y.capture = .{&x, y};
...
const lambda = Lambda_Ptr_x_Ptr_y.funcA;
const lambda_b = Lambda_Ptr_x_Ptr_y.funcB;Reacted by IgorGrabar, weim-o and Mehrad
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|