GeezerVison

I just pushed a commit to get stuff compiling. Excited to review this!

One thing to note is that some things are directly translation from PyPy, others from CPython's. Also there's that difference I said on Gitter, regarding declaring a class like:

class Custom(_SimpleCData):
   pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: class must define a '_type_' attribute

which here will succeed.
Besides this, I've tried to stay as close as CPython's behavior as possible.

What are the major blockers for this PR? I think I want to give this a try :)

What are the major blockers for this PR? I think I want to give this a try :)

Unfortunately there are several problems with the state of things and I haven't been able to continue working on this (and will not be able to do so for a while). The main problem is how it behaves differently from CPython's and PyPy's, a meta class should be added for Struct, CSimpleType, Pointer, ... (all but CFuntion), so that you can do c_int * 10 and return a proper array. In the current impl type(CSimpleType) == type, which does not has a __mul__ as required for a c_int, for instance.

@youknowone I tried to sync with master a while ago, but there was a problem and I got too busy to look at that again, so thanks for that! Are you planning into work on this? If so, I think there are some tests I ported from CPython that I altered just to see if the implementation was working-ish.

To be fully compatible to CPython and PyPy, the types must have a metaclass that's not PyType. I tried to work something there, but I couldn't find a solution. This happens because a base type has to implement some ops at type level, eg, c_int * 8 should call __mul__ from its meta (a yet non-existing class PrimitiveTypeMeta which is a subclass of PyType) and return a c_array.

I am happy to help with discussion and pointers, but I'm afraid I don't have much time to code :/

@darleybarreto Hi, thank you for the comment. I am not planning for working on ctypes at the moment, but looking for a way to merge it as it is for future contributors - because you already worked really a lot of it - if it doesn't break things too much. How do you think about it?

You mean merging as is?

yes, if we can manage it not to break CI

I think that we need to do a few things to help future contributions in this code before merging:

1. Refactor types structure.
2. Documentation.
3. A thorough review to improve code quality.

Undoubtedly 1 is the major blocker here. Since I don't know much of the internals of RustPython, I wasn't able to solve this problem. Perhaps you can give me pointers and I can work it out. PyPy does is basically this way:

class _CDataMeta(type):
    ...
    def __mul__(self, length):
        return create_array_type(self, length)
    ...

class SimpleType(_CDataMeta):
    ...

class SimpleCData(..., metaclass=SimpleType):
    ...

I couldn't find a way to make type(SimpleCData) be SimpleType instead of type.

Point 2 is extremely important since this code is basically too experimental and not that much well written. I could gladly do it based on PyPy, CPython and documentations.
Point 3 is somewhat important because I am not a very experienced Rust programmer, so if anyone could help me writing idiomatic and performant Rust, I would appreciate!

Thank you for detailed explanation. The metaclass issue is recently solved. Here is the test:

$ cat t.py 
class _CDataMeta(type):
    def __mul__(self, length):
        return create_array_type(self, length)

class SimpleType(_CDataMeta):
    pass

class SimpleCData(object, metaclass=SimpleType):
    pass


print(type(SimpleCData))
assert type(SimpleCData) == SimpleType
$ cargo run t.py 
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/rustpython t.py`
<class '__main__.SimpleType'>

I will look in the code for 3

Oh, I'm sorry, I meant to do the same thing in the Rust side. CPython does it in the C side of things by manually filling the type:

#define MOD_ADD_TYPE(TYPE_EXPR, TP_TYPE, TP_BASE) \
    do { \
        PyTypeObject *type = (TYPE_EXPR); \
        Py_SET_TYPE(type, TP_TYPE); \
        type->tp_base = TP_BASE; \
        if (PyModule_AddType(mod, type) < 0) { \
            return -1; \
        } \
    } while (0)

MOD_ADD_TYPE(&Simple_Type, &PyCSimpleType_Type, &PyCData_Type);

Here PyCSimpleType_Type is the meta and PyCData_Type is the base class.

by watching rustpython_vm::builtins::pytype::new, I feel like that'd be simply done by replacing typ field in PyObject, but not sure it actually will be that easy.
Because I don't understand this patch that much, would you make specific place in this PR that you need those type stuff? Then I will fill the fitting code there.

would you make specific place in this PR that you need those type stuff?

Let me get the CPython definitions as example

MOD_ADD_TYPE(&Struct_Type, &PyCStructType_Type, &PyCData_Type);
MOD_ADD_TYPE(&Union_Type, &UnionType_Type, &PyCData_Type);
MOD_ADD_TYPE(&PyCPointer_Type, &PyCPointerType_Type, &PyCData_Type);
MOD_ADD_TYPE(&PyCArray_Type, &PyCArrayType_Type, &PyCData_Type);
MOD_ADD_TYPE(&Simple_Type, &PyCSimpleType_Type, &PyCData_Type);
MOD_ADD_TYPE(&PyCFuncPtr_Type, &PyCFuncPtrType_Type, &PyCData_Type);

Here we have MOD_ADD_TYPE(&Class, &MetaClass, &BaseClass) right? Looking at Simple_Type (in RustPython this is PySimpleType in primitive.rs), we see that its metaclass implements the __mul__, so we can have c_int * 8 just fine, where c_int is subclass of Simple_Type. I used a trait PyCDataSequenceMethods in basics.rs to implement the __mul__, but doing impl PyCDataSequenceMethods for PySimpleTypeMeta {} in primitive.rs doesn't work because PySimpleTypeMeta is not the metaclass of PySimpleType.

I'm not sure why, but I cannot comment on the following reply

The safe expression of this type in Rust is Box<[u8]> which is exactly equivalent to this struct. Is this intended work to avoid somethiing like ownership check? I think there must be a better way.

The idea of RawBuffer is to be a "view" of the external buffer, so when needed, we copy the bytes from that buffer. I think there's one case we need to copy data and another where we need a view.

What happens when the buffer is destroyed? Does it deletes only the pointer(as a reference) or also deletes the data it contains (as an owner)? If it does both, then does it need a flag to distinguish them?

I forgot where these two functions are used, but based on the docs:
from_buffer_copy: C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer
from_buffer: C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer

@youknowone I basically patched all your comments, you wrote:

no, it isn't. slice is a sort of view

For this code

fn obj_bytes(&self) -> BorrowedValue<[u8]> {
    PyRwLockReadGuard::map(self.data.borrow_value(), |x| unsafe {
        slice::from_raw_parts(x.inner, x.size)
    })
    .into()
}

What did you meant there?

There was a question asking if it does copy. The comment is an answer about it. Creating an slice object doesn't copy anything.

I added a C file that should be compiled as a shared lib and bundled together with the interpreter. Its _ctypes_test.c. How would one make this happen? Would it be some sort of build script using cc? And we should be able to do this:

import _ctypes_test

build.rs file can contain any build script. Using cc crate will be helpful.

@youknowone I haven't figured out a nice way to implement the from_buffer. from_buffer_copy is straight forward, you simply create a new instance (default args for both tp_new and init) and copy the bytes from the value to create an instance. For example:

data = b'data'
ubyte = c_ubyte * len(data)
byteslike = ubyte.from_buffer_copy(data) # an instance of ubyte made from data by copying things

from_buffer does a similar thing, but it doesn't copy things, it points to the source. So if you change byteslike, you change data.

My implementation of from_buffer_copy is something in these lines

#[pyclassmethod]
fn from_buffer_copy(
    cls: PyRef<Self>,
    obj: PyObjectRef,
    offset: OptionalArg,
    vm: &VirtualMachine,
) -> PyResult {
    let buffer = try_buffer_from_object(vm, &obj)?;
    let opts = buffer.get_options().clone();

    let (size, offset) = Self::buffer_check(cls, opts, offset, vm); //ignore this for now

    let src_buffer = buffer.obj_bytes();
    let empty_instance = ... //creates a new empty instance
    let dst_buffer = empty_instance.obj_bytes_mut();

    dst_buffer.copy_from_slice(&src_buffer[offset..offset+size]);

    Ok(empty_instance.as_object().clone())
}

So the idea is simply get the bytes from obj (it should implement the buffer protocol) and point to them. How would I point to the data in obj instead of copying it? Should I use some refcounted type to avoid pointing to dangling things? obj should always be a buffer with write access.

I don't know well about your requirements, but I guess you can do same way as how PyMemoryView does. Or even just use it. it just holds a PyBufferRef and copy the data only when it needs to be. It also locks PyBufferRef when the lock is required.

I think this PR is (practically) not possible to rebase anymore in this state. Do you mind if we squash the commits into single commit for rebase?

I think this PR is (practically) not possible to rebase anymore in this state. Do you mind if we squash the commits into single commit for rebase?

Not at all :)

I'm afraid I can't contribute in the following months. I can help others eventually, tho.

Is this effort still ongoing / has this ctypes implementation been abandoned for another?

Hi folks, I think we should close this, it's too old for any merging attempt.
Happy to see @arihant2math tackling ctypes. I was actually look at this again (starting from scratch).
I'd like to help on designing the new implementation, but haven't got much spare time to code.

This is still a best trial of ctypes now. I wouldn't close it unless we actually merge any ctypes implementation.

For anyone who will try another ctypes implementation, I hope it can be gradually mergable, not to be suffered by huge size of rebase.

Agreed, the rebase effort was slowed by refactors that have happened since this pr, I'm planning to start incremental work on this soon.

Note that #5572 is now the tracking issue for further ctypes implementation.