GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing docs.kernel.org (truncated) — Memory Management APIs — The Linux Kernel documentation

English

Chinese (Simplified)

Memory Management APIs¶

User Space Memory Access¶

get_user¶

get_user (x, ptr)

Get a simple variable from user space.

Parameters

x: Variable to store result.
ptr: Source address, in user space.

Context

User context only. This function may sleep if pagefaults are enabled.

Description

This macro copies a single simple variable from user space to kernel space. It supports simple types like char and int, but not larger data types like structures or arrays.

ptr must have pointer-to-simple-variable type, and the result of dereferencing ptr must be assignable to x without a cast.

Return

zero on success, or -EFAULT on error. On error, the variable x is set to zero.

put_user¶

put_user (x, ptr)

Write a simple value into user space.

Parameters

x: Value to copy to user space.
ptr: Destination address, in user space.

Context

User context only. This function may sleep if pagefaults are enabled.

Description

This macro copies a single simple value from kernel space to user space. It supports simple types like char and int, but not larger data types like structures or arrays.

ptr must have pointer-to-simple-variable type, and x must be assignable to the result of dereferencing ptr.

Return

zero on success, or -EFAULT on error.

unsigned long __copy_to_user_inatomic(void __user *to, const void *from, unsigned long n)¶

Copy a block of data into user space, with less checking.

Parameters

void __user *to: Destination address, in user space.
const void *from: Source address, in kernel space.
unsigned long n: Number of bytes to copy.

Context

User context only.

Description

Copy data from kernel space to user space. Caller must check the specified block with access_ok() before calling this function. The caller should also make sure he pins the user space address so that we don’t result in page fault and sleep.

size_t probe_subpage_writeable(char __user *uaddr, size_t size)¶: probe the user range for write faults at sub-page granularity (e.g. arm64 MTE)

Parameters

char __user *uaddr: start of address range
size_t size: size of address range

Description

Returns 0 on success, the number of bytes not probed on fault.

It is expected that the caller checked for the write permission of each page in the range either by put_user() or GUP. The architecture port can implement a more efficient get_user() probing if the same sub-page faults are triggered by either a read or a write.

int copy_struct_from_user(void *dst, size_t ksize, const void __user *src, size_t usize)¶: copy a struct from userspace

Parameters

void *dst: Destination address, in kernel space. This buffer must be ksize bytes long.
size_t ksize: Size of dst struct.
const void __user *src: Source address, in userspace.
size_t usize: (Alleged) size of src struct.

Description

Copies a struct from userspace to kernel space, in a way that guarantees backwards-compatibility for struct syscall arguments (as long as future struct extensions are made such that all new fields are appended to the old struct, and zeroed-out new fields have the same meaning as the old struct).

ksize is just sizeof(dst), and **usize* should’ve been passed by userspace. The recommended usage is something like the following:

SYSCALL_DEFINE2(foobar, const struct foo __user *, uarg, size_t, usize)
{
   int err;
   struct foo karg = {};

   if (usize > PAGE_SIZE)
     return -E2BIG;
   if (usize < FOO_SIZE_VER0)
     return -EINVAL;

   err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize);
   if (err)
     return err;

   // ...
}

There are three cases to consider:

If usize == ksize, then it’s copied verbatim.
If usize < ksize, then the userspace has passed an old struct to a newer kernel. The rest of the trailing bytes in dst (ksize - usize) are to be zero-filled.
If usize > ksize, then the userspace has passed a new struct to an older kernel. The trailing bytes unknown to the kernel (usize - ksize) are checked to ensure they are zeroed, otherwise -E2BIG is returned.

Returns (in all cases, some data may have been copied):

-E2BIG: (usize > ksize) and there are non-zero trailing bytes in src.
-EFAULT: access to userspace failed.

int copy_struct_to_user(void __user *dst, size_t usize, const void *src, size_t ksize, bool *ignored_trailing)¶: copy a struct to userspace

Parameters

void __user *dst: Destination address, in userspace. This buffer must be ksize bytes long.
size_t usize: (Alleged) size of dst struct.
const void *src: Source address, in kernel space.
size_t ksize: Size of src struct.
bool *ignored_trailing: Set to true if there was a non-zero byte in src that userspace cannot see because they are using an smaller struct.

Description

Copies a struct from kernel space to userspace, in a way that guarantees backwards-compatibility for struct syscall arguments (as long as future struct extensions are made such that all new fields are appended to the old struct, and zeroed-out new fields have the same meaning as the old struct).

Some syscalls may wish to make sure that userspace knows about everything in the struct, and if there is a non-zero value that userspce doesn’t know about, they want to return an error (such as -EMSGSIZE) or have some other fallback (such as adding a “you’re missing some information” flag). If ignored_trailing is non-NULL, it will be set to true if there was a non-zero byte that could not be copied to userspace (ie. was past usize).

While unconditionally returning an error in this case is the simplest solution, for maximum backward compatibility you should try to only return -EMSGSIZE if the user explicitly requested the data that couldn’t be copied. Note that structure sizes can change due to header changes and simple recompilations without code changes(!), so if you care about ignored_trailing you probably want to make sure that any new field data is associated with a flag. Otherwise you might assume that a program knows about data it does not.

ksize is just sizeof(src), and **usize* should’ve been passed by userspace. The recommended usage is something like the following:

SYSCALL_DEFINE2(foobar, struct foo __user *, uarg, size_t, usize)
{
   int err;
   bool ignored_trailing;
   struct foo karg = {};

   if (usize > PAGE_SIZE)
           return -E2BIG;
   if (usize < FOO_SIZE_VER0)
           return -EINVAL;

   // ... modify karg somehow ...

   err = copy_struct_to_user(uarg, usize, &karg, sizeof(karg),
                             &ignored_trailing);
   if (err)
           return err;
   if (ignored_trailing)
           return -EMSGSIZE:

   // ...
}

There are three cases to consider:

If usize == ksize, then it’s copied verbatim.
If usize < ksize, then the kernel is trying to pass userspace a newer struct than it supports. Thus we only copy the interoperable portions (usize) and ignore the rest (but ignored_trailing is set to true if any of the trailing (ksize - usize) bytes are non-zero).
If usize > ksize, then the kernel is trying to pass userspace an older struct than userspace supports. In order to make sure the unknown-to-the-kernel fields don’t contain garbage values, we zero the trailing (usize - ksize) bytes.

Returns (in all cases, some data may have been copied):

-EFAULT: access to userspace failed.

get_kernel_nofault¶

get_kernel_nofault (val, ptr)

safely attempt to read from a location

Parameters

val: read into this variable
ptr: address to read from

Description

Returns 0 on success, or -EFAULT.

__scoped_user_access_begin¶

__scoped_user_access_begin (mode, uptr, size, elbl)

Start a scoped user access

Parameters

mode: The mode of the access class (read, write, rw)
uptr: The pointer to access user space memory
size: Size of the access
elbl: Error label to goto when the access region is rejected

Description

Internal helper for __scoped_user_access(). Don’t use directly.

__scoped_user_access¶

__scoped_user_access (mode, uptr, size, elbl)

Open a scope for user access

Parameters

mode: The mode of the access class (read, write, rw)
uptr: The pointer to access user space memory
size: Size of the access
elbl: Error label to goto when the access region is rejected. It must be placed outside the scope

Description

If the user access function inside the scope requires a fault label, it can use elbl or a different label outside the scope, which requires that user access which is implemented with ASM GOTO has been properly wrapped. See unsafe_get_user() for reference:

    scoped_user_rw_access(ptr, efault) {
            unsafe_get_user(rval, &ptr->rval, efault);
            unsafe_put_user(wval, &ptr->wval, efault);
    }
    return 0;
efault:
    return -EFAULT;

The scope is internally implemented as a autoterminating nested for() loop, which can be left with ‘return’, ‘break’ and ‘goto’ at any point.

When the scope is left user_##**_mode**##_access_end() is automatically invoked.

When the architecture supports masked user access and the access region which is determined by uptr and size is not a valid user space address, i.e. < TASK_SIZE, the scope sets the pointer to a faulting user space address and does not terminate early. This optimizes for the good case and lets the performance uncritical bad case go through the fault.

The eventual modification of the pointer is limited to the scope. Outside of the scope the original pointer value is unmodified, so that the original pointer value is available for diagnostic purposes in an out of scope fault path.

Nesting scoped user access into a user access scope is invalid and fails the build. Nesting into other guards, e.g. pagefault is safe.

The masked variant does not check the size of the access and relies on a mapping hole (e.g. guard page) to catch an out of range pointer, the first access to user memory inside the scope has to be within uptr ... uptr + PAGE_SIZE - 1

Don’t use directly. Use scoped_masked_user_$MODE_access() instead.

scoped_user_read_access_size¶

scoped_user_read_access_size (usrc, size, elbl)

Start a scoped user read access with given size

Parameters

usrc: Pointer to the user space address to read from
size: Size of the access starting from usrc
elbl: Error label to goto when the access region is rejected

Description

For further information see __scoped_user_access() above.

scoped_user_read_access¶

scoped_user_read_access (usrc, elbl)

Start a scoped user read access

Parameters

usrc: Pointer to the user space address to read from
elbl: Error label to goto when the access region is rejected

Description

The size of the access starting from usrc is determined via sizeof(*usrc)).

For further information see __scoped_user_access() above.

scoped_user_write_access_size¶

scoped_user_write_access_size (udst, size, elbl)

Start a scoped user write access with given size

Parameters

udst: Pointer to the user space address to write to
size: Size of the access starting from udst
elbl: Error label to goto when the access region is rejected

Description

For further information see __scoped_user_access() above.

scoped_user_write_access¶

scoped_user_write_access (udst, elbl)

Start a scoped user write access

Parameters

udst: Pointer to the user space address to write to
elbl: Error label to goto when the access region is rejected

Description

The size of the access starting from udst is determined via sizeof(*udst)).

For further information see __scoped_user_access() above.

scoped_user_rw_access_size¶

scoped_user_rw_access_size (uptr, size, elbl)

Start a scoped user read/write access with given size

Parameters

uptr: Pointer to the user space address to read from and write to
size: Size of the access starting from uptr
elbl: Error label to goto when the access region is rejected

Description

For further information see __scoped_user_access() above.

scoped_user_rw_access¶

scoped_user_rw_access (uptr, elbl)

Start a scoped user read/write access

Parameters

uptr: Pointer to the user space address to read from and write to
elbl: Error label to goto when the access region is rejected

Description

The size of the access starting from uptr is determined via sizeof(*uptr)).

For further information see __scoped_user_access() above.

get_user_inline¶

get_user_inline (val, usrc)

Read user data inlined

Parameters

val: The variable to store the value read from user memory
usrc: Pointer to the user space memory to read from

Return

0 if successful, -EFAULT when faulted

Description

Inlined variant of get_user(). Only use when there is a demonstrable performance reason.

put_user_inline¶

put_user_inline (val, udst)

Write to user memory inlined

Parameters

val: The value to write
udst: Pointer to the user space memory to write to

Return

0 if successful, -EFAULT when faulted

Description

Inlined variant of put_user(). Only use when there is a demonstrable performance reason.

unsigned long clear_user(void __user *to, unsigned long n)¶: Zero a block of memory in user space.

Parameters

void __user *to: Destination address, in user space.
unsigned long n: Number of bytes to zero.

Description

Zero a block of memory in user space.

Return

number of bytes that could not be cleared. On success, this will be zero.

unsigned long __clear_user(void __user *to, unsigned long n)¶: Zero a block of memory in user space, with less checking.

Parameters

void __user *to: Destination address, in user space.
unsigned long n: Number of bytes to zero.

Description

Zero a block of memory in user space. Caller must check the specified block with access_ok() before calling this function.

Return

number of bytes that could not be cleared. On success, this will be zero.

int get_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages)¶: pin user pages in memory

Parameters

unsigned long start: starting user address
int nr_pages: number of pages from start to pin
unsigned int gup_flags: flags modifying pin behaviour
struct page **pages: array that receives pointers to the pages pinned. Should be at least nr_pages long.

Description

Attempt to pin user pages in memory without taking mm->mmap_lock. If not successful, it will fall back to taking the lock and calling get_user_pages().

Returns number of pages pinned. This may be fewer than the number requested. If nr_pages is 0 or negative, returns 0. If no pages were pinned, returns -errno.

Memory Allocation Controls¶

Page mobility and placement hints¶

These flags provide hints about how mobile the page is. Pages with similar mobility are placed within the same pageblocks to minimise problems due to external fragmentation.

__GFP_MOVABLE (also a zone modifier) indicates that the page can be moved by page migration during memory compaction or can be reclaimed.

__GFP_RECLAIMABLE is used for slab allocations that specify SLAB_RECLAIM_ACCOUNT and whose pages can be freed via shrinkers.

__GFP_WRITE indicates the caller intends to dirty the page. Where possible, these pages will be spread between local zones to avoid all the dirty pages being in one zone (fair zone allocation policy).

__GFP_HARDWALL enforces the cpuset memory allocation policy.

__GFP_THISNODE forces the allocation to be satisfied from the requested node with no fallbacks or placement policy enforcements.

__GFP_ACCOUNT causes the allocation to be accounted to the active cgroup context.

Watermark modifiers -- controls access to emergency reserves¶

__GFP_HIGH indicates that the caller is high-priority and that granting the request is necessary before the system can make forward progress. For example creating an IO context to clean pages and requests from atomic context.

__GFP_MEMALLOC allows access to all memory. This should only be used when the caller guarantees the allocation will allow more memory to be freed very shortly e.g. process exiting or swapping. Users either should be the MM or co-ordinating closely with the VM (e.g. swap over NFS). Users of this flag have to be extremely careful to not deplete the reserve completely and implement a throttling mechanism which controls the consumption of the reserve based on the amount of freed memory. Usage of a pre-allocated pool (e.g. mempool) should be always considered before using this flag.

__GFP_NOMEMALLOC is used to explicitly forbid access to emergency reserves. This takes precedence over the __GFP_MEMALLOC flag if both are set.

Reclaim modifiers¶

Please note that all the following flags are only applicable to sleepable allocations (e.g. GFP_NOWAIT and GFP_ATOMIC will ignore them).

__GFP_IO can start physical IO.

__GFP_FS can call down to the low-level FS. Clearing the flag avoids the allocator recursing into the filesystem which might already be holding locks.

__GFP_DIRECT_RECLAIM indicates that the caller may enter direct reclaim. This flag can be cleared to avoid unnecessary delays when a fallback option is available.

__GFP_KSWAPD_RECLAIM indicates that the caller wants to wake kswapd when the low watermark is reached and have it reclaim pages until the high watermark is reached. A caller may wish to clear this flag when fallback options are available and the reclaim is likely to disrupt the system. The canonical example is THP allocation where a fallback is cheap but reclaim/compaction may cause indirect stalls.

__GFP_RECLAIM is shorthand to allow/forbid both direct and kswapd reclaim.

The default allocator behavior depends on the request size. We have a concept of so-called costly allocations (with order > PAGE_ALLOC_COSTLY_ORDER). !costly allocations are too essential to fail so they are implicitly non-failing by default (with some exceptions like OOM victims might fail so the caller still has to check for failures) while costly requests try to be not disruptive and back off even without invoking the OOM killer. The following three modifiers might be used to override some of these implicit rules. Please note that all of them must be used along with __GFP_DIRECT_RECLAIM flag.

__GFP_NORETRY: The VM implementation will try only very lightweight memory direct reclaim to get some memory under memory pressure (thus it can sleep). It will avoid disruptive actions like OOM killer. The caller must handle the failure which is quite likely to happen under heavy memory pressure. The flag is suitable when failure can easily be handled at small cost, such as reduced throughput.

__GFP_RETRY_MAYFAIL: The VM implementation will retry memory reclaim procedures that have previously failed if there is some indication that progress has been made elsewhere. It can wait for other tasks to attempt high-level approaches to freeing memory such as compaction (which removes fragmentation) and page-out. There is still a definite limit to the number of retries, but it is a larger limit than with __GFP_NORETRY. Allocations with this flag may fail, but only when there is genuinely little unused memory. While these allocations do not directly trigger the OOM killer, their failure indicates that the system is likely to need to use the OOM killer soon. The caller must handle failure, but can reasonably do so by failing a higher-level request, or completing it only in a much less efficient manner. If the allocation does fail, and the caller is in a position to free some non-essential memory, doing so could benefit the system as a whole.

__GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller cannot handle allocation failures. The allocation could block indefinitely but will never return with failure. Testing for failure is pointless. It _must_ be blockable and used together with __GFP_DIRECT_RECLAIM. It should _never_ be used in non-sleepable contexts. New users should be evaluated carefully (and the flag should be used only when there is no reasonable failure policy) but it is definitely preferable to use the flag rather than opencode endless loop around allocator. Allocating pages from the buddy with __GFP_NOFAIL and order > 1 is discouraged. Please consider using kvmalloc() instead if possible.

Useful GFP flag combinations¶

Useful GFP flag combinations that are commonly used. It is recommended that subsystems start with one of these combinations and then set/clear __GFP_FOO flags as necessary.

GFP_ATOMIC users can not sleep and need the allocation to succeed. A lower watermark is applied to allow access to “atomic reserves”. The current implementation doesn’t support NMI, nor contexts that disable preemption under PREEMPT_RT. This includes raw_spin_lock() and plain preempt_disable() - see “Memory allocation” in How realtime kernels differ for more info.

GFP_KERNEL is typical for kernel-internal allocations. The caller requires ZONE_NORMAL or a lower zone for direct access but can direct reclaim.

GFP_KERNEL_ACCOUNT is the same as GFP_KERNEL, except the allocation is accounted to the active cgroup context.

GFP_NOWAIT is for kernel allocations that should not stall for direct reclaim, start physical IO or use any filesystem callback. It is very likely to fail to allocate memory, even for very small allocations. The same restrictions on calling contexts apply as for GFP_ATOMIC.

GFP_NOIO will use direct reclaim to discard clean pages or slab pages that do not require the starting of any physical IO. Please try to avoid using this flag directly and instead use memalloc_noio_{save,restore} to mark the whole scope which cannot perform any IO with a short explanation why. All allocation requests will inherit GFP_NOIO implicitly.

GFP_NOFS will use direct reclaim but will not use any filesystem interfaces. Please try to avoid using this flag directly and instead use memalloc_nofs_{save,restore} to mark the whole scope which cannot/shouldn’t recurse into the FS layer with a short explanation why. All allocation requests will inherit GFP_NOFS implicitly.

GFP_USER is for userspace allocations that also need to be directly accessibly by the kernel or hardware. It is typically used by hardware for buffers that are mapped to userspace (e.g. graphics) that hardware still must DMA to. cpuset limits are enforced for these allocations.

GFP_DMA exists for historical reasons and should be avoided where possible. The flags indicates that the caller requires that the lowest zone be used (ZONE_DMA or 16M on x86-64). Ideally, this would be removed but it would require careful auditing as some users really require it and others use the flag to avoid lowmem reserves in ZONE_DMA and treat the lowest zone as a type of emergency reserve.

GFP_DMA32 is similar to GFP_DMA except that the caller requires a 32-bit address. Note that kmalloc(..., GFP_DMA32) does not return DMA32 memory because the DMA32 kmalloc cache array is not implemented. (Reason: there is no such user in kernel).

GFP_HIGHUSER is for userspace allocations that may be mapped to userspace, do not need to be directly accessible by the kernel but that cannot move once in use. An example may be a hardware allocation that maps data directly into userspace but has no addressing limitations.

GFP_HIGHUSER_MOVABLE is for userspace allocations that the kernel does not need direct access to but can use kmap() when access is required. They are expected to be movable via page reclaim or page migration. Typically, pages on the LRU would also be allocated with GFP_HIGHUSER_MOVABLE.

GFP_TRANSHUGE and GFP_TRANSHUGE_LIGHT are used for THP allocations. They are compound allocations that will generally fail quickly if memory is not available and will not wake kswapd/kcompactd on failure. The _LIGHT version does not attempt reclaim/compaction at all and is by default used in page fault path, while the non-light is used by khugepaged.

The Slab Cache¶

SLAB_HWCACHE_ALIGN¶

SLAB_HWCACHE_ALIGN

Align objects on cache line boundaries.

Description

Sufficiently large objects are aligned on cache line boundary. For object size smaller than a half of cache line size, the alignment is on the half of cache line size. In general, if object size is smaller than 1/2^n of cache line size, the alignment is adjusted to 1/2^n.

If explicit alignment is also requested by the respective struct kmem_cache_args field, the greater of both is alignments is applied.

SLAB_TYPESAFE_BY_RCU¶

SLAB_TYPESAFE_BY_RCU

WARNING READ THIS!

Description

This delays freeing the SLAB page by a grace period, it does _NOT_ delay object freeing. This means that if you do kmem_cache_free() that memory location is free to be reused at any time. Thus it may be possible to see another object there in the same RCU grace period.

This feature only ensures the memory location backing the object stays valid, the trick to using this is relying on an independent object validation pass. Something like:
begin:
 rcu_read_lock();
 obj = lockless_lookup(key);
 if (obj) {
   if (!try_get_ref(obj)) // might fail for free objects
     rcu_read_unlock();
     goto begin;

   if (obj->key != key) { // not the object we expected
     put_ref(obj);
     rcu_read_unlock();
     goto begin;
   }
 }
rcu_read_unlock();
This is useful if we need to approach a kernel structure obliquely, from its address obtained without the usual locking. We can lock the structure to stabilize it and check it’s still at the given address, only if we can be sure that the memory has not been meanwhile reused for some other kind of object (which our subsystem’s lock might corrupt).

rcu_read_lock before reading the address, then rcu_read_unlock after taking the spinlock within the structure expected at that address.

Note that object identity check has to be done after acquiring a reference, therefore user has to ensure proper ordering for loads. Similarly, when initializing objects allocated with SLAB_TYPESAFE_BY_RCU, the newly allocated object has to be fully initialized before its refcount gets initialized and proper ordering for stores is required. refcount_{add|inc}_not_zero_acquire() and refcount_set_release() are designed with the proper fences required for reference counting objects allocated with SLAB_TYPESAFE_BY_RCU.

Note that it is not possible to acquire a lock within a structure allocated with SLAB_TYPESAFE_BY_RCU without first acquiring a reference as described above. The reason is that SLAB_TYPESAFE_BY_RCU pages are not zeroed before being given to the slab, which means that any locks must be initialized after each and every kmem_struct_alloc(). Alternatively, make the ctor passed to kmem_cache_create() initialize the locks at page-allocation time, as is done in __i915_request_ctor(), sighand_ctor(), and anon_vma_ctor(). Such a ctor permits readers to safely acquire those ctor-initialized locks under rcu_read_lock() protection.

Note that SLAB_TYPESAFE_BY_RCU was originally named SLAB_DESTROY_BY_RCU.

SLAB_ACCOUNT¶

SLAB_ACCOUNT

Account allocations to memcg.

Description

All object allocations from this cache will be memcg accounted, regardless of __GFP_ACCOUNT being or not being passed to individual allocations.

SLAB_RECLAIM_ACCOUNT¶

SLAB_RECLAIM_ACCOUNT

Objects are reclaimable.

Description

Use this flag for caches that have an associated shrinker. As a result, slab pages are allocated with __GFP_RECLAIMABLE, which affects grouping pages by mobility, and are accounted in SReclaimable counter in /proc/meminfo

struct kmem_cache_args¶: Less common arguments for kmem_cache_create()

Definition:

struct kmem_cache_args {
    unsigned int align;
    unsigned int useroffset;
    unsigned int usersize;
    unsigned int freeptr_offset;
    bool use_freeptr_offset;
    void (*ctor)(void *);
    unsigned int sheaf_capacity;
};

Members

align

The required alignment for the objects.

0 means no specific alignment is requested.

useroffset

Usercopy region offset.

0 is a valid offset, when usersize is non-0

usersize

Usercopy region size.

0 means no usercopy region is specified.

freeptr_offset

Custom offset for the free pointer in caches with SLAB_TYPESAFE_BY_RCU or ctor

By default, SLAB_TYPESAFE_BY_RCU and ctor caches place the free pointer outside of the object. This might cause the object to grow in size. Cache creators that have a reason to avoid this can specify a custom free pointer offset in their data structure where the free pointer will be placed.

For caches with SLAB_TYPESAFE_BY_RCU, the caller must ensure that the free pointer does not overlay fields required to guard against object recycling (See SLAB_TYPESAFE_BY_RCU for details).

For caches with ctor, the caller must ensure that the free pointer does not overlay fields initialized by the constructor.

Currently, only caches with SLAB_TYPESAFE_BY_RCU or ctor may specify freeptr_offset.

Using 0 as a value for freeptr_offset is valid. If freeptr_offset is specified, use_freeptr_offset must be set true.

use_freeptr_offset

Whether a freeptr_offset is used.

ctor

A constructor for the objects.

The constructor is invoked for each object in a newly allocated slab page. It is the cache user’s responsibility to free object in the same state as after calling the constructor, or deal appropriately with any differences between a freshly constructed and a reallocated object.

NULL means no constructor.

sheaf_capacity

Enable sheaves of given capacity for the cache.

With a non-zero value, allocations from the cache go through caching arrays called sheaves. Each cpu has a main sheaf that’s always present, and a spare sheaf that may be not present. When both become empty, there’s an attempt to replace an empty sheaf with a full sheaf from the per-node barn.

When no full sheaf is available, and gfp flags allow blocking, a sheaf is allocated and filled from slab(s) using bulk allocation. Otherwise the allocation falls back to the normal operation allocating a single object from a slab.

Analogically when freeing and both percpu sheaves are full, the barn may replace it with an empty sheaf, unless it’s over capacity. In that case a sheaf is bulk freed to slab pages.

The sheaves do not enforce NUMA placement of objects, so allocations via kmem_cache_alloc_node() with a node specified other than NUMA_NO_NODE will bypass them.

Bulk allocation and free operations also try to use the cpu sheaves and barn, but fallback to using slab pages directly.

When slub_debug is enabled for the cache, the sheaf_capacity argument is ignored.

0 means no sheaves will be created.

Description

Any uninitialized fields of the structure are interpreted as unused. The exception is freeptr_offset where 0 is a valid value, so use_freeptr_offset must be also set to true in order to interpret the field as used. For useroffset 0 is also valid, but only with non-0 usersize.

When NULL args is passed to kmem_cache_create(), it is equivalent to all fields unused.

struct kmem_cache *kmem_cache_create_usercopy(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void*))¶: Create a kmem cache with a region suitable for copying to userspace.

Parameters

const char *name: A string which is used in /proc/slabinfo to identify this cache.
unsigned int size: The size of objects to be created in this cache.
unsigned int align: The required alignment for the objects.
slab_flags_t flags: SLAB flags
unsigned int useroffset: Usercopy region offset
unsigned int usersize: Usercopy region size
void (*ctor)(void *): A constructor for the objects, or NULL.

Description

This is a legacy wrapper, new code should use either KMEM_CACHE_USERCOPY() if whitelisting a single field is sufficient, or kmem_cache_create() with the necessary parameters passed via the args parameter (see struct kmem_cache_args)

Return

a pointer to the cache on success, NULL on failure.

kmem_cache_create¶

kmem_cache_create (__name, __object_size, __args, ...)

Create a kmem cache.

Parameters

__name: A string which is used in /proc/slabinfo to identify this cache.
__object_size: The size of objects to be created in this cache.
__args: Optional arguments, see struct kmem_cache_args. Passing NULL means defaults will be used for all the arguments.
...: variable arguments

Description

This is currently implemented as a macro using _Generic() to call either the new variant of the function, or a legacy one.

The new variant has 4 parameters: kmem_cache_create(name, object_size, args, flags)

See __kmem_cache_create_args() which implements this.

The legacy variant has 5 parameters: kmem_cache_create(name, object_size, align, flags, ctor)

The align and ctor parameters map to the respective fields of struct kmem_cache_args

Context

Cannot be called within a interrupt, but can be interrupted.

Return

a pointer to the cache on success, NULL on failure.

void *krealloc_node_align(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid)¶: reallocate memory. The contents will remain unchanged.

Parameters

const void *p: object to reallocate memory for.
size_t new_size: how many bytes of memory are required.
unsigned long align: desired alignment.
gfp_t flags: the type of memory to allocate.
int nid: NUMA node or NUMA_NO_NODE

Description

If p is NULL, krealloc() behaves exactly like kmalloc(). If new_size is 0 and p is not a NULL pointer, the object pointed to is freed.

Only alignments up to those guaranteed by kmalloc() will be honored. Please see Memory Allocation Guide for more details.

If __GFP_ZERO logic is requested, callers must ensure that, starting with the initial memory allocation, every subsequent call to this API for the same memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that __GFP_ZERO is not fully honored by this API.

When slub_debug_orig_size() is off, krealloc() only knows about the bucket size of an allocation (but not the exact size it was allocated with) and hence implements the following semantics for shrinking and growing buffers with __GFP_ZERO:

        new             bucket
0       size             size
|--------|----------------|
|  keep  |      zero      |

Otherwise, the original allocation size ‘orig_size’ could be used to precisely clear the requested size, and the new size will also be stored as the new ‘orig_size’.

In any case, the contents of the object pointed to are preserved up to the lesser of the new and old sizes.

Return

pointer to the allocated memory or NULL in case of error

void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)¶: Allocate an object

Parameters

struct kmem_cache *cachep: The cache to allocate from.
gfp_t flags: See kmalloc().

Description

Allocate an object from this cache. See kmem_cache_zalloc() for a shortcut of adding __GFP_ZERO to flags.

Return

pointer to the new object or NULL in case of error

bool kmem_cache_charge(void *objp, gfp_t gfpflags)¶: memcg charge an already allocated slab memory

Parameters

void *objp: address of the slab object to memcg charge
gfp_t gfpflags: describe the allocation context

Description

kmem_cache_charge allows charging a slab object to the current memcg, primarily in cases where charging at allocation time might not be possible because the target memcg is not known (i.e. softirq context)

The objp should be pointer returned by the slab allocator functions like kmalloc (with __GFP_ACCOUNT in flags) or kmem_cache_alloc. The memcg charge behavior can be controlled through gfpflags parameter, which affects how the necessary internal metadata can be allocated. Including __GFP_NOFAIL denotes that overcharging is requested instead of failure, but is not applied for the internal metadata allocation.

There are several cases where it will return true even if the charging was not done: More specifically:

For !CONFIG_MEMCG or cgroup_disable=memory systems.
Already charged slab objects.
For slab objects from KMALLOC_NORMAL caches - allocated by kmalloc() without __GFP_ACCOUNT
Allocating internal metadata has failed

Return

true if charge was successful otherwise false.

void *kmalloc(size_t size, gfp_t flags)¶: allocate kernel memory

Parameters

size_t size: how many bytes of memory are required.
gfp_t flags: describe the allocation context

Description

kmalloc is the normal method of allocating memory for objects smaller than page size in the kernel.

The allocated object address is aligned to at least ARCH_KMALLOC_MINALIGN bytes. For size of power of two bytes, the alignment is also guaranteed to be at least to the size. For other sizes, the alignment is guaranteed to be at least the largest power-of-two divisor of size.

The flags argument may be one of the GFP flags defined at include/linux/gfp_types.h and described at Documentation/core-api/mm-api.rst

The recommended usage of the flags is described at Documentation/core-api/memory-allocation.rst

Below is a brief outline of the most useful GFP flags

GFP_KERNEL: Allocate normal kernel ram. May sleep.
GFP_NOWAIT: Allocation will not sleep.
GFP_ATOMIC: Allocation will not sleep. May use emergency pools.

Also it is possible to set different flags by OR’ing in one or more of the following additional flags:

__GFP_ZERO: Zero the allocated memory before returning. Also see kzalloc().
__GFP_HIGH: This allocation has high priority and may use emergency pools.
__GFP_NOFAIL: Indicate that this allocation is in no way allowed to fail (think twice before using).
__GFP_NORETRY: If memory is not immediately available, then give up at once.
__GFP_NOWARN: If allocation fails, don’t issue any warnings.
__GFP_RETRY_MAYFAIL: Try really hard to succeed the allocation but fail eventually.

void *kmalloc_nolock(size_t size, gfp_t gfp_flags, int node)¶: Allocate an object of given size from any context.

Parameters

size_t size: size to allocate
gfp_t gfp_flags: GFP flags. Only __GFP_ACCOUNT and __GFP_ZERO allowed. Also __GFP_NOWARN and __GFP_NOMEMALLOC are allowed but added internally thus not necessary.
int node: node number of the target node.

Return

pointer to the new object or NULL in case of error. NULL does not mean EBUSY or EAGAIN. It means ENOMEM. There is no reason to call it again and expect !NULL.

__alloc_objs¶

__alloc_objs (KMALLOC, GFP, TYPE, COUNT)

Allocate objects of a given type using

Parameters

KMALLOC: which size-based kmalloc wrapper to allocate with.
GFP: GFP flags for the allocation.
TYPE: type to allocate space for.
COUNT: how many TYPE objects to allocate.

Return

Newly allocated pointer to (first) TYPE of COUNT-many allocated TYPE objects, or NULL on failure.

__alloc_flex¶

__alloc_flex (KMALLOC, GFP, TYPE, FAM, COUNT)

Allocate an object that has a trailing flexible array

Parameters

KMALLOC: kmalloc wrapper function to use for allocation.
GFP: GFP flags for the allocation.
TYPE: type of structure to allocate space for.
FAM: The name of the flexible array member of TYPE structure.
COUNT: how many FAM elements to allocate space for.

Return

Newly allocated pointer to TYPE with COUNT-many trailing FAM elements, or NULL on failure or if COUNT cannot be represented by the member of TYPE that counts the FAM elements (annotated via __counted_by()).

kmalloc_obj¶

kmalloc_obj (VAR_OR_TYPE, ...)

Allocate a single instance of the given type

Parameters

VAR_OR_TYPE: Variable or type to allocate.
...: optional GFP flags for the allocation (GFP_KERNEL when not specified).

Return

newly allocated pointer to a VAR_OR_TYPE on success, or NULL on failure.

kmalloc_objs¶

kmalloc_objs (VAR_OR_TYPE, COUNT, ...)

Allocate an array of the given type

Parameters

VAR_OR_TYPE: Variable or type to allocate an array of.
COUNT: How many elements in the array.
...: optional GFP flags for the allocation (GFP_KERNEL when not specified).

Return

newly allocated pointer to array of VAR_OR_TYPE on success, or NULL on failure.

kmalloc_flex¶

kmalloc_flex (VAR_OR_TYPE, FAM, COUNT, ...)

Allocate a single instance of the given flexible structure

Parameters

VAR_OR_TYPE: Variable or type to allocate (with its flex array).
FAM: The name of the flexible array member of the structure.
COUNT: How many flexible array member elements are desired.
...: optional GFP flags for the allocation (GFP_KERNEL when not specified).

Return

newly allocated pointer to VAR_OR_TYPE on success, NULL on failure. If FAM has been annotated with __counted_by(), the allocation will immediately fail if COUNT is larger than what the type of the struct’s counter variable can represent.

void *kmalloc_array(size_t n, size_t size, gfp_t flags)¶: allocate memory for an array.

Parameters

size_t n: number of elements.
size_t size: element size.
gfp_t flags: the type of memory to allocate (see kmalloc).

void *krealloc_array(void *p, size_t new_n, size_t new_size, gfp_t flags)¶: reallocate memory for an array.

Parameters

void *p: pointer to the memory chunk to reallocate
size_t new_n: new number of elements to alloc
size_t new_size: new size of a single member of the array
gfp_t flags: the type of memory to allocate (see kmalloc)

Description

See krealloc_noprof() for further details.

In any case, the contents of the object pointed to are preserved up to the lesser of the new and old sizes.

kcalloc¶

kcalloc (n, size, flags)

allocate memory for an array. The memory is set to zero.

Parameters

n: number of elements.
size: element size.
flags: the type of memory to allocate (see kmalloc).

void *kzalloc(size_t size, gfp_t flags)¶: allocate memory. The memory is set to zero.

Parameters

size_t size: how many bytes of memory are required.
gfp_t flags: the type of memory to allocate (see kmalloc).

void *kvmalloc_node(size_t size, gfp_t flags, int node)¶: attempt to allocate physically contiguous memory, but upon failure, fall back to non-contiguous (vmalloc) allocation.

Parameters

size_t size: size of the request.
gfp_t flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
int node: numa node to allocate from

Description

Only alignments up to those guaranteed by kmalloc() will be honored. Please see Memory Allocation Guide for more details.

Uses kmalloc to get the memory but if the allocation fails then falls back to the vmalloc allocator. Use kvfree for freeing the memory.

GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not. __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is preferable to the vmalloc fallback, due to visible performance drawbacks.

Return

pointer to the allocated memory of NULL in case of failure

void *kvrealloc_node_align(const void *p, size_t size, unsigned long align, gfp_t flags, int nid)¶: reallocate memory; contents remain unchanged

Parameters

const void *p: object to reallocate memory for
size_t size: the size to reallocate
unsigned long align: desired alignment
gfp_t flags: the flags for the page level allocator
int nid: NUMA node id

Description

If p is NULL, kvrealloc() behaves exactly like kvmalloc(). If size is 0 and p is not a NULL pointer, the object pointed to is freed.

Only alignments up to those guaranteed by kmalloc() will be honored. Please see Memory Allocation Guide for more details.

In any case, the contents of the object pointed to are preserved up to the lesser of the new and old sizes.

This function must not be called concurrently with itself or kvfree() for the same memory allocation.

Return

pointer to the allocated memory or NULL in case of error

size_t kmalloc_size_roundup(size_t size)¶: Report allocation bucket size for the given size

Parameters

size_t size: Number of bytes to round up from.

Description

This returns the number of bytes that would be available in a kmalloc() allocation of size bytes. For example, a 126 byte request would be rounded up to the next sized kmalloc bucket, 128 bytes. (This is strictly for the general-purpose kmalloc()-based allocations, and is not for the pre-sized kmem_cache_alloc()-based allocations.)

Use this to kmalloc() the full bucket size ahead of time instead of using ksize() to query the size after an allocation.

void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node)¶: Allocate an object on the specified node

Parameters

struct kmem_cache *s: The cache to allocate from.
gfp_t gfpflags: See kmalloc().
int node: node number of the target node.

Description

Identical to kmem_cache_alloc but it will allocate memory on the given node, which can improve the performance for cpu bound structures.

Fallback to other node is possible if __GFP_THISNODE is not set.

Return

pointer to the new object or NULL in case of error

void kmem_cache_free(struct kmem_cache *s, void *x)¶: Deallocate an object

Parameters

struct kmem_cache *s: The cache the allocation was from.
void *x: The previously allocated object.

Description

Free an object which was previously allocated from this cache.

size_t ksize(const void *objp)¶

Report full size of underlying allocation

Parameters

const void *objp: pointer to the object

Description

This should only be used internally to query the true size of allocations. It is not meant to be a way to discover the usable size of an allocation after the fact. Instead, use kmalloc_size_roundup(). Using memory beyond the originally requested allocation size may trigger KASAN, UBSAN_BOUNDS, and/or FORTIFY_SOURCE.

Return

size of the actual memory used by objp in bytes

void kfree(const void *object)¶: free previously allocated memory

Parameters

const void *object: pointer returned by kmalloc(), kmalloc_nolock(), or kmem_cache_alloc()

Description

If object is NULL, no operation is performed.

void kvfree(const void *addr)¶: Free memory.

Parameters

const void *addr: Pointer to allocated memory.

Description

kvfree frees memory allocated by any of vmalloc(), kmalloc() or kvmalloc(). It is slightly more efficient to use kfree() or vfree() if you are certain that you know which one to use.

Context

Either preemptible task context or not-NMI interrupt.

void kvfree_atomic(const void *addr)¶: Free memory.

Parameters

const void *addr: Pointer to allocated memory.

Description

Same as kvfree(), but uses vfree_atomic() for vmalloc backed memory. Must not be called from NMI context.

void kvfree_sensitive(const void *addr, size_t len)¶: Free a data object containing sensitive information.

Parameters

const void *addr: address of the data object to be freed.
size_t len: length of the data object.

Description

Use the special memzero_explicit() function to clear the content of a kvmalloc’ed object containing sensitive data to make sure that the compiler won’t optimize out the data clearing.

bool kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p)¶: Allocate multiple objects

Parameters

struct kmem_cache *s: The cache to allocate from
gfp_t flags: GFP_* flags. See kmalloc().
size_t size: Number of objects to allocate
void **p: Array of allocated objects

Description

Allocate size objects from s and places them into p. size must be larger than 0.

Interrupts must be enabled when calling this function.

Unlike alloc_pages_bulk(), this function does not check for already allocated objects in p, and thus the caller does not need to zero it.

Return

true if the allocation succeeded, or false if it failed.

struct kmem_cache *__kmem_cache_create_args(const char *name, unsigned int object_size, struct kmem_cache_args *args, slab_flags_t flags)¶: Create a kmem cache.

Parameters

const char *name: A string which is used in /proc/slabinfo to identify this cache.
unsigned int object_size: The size of objects to be created in this cache.
struct kmem_cache_args *args: Additional arguments for the cache creation (see struct kmem_cache_args).
slab_flags_t flags: See the descriptions of individual flags. The common ones are listed in the description below.

Description

Not to be called directly, use the kmem_cache_create() wrapper with the same parameters.

Commonly used flags:

SLAB_ACCOUNT - Account allocations to memcg.

SLAB_HWCACHE_ALIGN - Align objects on cache line boundaries.

SLAB_RECLAIM_ACCOUNT - Objects are reclaimable.

SLAB_TYPESAFE_BY_RCU - Slab page (not individual objects) freeing delayed by a grace period - see the full description before using.

Context

Cannot be called within an interrupt, but can be interrupted.

Return

a pointer to the cache on success, NULL on failure.

kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void*))¶: Create a set of caches that handle dynamic sized allocations via kmem_buckets_alloc()

Parameters

const char *name: A prefix string which is used in /proc/slabinfo to identify this cache. The individual caches with have their sizes as the suffix.
slab_flags_t flags: SLAB flags (see kmem_cache_create() for details).
unsigned int useroffset: Starting offset within an allocation that may be copied to/from userspace.
unsigned int usersize: How many bytes, starting at useroffset, may be copied to/from userspace.
void (*ctor)(void *): A constructor for the objects, run when new allocations are made.

Context

Cannot be called within an interrupt, but can be interrupted.

Return

a pointer to the cache on success, NULL on failure. When CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). (i.e. callers only need to check for NULL on failure.)

int kmem_cache_shrink(struct kmem_cache *cachep)¶: Shrink a cache.

Parameters

struct kmem_cache *cachep: The cache to shrink.

Description

Releases as many slabs as possible for a cache. To help debugging, a zero exit status indicates all slabs were released.

Return

0 if all slabs were released, non-zero otherwise

bool kmem_dump_obj(void *object)¶: Print available slab provenance information

Parameters

void *object: slab object for which to find provenance information.

Description

This function uses pr_cont(), so that the caller is expected to have printed out whatever preamble is appropriate. The provenance information depends on the type of object and on how much debugging is enabled. For a slab-cache object, the fact that it is a slab object is printed, and, if available, the slab name, return address, and stack trace from the allocation and last free path of that object.

Return

true if the pointer is to a not-yet-freed object from kmalloc() or kmem_cache_alloc(), either true or false if the pointer is to an already-freed object, and false otherwise.

void kfree_sensitive(const void *p)¶: Clear sensitive information in memory before freeing

Parameters

const void *p: object to free memory of

Description

The memory of the object p points to is zeroed before freed. If p is NULL, kfree_sensitive() does nothing.

Note

this function zeroes the whole allocated buffer which can be a good deal bigger than the requested buffer size passed to kmalloc(). So be careful when using this function in performance sensitive code.

void kfree_const(const void *x)¶: conditionally free memory

Parameters

const void *x: pointer to the memory

Description

Function calls kfree only if x is not in .rodata section.

Virtually Contiguous Mappings¶

void vm_unmap_aliases(void)¶: unmap outstanding lazy aliases in the vmap layer

Parameters

void: no arguments

Description

The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily to amortize TLB flushing overheads. What this means is that any page you have now, may, in a former life, have been mapped into kernel virtual address by the vmap layer and so there might be some CPUs with TLB entries still referencing that page (additional to the regular 1:1 kernel mapping).

vm_unmap_aliases flushes all such lazy mappings. After it returns, we can be sure that none of the pages we have control over will have any aliases from the vmap layer.

void vm_unmap_ram(const void *mem, unsigned int count)¶: unmap linear kernel address space set up by vm_map_ram

Parameters

const void *mem: the pointer returned by vm_map_ram
unsigned int count: the count passed to that vm_map_ram call (cannot unmap partial)

void *vm_map_ram(struct page **pages, unsigned int count, int node)¶: map pages linearly into kernel virtual address (vmalloc space)

Parameters

struct page **pages: an array of pointers to the pages to be mapped
unsigned int count: number of pages
int node: prefer to allocate data structures on this node

Description

If you use this function for less than VMAP_MAX_ALLOC pages, it could be faster than vmap so it’s good. But if you mix long-life and short-life objects with vm_map_ram(), it could consume lots of address space through fragmentation (especially on a 32bit machine). You could see failures in the end. Please use this function for short-lived objects.

Return

a pointer to the address that has been mapped, or NULL on failure

void vfree(const void *addr)¶: Release memory allocated by vmalloc()

Parameters

const void *addr: Memory base address

Description

Free the virtually continuous memory area starting at addr, as obtained from one of the vmalloc() family of APIs. This will usually also free the physical memory underlying the virtual allocation, but that memory is reference counted, so it will not be freed until the last user goes away.

If addr is NULL, no operation is performed.

Context

May sleep if called not from interrupt context. Must not be called in NMI context (strictly speaking, it could be if we have CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG, but making the calling conventions for vfree() arch-dependent would be a really bad idea).

void vunmap(const void *addr)¶: release virtual mapping obtained by vmap()

Parameters

const void *addr: memory base address

Description

Free the virtually contiguous memory area starting at addr, which was created from the page array passed to vmap().

Must not be called in interrupt context.

void *vmap(struct page **pages, unsigned int count, unsigned long flags, pgprot_t prot)¶: map an array of pages into virtually contiguous space

Parameters

struct page **pages: array of page pointers
unsigned int count: number of pages to map
unsigned long flags: vm_area->flags
pgprot_t prot: page protection for the mapping

Description

Maps count pages from pages into contiguous kernel virtual space. If flags contains VM_MAP_PUT_PAGES the ownership of the pages array itself (which must be kmalloc or vmalloc memory) and one reference per pages in it are transferred from the caller to vmap(), and will be freed / dropped when vfree() is called on the return value.

Return

the address of the area or NULL on failure

void *vmap_pfn(unsigned long *pfns, unsigned int count, pgprot_t prot)¶: map an array of PFNs into virtually contiguous space

Parameters

unsigned long *pfns: array of PFNs
unsigned int count: number of pages to map
pgprot_t prot: page protection for the mapping

Description

Maps count PFNs from pfns into contiguous kernel virtual space and returns the start address of the mapping.

void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller)¶: allocate virtually contiguous memory

Parameters

unsigned long size: allocation size
unsigned long align: desired alignment
gfp_t gfp_mask: flags for the page level allocator
int node: node to use for allocation or NUMA_NO_NODE
const void *caller: caller’s return address

Description

Allocate enough pages to cover size from the page level allocator with gfp_mask flags. Map them into contiguous kernel virtual space.

Semantics of gfp_mask (including reclaim/retry modifiers such as __GFP_NOFAIL) are the same as in __vmalloc_node_range_noprof().

Return

pointer to the allocated memory or NULL on error

void *vmalloc(unsigned long size)¶: allocate virtually contiguous memory

Parameters

unsigned long size: allocation size

Description

Allocate enough pages to cover size from the page level allocator and map them into contiguous kernel virtual space.

For tight control over page level allocator and protection flags use __vmalloc() instead.

Return

pointer to the allocated memory or NULL on error

void *vmalloc_huge_node(unsigned long size, gfp_t gfp_mask, int node)¶: allocate virtually contiguous memory, allow huge pages

Parameters

unsigned long size: allocation size
gfp_t gfp_mask: flags for the page level allocator
int node: node to use for allocation or NUMA_NO_NODE

Description

Allocate enough pages to cover size from the page level allocator and map them into contiguous kernel virtual space. If size is greater than or equal to PMD_SIZE, allow using huge pages for the memory

Return

pointer to the allocated memory or NULL on error

void *vzalloc(unsigned long size)¶: allocate virtually contiguous memory with zero fill

Parameters

unsigned long size: allocation size

Description

Allocate enough pages to cover size from the page level allocator and map them into contiguous kernel virtual space. The memory allocated is set to zero.

For tight control over page level allocator and protection flags use __vmalloc() instead.

Return

pointer to the allocated memory or NULL on error

void *vmalloc_user(unsigned long size)¶: allocate zeroed virtually contiguous memory for userspace

Parameters

unsigned long size: allocation size

Description

The resulting memory area is zeroed so it can be mapped to userspace without leaking data.

Return

pointer to the allocated memory or NULL on error

void *vmalloc_node(unsigned long size, int node)¶: allocate memory on a specific node

Parameters

unsigned long size: allocation size
int node: numa node

Description

Allocate enough pages to cover size from the page level allocator and map them into contiguous kernel virtual space.

For tight control over page level allocator and protection flags use __vmalloc() instead.

Return

pointer to the allocated memory or NULL on error

void *vzalloc_node(unsigned long size, int node)¶: allocate memory on a specific node with zero fill

Parameters

unsigned long size: allocation size
int node: numa node

Description

Allocate enough pages to cover size from the page level allocator and map them into contiguous kernel virtual space. The memory allocated is set to zero.

Return

pointer to the allocated memory or NULL on error

void *vrealloc_node_align(const void *p, size_t size, unsigned long align, gfp_t flags, int nid)¶: reallocate virtually contiguous memory; contents remain unchanged

Parameters

const void *p: object to reallocate memory for
size_t size: the size to reallocate
unsigned long align: requested alignment
gfp_t flags: the flags for the page level allocator
int nid: node number of the target node

Description

If p is NULL, vrealloc_XXX() behaves exactly like vmalloc_XXX(). If size is 0 and p is not a NULL pointer, the object pointed to is freed.

If the caller wants the new memory to be on specific node only, __GFP_THISNODE flag should be set, otherwise the function will try to avoid reallocation and possibly disregard the specified nid.

Requesting an alignment that is bigger than the alignment of the existing allocation will fail.

In any case, the contents of the object pointed to are preserved up to the lesser of the new and old sizes.

This function must not be called concurrently with itself or vfree() for the same memory allocation.

Return

pointer to the allocated memory; NULL if size is zero or in case of failure

void *vmalloc_32(unsigned long size)¶: allocate virtually contiguous memory (32bit addressable)

Parameters

unsigned long size: allocation size

Description

Allocate enough 32bit PA addressable pages to cover size from the page level allocator and map them into contiguous kernel virtual space.

Return

pointer to the allocated memory or NULL on error

void *vmalloc_32_user(unsigned long size)¶: allocate zeroed virtually contiguous 32bit memory

Parameters

unsigned long size: allocation size

Description

The resulting memory area is 32bit addressable and zeroed so it can be mapped to userspace without leaking data.

Return

pointer to the allocated memory or NULL on error

int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, unsigned long pgoff)¶: map vmalloc pages to userspace

Parameters

struct vm_area_struct *vma: vma to cover (map full range of vma)
void *addr: vmalloc memory
unsigned long pgoff: number of pages into addr before first page to map

Return

0 for success, -Exxx on failure

Description

This function checks that addr is a valid vmalloc’ed area, and that it is big enough to cover the vma. Will return failure if that criteria isn’t met.

Similar to remap_pfn_range() (see mm/memory.c)

Kernel Page Table Permissions¶

The set_memory() and set_direct_map() APIs update permissions of existing kernel mappings.

The set_memory() functions operate on a range of kernel virtual addresses, the set_direct_map() functions operate on the direct map.

The updates are not atomic: when a call fails, an arbitrary prefix of the range may have been updated already and there is no automatic rollback. A caller must restore the required permissions before reusing or freeing the memory.

When an architecture does not implement these APIs they succeed without doing anything, so a return value of 0 does not mean that the permissions were actually changed.

Callers that depend on the permissions being applied must ensure that the architecture supports the required operation for the target addresses. The Kconfig symbols alone do not guarantee this.

See Kernel Page Tables for the details and for the differences between the architecture implementations.

int set_memory_ro(unsigned long addr, int numpages)¶: make a kernel mapping read-only

Parameters

unsigned long addr: page aligned start of the kernel virtual address range
int numpages: number of pages in the range

Description

Flushes the TLB for the range.

Return

0 on success, negative error code on failure.

int set_memory_rw(unsigned long addr, int numpages)¶: make a kernel mapping writable

Parameters

unsigned long addr: page aligned start of the kernel virtual address range
int numpages: number of pages in the range

Description

Flushes the TLB for the range.

Return

0 on success, negative error code on failure.

int set_memory_x(unsigned long addr, int numpages)¶: make a kernel mapping executable

Parameters

unsigned long addr: page aligned start of the kernel virtual address range
int numpages: number of pages in the range

Description

Flushes the TLB for the range.

Return

0 on success, negative error code on failure.

int set_memory_nx(unsigned long addr, int numpages)¶: make a kernel mapping non-executable

Parameters

unsigned long addr: page aligned start of the kernel virtual address range
int numpages: number of pages in the range

Description

Flushes the TLB for the range.

Return

0 on success, negative error code on failure.

int set_memory_rox(unsigned long addr, int numpages)¶: make a kernel mapping read-only and executable

Parameters

unsigned long addr: page aligned start of the kernel virtual address range
int numpages: number of pages in the range

Description

A failure may leave the range read-only but not executable.

Flushes the TLB for the range.

Return

0 on success, negative error code on failure.

int set_direct_map_invalid_noflush(struct page *page, unsigned int nr)¶: remove pages from the direct map

Parameters

struct page *page: first page to update
unsigned int nr: number of pages to update

Description

Makes the direct mapping of nr pages starting at page not present. The caller is responsible for any required TLB flushing.

Return

0 on success, negative error code on failure.

int set_direct_map_default_noflush(struct page *page, unsigned int nr)¶: restore the direct map of pages

Parameters

struct page *page: first page to update
unsigned int nr: number of pages to update

Description

Restores the default kernel permissions of the direct mapping of nr pages starting at page. The caller is responsible for any required TLB flushing.

Return

0 on success, negative error code on failure.

bool can_set_direct_map(void)¶: check if the direct map can be modified

Parameters

void: no arguments

Description

Available with CONFIG_ARCH_HAS_SET_DIRECT_MAP. Architectures may override this to report whether direct map updates are enabled at runtime. Even though the generic implementation returns true this does not guarantee that every address can be updated.

See Kernel Page Tables for the details

Return

true unless the architecture reports direct map updates disabled.

File Mapping and Page Cache¶

Filemap¶

int filemap_fdatawrite_range(struct address_space *mapping, loff_t start, loff_t end)¶: start writeback on mapping dirty pages in range

Parameters

struct address_space *mapping: address space structure to write
loff_t start: offset in bytes where the range starts
loff_t end: offset in bytes where the range ends (inclusive)

Description

Start writeback against all of a mapping’s dirty pages that lie within the byte offsets <start, end> inclusive.

This is a data integrity operation that waits upon dirty or in writeback pages.

Return

0 on success, negative error code otherwise.

int filemap_flush_range(struct address_space *mapping, loff_t start, loff_t end)¶: start writeback on a range

Parameters

struct address_space *mapping: target address_space
loff_t start: index to start writeback on
loff_t end: last (inclusive) index for writeback

Description

This is a non-integrity writeback helper, to start writing back folios for the indicated range.

Return

0 on success, negative error code otherwise.

int filemap_flush(struct address_space *mapping)¶: mostly a non-blocking flush

Parameters

struct address_space *mapping: target address_space

Description

This is a mostly non-blocking flush. Not suitable for data-integrity purposes - I/O may not be started against all dirty pages.

Return

0 on success, negative error code otherwise.

bool filemap_range_has_page(struct address_space *mapping, loff_t start_byte, loff_t end_byte)¶: check if a page exists in range.

Parameters

struct address_space *mapping: address space within which to check
loff_t start_byte: offset in bytes where the range starts
loff_t end_byte: offset in bytes where the range ends (inclusive)

Description

Find at least one page in the range supplied, usually used to check if direct writing in this range will trigger a writeback.

Return

true if at least one page exists in the specified range, false otherwise.

int filemap_fdatawait_range(struct address_space *mapping, loff_t start_byte, loff_t end_byte)¶: wait for writeback to complete

Parameters

struct address_space *mapping: address space structure to wait for
loff_t start_byte: offset in bytes where the range starts
loff_t end_byte: offset in bytes where the range ends (inclusive)

Description

Walk the list of under-writeback pages of the given address space in the given range and wait for all of them. Check error status of the address space and return it.

Since the error status of the address space is cleared by this function, callers are responsible for checking the return value and handling and/or reporting the error.

Return

error status of the address space.

int filemap_fdatawait_range_keep_errors(struct address_space *mapping, loff_t start_byte, loff_t end_byte)¶: wait for writeback to complete

Parameters

struct address_space *mapping: address space structure to wait for
loff_t start_byte: offset in bytes where the range starts
loff_t end_byte: offset in bytes where the range ends (inclusive)

Description

Walk the list of under-writeback pages of the given address space in the given range and wait for all of them. Unlike filemap_fdatawait_range(), this function does not clear error status of the address space.

Use this function if callers don’t handle errors themselves. Expected call sites are system-wide / filesystem-wide data flushers: e.g. sync(2), fsfreeze(8)

int file_fdatawait_range(struct file *file, loff_t start_byte, loff_t end_byte)¶: wait for writeback to complete

Parameters

struct file *file: file pointing to address space structure to wait for
loff_t start_byte: offset in bytes where the range starts
loff_t end_byte: offset in bytes where the range ends (inclusive)

Description

Walk the list of under-writeback pages of the address space that file refers to, in the given range and wait for all of them. Check error status of the address space vs. the file->f_wb_err cursor and return it.

Since the error status of the file is advanced by this function, callers are responsible for checking the return value and handling and/or reporting the error.

Return

error status of the address space vs. the file->f_wb_err cursor.

int filemap_fdatawait_keep_errors(struct address_space *mapping)¶: wait for writeback without clearing errors

Parameters

struct address_space *mapping: address space structure to wait for

Description

Walk the list of under-writeback pages of the given address space and wait for all of them. Unlike filemap_fdatawait(), this function does not clear error status of the address space.

Use this function if callers don’t handle errors themselves. Expected call sites are system-wide / filesystem-wide data flushers: e.g. sync(2), fsfreeze(8)

Return

error status of the address space.

int filemap_write_and_wait_range(struct address_space *mapping, loff_t lstart, loff_t lend)¶: write out & wait on a file range

Parameters

struct address_space *mapping: the address_space for the pages
loff_t lstart: offset in bytes where the range starts
loff_t lend: offset in bytes where the range ends (inclusive)

Description

Write out and wait upon file offsets lstart->lend, inclusive.

Note that lend is inclusive (describes the last byte to be written) so that this function can be used to write to the very end-of-file (end = -1).

Return

error status of the address space.

int file_check_and_advance_wb_err(struct file *file)¶: report wb error (if any) that was previously and advance wb_err to current one

Parameters

struct file *file: struct file on which the error is being reported

Description

When userland calls fsync (or something like nfsd does the equivalent), we want to report any writeback errors that occurred since the last fsync (or since the file was opened if there haven’t been any).

Grab the wb_err from the mapping. If it matches what we have in the file, then just quickly return 0. The file is all caught up.

If it doesn’t match, then take the mapping value, set the “seen” flag in it and try to swap it into place. If it works, or another task beat us to it with the new value, then update the f_wb_err and return the error portion. The error at this point must be reported via proper channels (a’la fsync, or NFS COMMIT operation, etc.).

While we handle mapping->wb_err with atomic operations, the f_wb_err value is protected by the f_lock since we must ensure that it reflects the latest value swapped in for this file descriptor.

Return

0 on success, negative error code otherwise.

int file_write_and_wait_range(struct file *file, loff_t lstart, loff_t lend)¶: write out & wait on a file range

Parameters

struct file *file: file pointing to address_space with pages
loff_t lstart: offset in bytes where the range starts
loff_t lend: offset in bytes where the range ends (inclusive)

Description

Write out and wait upon file offsets lstart->lend, inclusive.

Note that lend is inclusive (describes the last byte to be written) so that this function can be used to write to the very end-of-file (end = -1).

After writing out and waiting on the data, we check and advance the f_wb_err cursor to the latest value, and return any errors detected there.

Return

0 on success, negative error code otherwise.

void replace_page_cache_folio(struct folio *old, struct folio *new)¶: replace a pagecache folio with a new one

Parameters

struct folio *old: folio to be replaced
struct folio *new: folio to replace with

Description

This function replaces a folio in the pagecache with a new one. On success it acquires the pagecache reference for the new folio and drops it for the old folio. Both the old and new folios must be locked. This function does not add the new folio to the LRU, the caller must do that.

The remove + add is atomic. This function cannot fail.

void folio_unlock(struct folio *folio)¶: Unlock a locked folio.

Parameters

struct folio *folio: The folio.

Description

Unlocks the folio and wakes up any thread sleeping on the page lock.

Context

May be called from interrupt or process context. May not be called from NMI context.

void folio_end_read(struct folio *folio, bool success)¶: End read on a folio.

Parameters

struct folio *folio: The folio.
bool success: True if all reads completed successfully.

Description

When all reads against a folio have completed, filesystems should call this function to let the pagecache know that no more reads are outstanding. This will unlock the folio and wake up any thread sleeping on the lock. The folio will also be marked uptodate if all reads succeeded.

Context

May be called from interrupt or process context. May not be called from NMI context.

void folio_end_private_2(struct folio *folio)¶: Clear PG_private_2 and wake any waiters.

Parameters

struct folio *folio: The folio.

Description

Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for it. The folio reference held for PG_private_2 being set is released.

This is, for example, used when a netfs folio is being written to a local disk cache, thereby allowing writes to the cache for the same folio to be serialised.

void folio_wait_private_2(struct folio *folio)¶: Wait for PG_private_2 to be cleared on a folio.

Parameters

struct folio *folio: The folio to wait on.

Description

Wait for PG_private_2 to be cleared on a folio.

int folio_wait_private_2_killable(struct folio *folio)¶: Wait for PG_private_2 to be cleared on a folio.

Parameters

struct folio *folio: The folio to wait on.

Description

Wait for PG_private_2 to be cleared on a folio or until a fatal signal is received by the calling task.

Return

0 if successful.
-EINTR if a fatal signal was encountered.

void folio_end_writeback_no_dropbehind(struct folio *folio)¶: End writeback against a folio.

Parameters

struct folio *folio: The folio.

Description

The folio must actually be under writeback. This call is intended for filesystems that need to defer dropbehind.

Context

May be called from process or interrupt context.

void folio_end_writeback(struct folio *folio)¶: End writeback against a folio.

Parameters

struct folio *folio: The folio.

Description

The folio must actually be under writeback.

Context

May be called from process or interrupt context.

void __folio_lock(struct folio *folio)¶: Get a lock on the folio, assuming we need to sleep to get it.

Parameters

struct folio *folio: The folio to lock

pgoff_t page_cache_next_miss(struct address_space *mapping, pgoff_t index, unsigned long max_scan)¶: Find the next gap in the page cache.

Parameters

struct address_space *mapping: Mapping.
pgoff_t index: Index.
unsigned long max_scan: Maximum range to search.

Description

Search the range [index, min(index + max_scan - 1, ULONG_MAX)] for the gap with the lowest index.

This function may be called under the rcu_read_lock. However, this will not atomically search a snapshot of the cache at a single point in time. For example, if a gap is created at index 5, then subsequently a gap is created at index 10, page_cache_next_miss covering both indices may return 10 if called under the rcu_read_lock.

Return

The index of the gap if found, otherwise an index outside the range specified (in which case ‘return - index >= max_scan’ will be true). In the rare case of index wrap-around, 0 will be returned.

pgoff_t page_cache_prev_miss(struct address_space *mapping, pgoff_t index, unsigned long max_scan)¶: Find the previous gap in the page cache.

Parameters

struct address_space *mapping: Mapping.
pgoff_t index: Index.
unsigned long max_scan: Maximum range to search.

Description

Search the range [max(index - max_scan + 1, 0), index] for the gap with the highest index.

This function may be called under the rcu_read_lock. However, this will not atomically search a snapshot of the cache at a single point in time. For example, if a gap is created at index 10, then subsequently a gap is created at index 5, page_cache_prev_miss() covering both indices may return 5 if called under the rcu_read_lock.

Return

The index of the gap if found, otherwise an index outside the range specified (in which case ‘index - return >= max_scan’ will be true). In the rare case of wrap-around, ULONG_MAX will be returned.

struct folio *__filemap_get_folio_mpol(struct address_space *mapping, pgoff_t index, fgf_t fgp_flags, gfp_t gfp, struct mempolicy *policy)¶: Find and get a reference to a folio.

Parameters

struct address_space *mapping: The address_space to search.
pgoff_t index: The page index.
fgf_t fgp_flags: FGP flags modify how the folio is returned.
gfp_t gfp: Memory allocation flags to use if FGP_CREAT is specified.
struct mempolicy *policy: NUMA memory allocation policy to follow.

Description

Looks up the page cache entry at mapping & index.

If FGP_LOCK or FGP_CREAT are specified then the function may sleep even if the GFP flags specified for FGP_CREAT are atomic.

If this function returns a folio, it is returned with an increased refcount.

Return

The found folio or an ERR_PTR() otherwise.

unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch)¶: Get a batch of folios

Parameters

struct address_space *mapping: The address_space to search
pgoff_t *start: The starting page index
pgoff_t end: The final page index (inclusive)
struct folio_batch *fbatch: The batch to fill.

Description

Search for and return a batch of folios in the mapping starting at index start and up to index end (inclusive). The folios are returned in fbatch with an elevated reference count.

Return

The number of folios which were found. We also update start to index the next folio for the traversal.

unsigned filemap_get_folios_contig(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch)¶: Get a batch of contiguous folios

Parameters

struct address_space *mapping: The address_space to search
pgoff_t *start: The starting page index
pgoff_t end: The final page index (inclusive)
struct folio_batch *fbatch: The batch to fill

Description

filemap_get_folios_contig() works exactly like filemap_get_folios(), except the returned folios are guaranteed to be contiguous. This may not return all contiguous folios if the batch gets filled up.

Return

The number of folios found. Also update start to be positioned for traversal of the next folio.

unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start, pgoff_t end, xa_mark_t tag, struct folio_batch *fbatch)¶: Get a batch of folios matching tag

Parameters

struct address_space *mapping: The address_space to search
pgoff_t *start: The starting page index
pgoff_t end: The final page index (inclusive)
xa_mark_t tag: The tag index
struct folio_batch *fbatch: The batch to fill

Description

The first folio may start before start; if it does, it will contain start. The final folio may extend beyond end; if it does, it will contain end. The folios have ascending indices. There may be gaps between the folios if there are indices which have no folio in the page cache. If folios are added to or removed from the page cache while this is running, they may or may not be found by this call. Only returns folios that are tagged with tag.

Return

The number of folios found. Also update start to index the next folio for traversal.

ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, ssize_t already_read)¶: Read data from the page cache.

Parameters

struct kiocb *iocb: The iocb to read.
struct iov_iter *iter: Destination for the data.
ssize_t already_read: Number of bytes already read by the caller.

Description

Copies data from the page cache. If the data is not currently present, uses the readahead and read_folio address_space operations to fetch it.

Return

Total number of bytes copied, including those already read by the caller. If an error happens before any bytes are copied, returns a negative error number.

ssize_t generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)¶: generic filesystem read routine

Parameters

struct kiocb *iocb: kernel I/O control block
struct iov_iter *iter: destination for the data read

Description

This is the “read_iter()” routine for all filesystems that can use the page cache directly.

The IOCB_NOWAIT flag in iocb->ki_flags indicates that -EAGAIN shall be returned when no data can be read without waiting for I/O requests to complete; it doesn’t prevent readahead.

The IOCB_NOIO flag in iocb->ki_flags indicates that no new I/O requests shall be made for the read or for readahead. When no data can be read, -EAGAIN shall be returned. When readahead would be triggered, a partial, possibly empty read shall be returned.

Return

number of bytes copied, even for partial reads
negative error code (or 0 if IOCB_NOIO) if nothing was read

ssize_t filemap_splice_read(struct file *in, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags)¶: Splice data from a file’s pagecache into a pipe

Parameters

struct file *in: The file to read from
loff_t *ppos: Pointer to the file position to read from
struct pipe_inode_info *pipe: The pipe to splice into
size_t len: The amount to splice
unsigned int flags: The SPLICE_F_* flags

Description

This function gets folios from a file’s pagecache and splices them into the pipe. Readahead will be called as necessary to fill more folios. This may be used for blockdevs also.

Return

On success, the number of bytes read will be returned and *ppos will be updated if appropriate; 0 will be returned if there is no more data to be read; -EAGAIN will be returned if the pipe had no space, and some other negative error code will be returned on error. A short read may occur if the pipe has insufficient space, we reach the end of the data or we hit a hole.

vm_fault_t filemap_fault(struct vm_fault *vmf)¶: read in file data for page fault handling

Parameters

struct vm_fault *vmf: struct vm_fault containing details of the fault

Description

filemap_fault() is invoked via the vma operations vector for a mapped memory region to read in file data during a page fault.

The goto’s are kind of ugly, but this streamlines the normal case of having it in the page cache, and handles the special cases reasonably without having a lot of duplicated code.

vma->vm_mm->mmap_lock must be held on entry.

If our return value has VM_FAULT_RETRY set, it’s because the mmap_lock may be dropped before doing I/O or when aborting an interrupted wait.

If our return value does not have VM_FAULT_RETRY set, the mmap_lock has not been released.

We never return with VM_FAULT_RETRY and a bit from VM_FAULT_ERROR set.

Return

bitwise-OR of VM_FAULT_ codes.

struct folio *read_cache_folio(struct address_space *mapping, pgoff_t index, filler_t filler, struct file *file)¶: Read into page cache, fill it if needed.

Parameters

struct address_space *mapping: The address_space to read from.
pgoff_t index: The index to read.
filler_t filler: Function to perform the read, or NULL to use aops->read_folio().
struct file *file: Passed to filler function, may be NULL if not required.

Description

Read one page into the page cache. If it succeeds, the folio returned will contain index, but it may not be the first page of the folio.

If the filler function returns an error, it will be returned to the caller.

Context

May sleep. Expects mapping->invalidate_lock to be held.

Return

An uptodate folio on success, ERR_PTR() on failure.

struct folio *mapping_read_folio_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp)¶: Read into page cache, using specified allocation flags.

Parameters

struct address_space *mapping: The address_space for the folio.
pgoff_t index: The index that the allocated folio will contain.
gfp_t gfp: The page allocator flags to use if allocating.

Description

This is the same as “read_cache_folio(mapping, index, NULL, NULL)”, but with any new memory allocations done using the specified allocation flags.

The most likely error from this function is EIO, but ENOMEM is possible and so is EINTR. If ->read_folio returns another error, that will be returned to the caller.

The function expects mapping->invalidate_lock to be already held.

Return

Uptodate folio on success, ERR_PTR() on failure.

struct page *read_cache_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp)¶: read into page cache, using specified page allocation flags.

Parameters

struct address_space *mapping: the page’s address_space
pgoff_t index: the page index
gfp_t gfp: the page allocator flags to use if allocating

Description

This is the same as “read_mapping_page(mapping, index, NULL)”, but with any new page allocations done using the specified allocation flags.

If the page does not get brought uptodate, return -EIO.

The function expects mapping->invalidate_lock to be already held.

Return

up to date page on success, ERR_PTR() on failure.

ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)¶: write data to a file

Parameters

struct kiocb *iocb: IO state structure (file, offset, etc.)
struct iov_iter *from: iov_iter with data to write

Description

This function does all the work needed for actually writing data to a file. It does all basic checks, removes SUID from the file, updates modification times and calls proper subroutines depending on whether we do direct IO or a standard buffered write.

It expects i_rwsem to be grabbed unless we work on a block device or similar object which does not need locking at all.

This function does not take care of syncing data in case of O_SYNC write. A caller has to handle it. This is mainly due to the fact that we want to avoid syncing under i_rwsem.

Return

number of bytes written, even for truncated writes
negative error code if no data has been written at all

ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)¶: write data to a file

Parameters

struct kiocb *iocb: IO state structure
struct iov_iter *from: iov_iter with data to write

Description

This is a wrapper around __generic_file_write_iter() to be used by most filesystems. It takes care of syncing the file in case of O_SYNC file and acquires i_rwsem as needed.

Return

negative error code if no data has been written at all of vfs_fsync_range() failed for a synchronous write
number of bytes written, even for truncated writes

bool filemap_release_folio(struct folio *folio, gfp_t gfp)¶: Release fs-specific metadata on a folio.

Parameters

struct folio *folio: The folio which the kernel is trying to free.
gfp_t gfp: Memory allocation flags (and I/O mode).

Description

The address_space is trying to release any data attached to a folio (presumably at folio->private).

This will also be called if the private_2 flag is set on a page, indicating that the folio has other metadata associated with it.

The gfp argument specifies whether I/O may be performed to release this page (__GFP_IO), and whether the call may block (__GFP_RECLAIM & __GFP_FS).

Return

true if the release was successful, otherwise false.

int filemap_invalidate_inode(struct inode *inode, bool flush, loff_t start, loff_t end)¶: Invalidate/forcibly write back a range of an inode’s pagecache

Parameters

struct inode *inode: The inode to flush
bool flush: Set to write back rather than simply invalidate.
loff_t start: First byte to in range.
loff_t end: Last byte in range (inclusive), or LLONG_MAX for everything from start onwards.

Description

Invalidate all the folios on an inode that contribute to the specified range, possibly writing them back first. Whilst the operation is undertaken, the invalidate lock is held to prevent new folios from being installed.

Readahead¶

Readahead is used to read content into the page cache before it is explicitly requested by the application. Readahead only ever attempts to read folios that are not yet in the page cache. If a folio is present but not up-to-date, readahead will not try to read it. In that case a simple ->read_folio() will be requested.

Readahead is triggered when an application read request (whether a system call or a page fault) finds that the requested folio is not in the page cache, or that it is in the page cache and has the readahead flag set. This flag indicates that the folio was read as part of a previous readahead request and now that it has been accessed, it is time for the next readahead.

Each readahead request is partly synchronous read, and partly async readahead. This is reflected in the struct file_ra_state which contains ->size being the total number of pages, and ->async_size which is the number of pages in the async section. The readahead flag will be set on the first folio in this async section to trigger a subsequent readahead. Once a series of sequential reads has been established, there should be no need for a synchronous component and all readahead request will be fully asynchronous.

When either of the triggers causes a readahead, three numbers need to be determined: the start of the region to read, the size of the region, and the size of the async tail.

The start of the region is simply the first page address at or after the accessed address, which is not currently populated in the page cache. This is found with a simple search in the page cache.

The size of the async tail is determined by subtracting the size that was explicitly requested from the determined request size, unless this would be less than zero - then zero is used. NOTE THIS CALCULATION IS WRONG WHEN THE START OF THE REGION IS NOT THE ACCESSED PAGE. ALSO THIS CALCULATION IS NOT USED CONSISTENTLY.

The size of the region is normally determined from the size of the previous readahead which loaded the preceding pages. This may be discovered from the struct file_ra_state for simple sequential reads, or from examining the state of the page cache when multiple sequential reads are interleaved. Specifically: where the readahead was triggered by the readahead flag, the size of the previous readahead is assumed to be the number of pages from the triggering page to the start of the new readahead. In these cases, the size of the previous readahead is scaled, often doubled, for the new readahead, though see get_next_ra_size() for details.

If the size of the previous read cannot be determined, the number of preceding pages in the page cache is used to estimate the size of a previous read. This estimate could easily be misled by random reads being coincidentally adjacent, so it is ignored unless it is larger than the current request, and it is not scaled up, unless it is at the start of file.

In general readahead is accelerated at the start of the file, as reads from there are often sequential. There are other minor adjustments to the readahead size in various special cases and these are best discovered by reading the code.

The above calculation, based on the previous readahead size, determines the size of the readahead, to which any requested read size may be added.

Readahead requests are sent to the filesystem using the ->readahead() address space operation, for which mpage_readahead() is a canonical implementation. ->readahead() should normally initiate reads on all folios, but may fail to read any or all folios without causing an I/O error. The page cache reading code will issue a ->read_folio() request for any folio which ->readahead() did not read, and only an error from this will be final.

->readahead() will generally call readahead_folio() repeatedly to get each folio from those prepared for readahead. It may fail to read a folio by:

not calling readahead_folio() sufficiently many times, effectively ignoring some folios, as might be appropriate if the path to storage is congested.
failing to actually submit a read request for a given folio, possibly due to insufficient resources, or
getting an error during subsequent processing of a request.

In the last two cases, the folio should be unlocked by the filesystem to indicate that the read attempt has failed. In the first case the folio will be unlocked by the VFS.

Those folios not in the final async_size of the request should be considered to be important and ->readahead() should not fail them due to congestion or temporary resource unavailability, but should wait for necessary resources (e.g. memory or indexing information) to become available. Folios in the final async_size may be considered less urgent and failure to read them is more acceptable. In this case it is best to use filemap_remove_folio() to remove the folios from the page cache as is automatically done for folios that were not fetched with readahead_folio(). This will allow a subsequent synchronous readahead request to try them again. If they are left in the page cache, then they will be read individually using ->read_folio() which may be less efficient.

void page_cache_ra_unbounded(struct readahead_control *ractl, unsigned long nr_to_read, unsigned long lookahead_size)¶: Start unchecked readahead.

Parameters

struct readahead_control *ractl: Readahead control.
unsigned long nr_to_read: The number of pages to read.
unsigned long lookahead_size: Where to start the next readahead.

Description

This function is for filesystems to call when they want to start readahead beyond a file’s stated i_size. This is almost certainly not the function you want to call. Use page_cache_async_readahead() or page_cache_sync_readahead() instead.

Context

File is referenced by caller, and ractl->mapping->invalidate_lock must be held by the caller at least in shared mode. Mutexes may be held by caller. May sleep, but will not reenter filesystem to reclaim memory.

void readahead_expand(struct readahead_control *ractl, loff_t new_start, size_t new_len)¶: Expand a readahead request

Parameters

struct readahead_control *ractl: The request to be expanded
loff_t new_start: The revised start
size_t new_len: The revised size of the request

Description

Attempt to expand a readahead request outwards from the current size to the specified size by inserting locked pages before and after the current window to increase the size to the new window. This may involve the insertion of THPs, in which case the window may get expanded even beyond what was requested.

The algorithm will stop if it encounters a conflicting page already in the pagecache and leave a smaller expansion than requested.

The caller must check for this by examining the revised ractl object for a different expansion than was requested.

Writeback¶

void wb_domain_size_changed(struct wb_domain *dom)¶: memory available to a wb_domain has changed

Parameters

struct wb_domain *dom: wb_domain of interest

Description

This function should be called when the amount of memory available to dom has changed. It resets dom’s dirty limit parameters to prevent the past values which don’t match the current configuration from skewing dirty throttling. Without this, when memory size of a wb_domain is greatly reduced, the dirty throttling logic may allow too many pages to be dirtied leading to consecutive unnecessary OOMs and may get stuck in that situation.

void inode_attach_wb(struct inode *inode, struct folio *folio)¶: associate an inode with its wb

Parameters

struct inode *inode: inode of interest
struct folio *folio: folio being dirtied (may be NULL)

Description

If inode doesn’t have its wb, associate it with the wb matching the memcg of folio or, if folio is NULL, current. May be called w/ or w/o inode->i_lock.

void inode_detach_wb(struct inode *inode)¶: disassociate an inode from its wb

Parameters

struct inode *inode: inode of interest

Description

inode is being freed. Detach from its wb.

void wbc_init_bio(struct writeback_control *wbc, struct bio *bio)¶: writeback specific initialization of bio

Parameters

struct writeback_control *wbc: writeback_control for the writeback in progress
struct bio *bio: bio to be initialized

Description

bio is a part of the writeback in progress controlled by wbc. Perform writeback specific initialization. This is used to apply the cgroup writeback context. Must be called after the bio has been associated with a device.

int balance_dirty_pages_ratelimited_flags(struct address_space *mapping, unsigned int flags)¶: Balance dirty memory state.

Parameters

struct address_space *mapping: address_space which was dirtied.
unsigned int flags: BDP flags.

Description

Processes which are dirtying memory should call in here once for each page which was newly dirtied. The function will periodically check the system’s dirty state and will initiate writeback if needed.

See balance_dirty_pages_ratelimited() for details.

Return

If flags contains BDP_ASYNC, it may return -EAGAIN to indicate that memory is out of balance and the caller must wait for I/O to complete. Otherwise, it will return 0 to indicate that either memory was already in balance, or it was able to sleep until the amount of dirty memory returned to balance.

void balance_dirty_pages_ratelimited(struct address_space *mapping)¶: balance dirty memory state.

Parameters

struct address_space *mapping: address_space which was dirtied.

Description

Once we’re over the dirty memory limit we decrease the ratelimiting by a lot, to prevent individual processes from overshooting the limit by (ratelimit_pages) each.

void tag_pages_for_writeback(struct address_space *mapping, pgoff_t start, pgoff_t end)¶: tag pages to be written by writeback

Parameters

struct address_space *mapping: address space structure to write
pgoff_t start: starting page index
pgoff_t end: ending page index (inclusive)

Description

This function scans the page range from start to end (inclusive) and tags all pages that have DIRTY tag set with a special TOWRITE tag. The caller can then use the TOWRITE tag to identify pages eligible for writeback. This mechanism is used to avoid livelocking of writeback by a process steadily creating new dirty pages in the file (thus it is important for this function to be quick so that it can tag pages faster than a dirtying process can create them).

struct folio *writeback_iter(struct address_space *mapping, struct writeback_control *wbc, struct folio *folio, int *error)¶: iterate folio of a mapping for writeback

Parameters

struct address_space *mapping: address space structure to write
struct writeback_control *wbc: writeback context
struct folio *folio: previously iterated folio (NULL to start)
int *error: in-out pointer for writeback errors (see below)

Description

This function returns the next folio for the writeback operation described by wbc on mapping and should be called in a while loop in the ->writepages implementation.

To start the writeback operation, NULL is passed in the folio argument, and for every subsequent iteration the folio returned previously should be passed back in.

If there was an error in the per-folio writeback inside the writeback_iter() loop, error should be set to the error value.

Once the writeback described in wbc has finished, this function will return NULL and if there was an error in any iteration restore it to error.

Note

callers should not manually break out of the loop using break or goto but must keep calling writeback_iter() until it returns NULL.

Return

the folio to write or NULL if the loop is done.

bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)¶: Mark a folio dirty for filesystems which do not use buffer_heads.

Parameters

struct address_space *mapping: Address space this folio belongs to.
struct folio *folio: Folio to be marked as dirty.

Description

Filesystems which do not use buffer heads should call this function from their dirty_folio address space operation. It ignores the contents of folio_get_private(), so if the filesystem marks individual blocks as dirty, the filesystem should handle that itself.

This is also sometimes used by filesystems which use buffer_heads when a single buffer is being dirtied: we want to set the folio dirty in that case, but not all the buffers. This is a “bottom-up” dirtying, whereas block_dirty_folio() is a “top-down” dirtying.

The caller must ensure this doesn’t race with truncation. Most will simply hold the folio lock, but e.g. zap_pte_range() calls with the folio mapped and the pte lock held, which also locks out truncation.

bool folio_redirty_for_writepage(struct writeback_control *wbc, struct folio *folio)¶: Decline to write a dirty folio.

Parameters

struct writeback_control *wbc: The writeback control.
struct folio *folio: The folio.

Description

When a writepage implementation decides that it doesn’t want to write folio for some reason, it should call this function, unlock folio and return 0.

Return

True if we redirtied the folio. False if someone else dirtied it first.

bool folio_mark_dirty(struct folio *folio)¶: Mark a folio as being modified.

Parameters

struct folio *folio: The folio.

Description

The folio may not be truncated while this function is running. Holding the folio lock is sufficient to prevent truncation, but some callers cannot acquire a sleeping lock. These callers instead hold the page table lock for a page table which contains at least one page in this folio. Truncation will block on the page table lock as it unmaps pages before removing the folio from its mapping.

Danger

Do not use this on a folio obtained from a function like get_user_pages_fast() without holding appropriate locks; you might want to use set_page_dirty_lock() or folio_mark_dirty_lock() instead.

Return

True if the folio was newly dirtied, false if it was already dirty.

void folio_wait_writeback(struct folio *folio)¶: Wait for a folio to finish writeback.

Parameters

struct folio *folio: The folio to wait for.

Description

If the folio is currently being written back to storage, wait for the I/O to complete.

Context

Sleeps. Must be called in process context and with no spinlocks held. Caller should hold a reference on the folio. If the folio is not locked, writeback may start again after writeback has finished.

int folio_wait_writeback_killable(struct folio *folio)¶: Wait for a folio to finish writeback.

Parameters

struct folio *folio: The folio to wait for.

Description

If the folio is currently being written back to storage, wait for the I/O to complete or a fatal signal to arrive.

Context

Sleeps. Must be called in process context and with no spinlocks held. Caller should hold a reference on the folio. If the folio is not locked, writeback may start again after writeback has finished.

Return

0 on success, -EINTR if we get a fatal signal while waiting.

void folio_wait_stable(struct folio *folio)¶: wait for writeback to finish, if necessary.

Parameters

struct folio *folio: The folio to wait on.

Description

This function determines if the given folio is related to a backing device that requires folio contents to be held stable during writeback. If so, then it will wait for any pending writeback to complete.

Context

Sleeps. Must be called in process context and with no spinlocks held. Caller should hold a reference on the folio. If the folio is not locked, writeback may start again after writeback has finished.

Truncate¶

void folio_invalidate(struct folio *folio, size_t offset, size_t length)¶: Invalidate part or all of a folio.

Parameters

struct folio *folio: The folio which is affected.
size_t offset: start of the range to invalidate
size_t length: length of the range to invalidate

Description

folio_invalidate() is called when all or part of the folio has become invalidated by a truncate operation.

folio_invalidate() does not have to release all buffers, but it must ensure that no dirty buffer is left outside offset and that no I/O is underway against any of the blocks which are outside the truncation point. Because the caller is about to free (and possibly reuse) those blocks on-disk.

void truncate_inode_pages_range(struct address_space *mapping, loff_t lstart, uoff_t lend)¶: truncate range of pages specified by start & end byte offsets

Parameters

struct address_space *mapping: mapping to truncate
loff_t lstart: offset from which to truncate
uoff_t lend: offset to which to truncate (inclusive)

Description

Truncate the page cache, removing the folios that are between specified offsets (and zeroing out partial folios if lstart or lend + 1 is not folio aligned). For mappings with a non-zero minimum folio order, the boundaries are aligned inwards to 1 << min_order so the edge sub-folio straddling the range is kept.

Truncate takes two passes - the first pass is nonblocking. It will not block on page locks and it will not block on writeback. The second pass will wait. This is to prevent as much IO as possible in the affected region. The first pass will remove most pages, so the search cost of the second pass is low.

We pass down the cache-hot hint to the page freeing code. Even if the mapping is large, it is probably the case that the final pages are the most recently touched, and freeing happens in ascending file offset order.

Note that since ->invalidate_folio() accepts range to invalidate truncate_inode_pages_range is able to handle cases where lend + 1 is not page aligned properly.

void truncate_inode_pages(struct address_space *mapping, loff_t lstart)¶: truncate all the pages from an offset

Parameters

struct address_space *mapping: mapping to truncate
loff_t lstart: offset from which to truncate

Description

Called under (and serialised by) inode->i_rwsem and mapping->invalidate_lock.

Note

When this function returns, there can be a page in the process of deletion (inside __filemap_remove_folio()) in the specified range. Thus mapping->nrpages can be non-zero when this function returns even after truncation of the whole mapping.

void truncate_inode_pages_final(struct address_space *mapping)¶: truncate all pages before inode dies

Parameters

struct address_space *mapping: mapping to truncate

Description

Called under (and serialized by) inode->i_rwsem.

Filesystems have to use this in the .evict_inode path to inform the VM that this is the final truncate and the inode is going away.

unsigned long invalidate_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t end)¶: Invalidate all clean, unlocked cache of one inode

Parameters

struct address_space *mapping: the address_space which holds the cache to invalidate
pgoff_t start: the offset ‘from’ which to invalidate
pgoff_t end: the offset ‘to’ which to invalidate (inclusive)

Description

This function removes pages that are clean, unmapped and unlocked, as well as shadow entries. It will not block on IO activity.

If you want to remove all the pages of one inode, regardless of their use and writeback state, use truncate_inode_pages().

Return

The number of indices that had their contents invalidated

int invalidate_inode_pages2_range(struct address_space *mapping, pgoff_t start, pgoff_t end)¶: remove range of pages from an address_space

Parameters

struct address_space *mapping: the address_space
pgoff_t start: the page offset ‘from’ which to invalidate
pgoff_t end: the page offset ‘to’ which to invalidate (inclusive)

Description

Any pages which are found to be mapped into pagetables are unmapped prior to invalidation.

Return

-EBUSY if any pages could not be invalidated.

int invalidate_inode_pages2(struct address_space *mapping)¶: remove all pages from an address_space

Parameters

struct address_space *mapping: the address_space

Description

Any pages which are found to be mapped into pagetables are unmapped prior to invalidation.

Return

-EBUSY if any pages could not be invalidated.

void truncate_pagecache(struct inode *inode, loff_t newsize)¶: unmap and remove pagecache that has been truncated

Parameters

struct inode *inode: inode
loff_t newsize: new file size

Description

inode’s new i_size must already be written before truncate_pagecache is called.

This function should typically be called before the filesystem releases resources associated with the freed range (eg. deallocates blocks). This way, pagecache will always stay logically coherent with on-disk format, and the filesystem would not have to deal with situations such as writepage being called for a page that has already had its underlying blocks deallocated.

void truncate_setsize(struct inode *inode, loff_t newsize)¶: update inode and pagecache for a new file size

Parameters

struct inode *inode: inode
loff_t newsize: new file size

Description

truncate_setsize updates i_size and performs pagecache truncation (if necessary) to newsize. It will be typically be called from the filesystem’s setattr function when ATTR_SIZE is passed in.

Must be called with a lock serializing truncates and writes (generally i_rwsem but e.g. xfs uses a different lock) and before all filesystem specific block truncation has been performed.

void pagecache_isize_extended(struct inode *inode, loff_t from, loff_t to)¶: update pagecache after extension of i_size

Parameters

struct inode *inode: inode for which i_size was extended
loff_t from: original inode size
loff_t to: new inode size

Description

Handle extension of inode size either caused by extending truncate or by write starting after current i_size. We mark the page straddling current i_size RO so that page_mkwrite() is called on the first write access to the page. The filesystem will update its per-block information before user writes to the page via mmap after the i_size has been changed.

The function must be called after i_size is updated so that page fault coming after we unlock the folio will already see the new i_size. The function must be called while we still hold i_rwsem - this not only makes sure i_size is stable but also that userspace cannot observe new i_size value before we are prepared to store mmap writes at new inode size.

void truncate_pagecache_range(struct inode *inode, loff_t lstart, loff_t lend)¶: unmap and remove pagecache that is hole-punched

Parameters

struct inode *inode: inode
loff_t lstart: offset of beginning of hole
loff_t lend: offset of last byte of hole

Description

void filemap_set_wb_err(struct address_space *mapping, int err)¶: set a writeback error on an address_space

Parameters

struct address_space *mapping: mapping in which to set writeback error
int err: error to be set in mapping

Description

When writeback fails in some way, we must record that error so that userspace can be informed when fsync and the like are called. We endeavor to report errors on any file that was open at the time of the error. Some internal callers also need to know when writeback errors have occurred.

When a writeback error occurs, most filesystems will want to call filemap_set_wb_err to record the error in the mapping so that it will be automatically reported whenever fsync is called on the file.

int filemap_check_wb_err(struct address_space *mapping, errseq_t since)¶: has an error occurred since the mark was sampled?

Parameters

struct address_space *mapping: mapping to check for writeback errors
errseq_t since: previously-sampled errseq_t

Description

Grab the errseq_t value from the mapping, and see if it has changed “since” the given value was sampled.

If it has then report the latest error set, otherwise return 0.

errseq_t filemap_sample_wb_err(struct address_space *mapping)¶: sample the current errseq_t to test for later errors

Parameters

struct address_space *mapping: mapping to be sampled

Description

Writeback errors are always reported relative to a particular sample point in the past. This function provides those sample points.

errseq_t file_sample_sb_err(struct file *file)¶: sample the current errseq_t to test for later errors

Parameters

struct file *file: file pointer to be sampled

Description

Grab the most current superblock-level errseq_t value for the given struct file.

void mapping_set_error(struct address_space *mapping, int error)¶: record a writeback error in the address_space

Parameters

struct address_space *mapping: the mapping in which an error should be set
int error: the error to set in the mapping

Description

When a writeback error occurs, most filesystems will want to call mapping_set_error to record the error in the mapping so that it can be reported when the application calls fsync(2).

void mapping_set_large_folios(struct address_space *mapping)¶: Indicate the file supports large folios.

Parameters

struct address_space *mapping: The address space of the file.

Description

The filesystem should call this function in its inode constructor to indicate that the VFS can use large folios to cache the contents of the file.

Context

This should not be called while the inode is active as it is non-atomic.

pgoff_t mapping_align_index(const struct address_space *mapping, pgoff_t index)¶: Align index for this mapping.

Parameters

const struct address_space *mapping: The address_space.
pgoff_t index: The page index.

Description

The index of a folio must be naturally aligned. If you are adding a new folio to the page cache and need to know what index to give it, call this function.

bool mapping_pmd_folio_support(const struct address_space *mapping)¶: Check if a mapping supports PMD-sized folio

Parameters

const struct address_space *mapping: The address_space

Description

While some mappings support large folios, they might not support PMD-sized folios. This function checks whether a mapping supports PMD-sized folios. For example, khugepaged needs this information before attempting to collapsing THPs.

Return

True if PMD-sized folios are supported, otherwise false.

struct address_space *folio_flush_mapping(struct folio *folio)¶: Find the file mapping this folio belongs to.

Parameters

struct folio *folio: The folio.

Description

For folios which are in the page cache, return the mapping that this page belongs to. Anonymous folios return NULL, even if they’re in the swap cache. Other kinds of folio also return NULL.

This is ONLY used by architecture cache flushing code. If you aren’t writing cache flushing code, you want either folio_mapping() or folio_file_mapping().

struct inode *folio_inode(struct folio *folio)¶: Get the host inode for this folio.

Parameters

struct folio *folio: The folio.

Description

For folios which are in the page cache, return the inode that this folio belongs to.

Do not call this for folios which aren’t in the page cache.

void folio_attach_private(struct folio *folio, void *data)¶: Attach private data to a folio.

Parameters

struct folio *folio: Folio to attach data to.
void *data: Data to attach to folio.

Description

Attaching private data to a folio increments the page’s reference count. The data must be detached before the folio will be freed.

void *folio_change_private(struct folio *folio, void *data)¶: Change private data on a folio.

Parameters

struct folio *folio: Folio to change the data on.
void *data: Data to set on the folio.

Description

Change the private data attached to a folio and return the old data. The page must previously have had data attached and the data must be detached before the folio will be freed.

Return

Data that was previously attached to the folio.

void *folio_detach_private(struct folio *folio)¶: Detach private data from a folio.

Parameters

struct folio *folio: Folio to detach data from.

Description

Removes the data that was previously attached to the folio and decrements the refcount on the page.

Return

Data that was attached to the folio.

type fgf_t¶: Flags for getting folios from the page cache.

Description

Most users of the page cache will not need to use these flags; there are convenience functions such as filemap_get_folio() and filemap_lock_folio(). For users which need more control over exactly what is done with the folios, these flags to __filemap_get_folio() are available.

FGP_ACCESSED - The folio will be marked accessed.
FGP_LOCK - The folio is returned locked.
FGP_CREAT - If no folio is present then a new folio is allocated, added to the page cache and the VM’s LRU list. The folio is returned locked.
FGP_FOR_MMAP - The caller wants to do its own locking dance if the folio is already in cache. If the folio was allocated, unlock it before returning so the caller can do the same dance.
FGP_WRITE - The folio will be written to by the caller.
FGP_NOFS - __GFP_FS will get cleared in gfp.
FGP_NOWAIT - Don’t block on the folio lock.
FGP_STABLE - Wait for the folio to be stable (finished writeback)
FGP_DONTCACHE - Uncached buffered IO
FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() implementation.

fgf_t fgf_set_order(size_t size)¶: Encode a length in the fgf_t flags.

Parameters

size_t size: The suggested size of the folio to create.

Description

The caller of __filemap_get_folio() can use this to suggest a preferred size for the folio that is created. If there is already a folio at the index, it will be returned, no matter what its size. If a folio is freshly created, it may be of a different size than requested due to alignment constraints, memory pressure, or the presence of other folios at nearby indices.

struct folio *write_begin_get_folio(const struct kiocb *iocb, struct address_space *mapping, pgoff_t index, size_t len)¶: Get folio for write_begin with flags.

Parameters

const struct kiocb *iocb: The kiocb passed from write_begin (may be NULL).
struct address_space *mapping: The address space to search.
pgoff_t index: The page cache index.
size_t len: Length of data being written.

Description

This is a helper for filesystem write_begin() implementations. It wraps __filemap_get_folio(), setting appropriate flags in the write begin context.

Return

A folio or an ERR_PTR.

struct folio *filemap_get_folio(struct address_space *mapping, pgoff_t index)¶: Find and get a folio.

Parameters

struct address_space *mapping: The address_space to search.
pgoff_t index: The page index.

Description

Looks up the page cache entry at mapping & index. If a folio is present, it is returned with an increased refcount.

Return

A folio or ERR_PTR(-ENOENT) if there is no folio in the cache for this index. Will not return a shadow, swap or DAX entry.

struct folio *filemap_lock_folio(struct address_space *mapping, pgoff_t index)¶: Find and lock a folio.

Parameters

struct address_space *mapping: The address_space to search.
pgoff_t index: The page index.

Description

Looks up the page cache entry at mapping & index. If a folio is present, it is returned locked with an increased refcount.

Context

May sleep.

Return

A folio or ERR_PTR(-ENOENT) if there is no folio in the cache for this index. Will not return a shadow, swap or DAX entry.

struct folio *filemap_grab_folio(struct address_space *mapping, pgoff_t index)¶: grab a folio from the page cache

Parameters

struct address_space *mapping: The address space to search
pgoff_t index: The page index

Description

Looks up the page cache entry at mapping & index. If no folio is found, a new folio is created. The folio is locked, marked as accessed, and returned.

Return

A found or created folio. ERR_PTR(-ENOMEM) if no folio is found and failed to create a folio.

struct page *find_get_page(struct address_space *mapping, pgoff_t offset)¶: find and get a page reference

Parameters

struct address_space *mapping: the address_space to search
pgoff_t offset: the page index

Description

Looks up the page cache slot at mapping & offset. If there is a page cache page, it is returned with an increased refcount.

Otherwise, NULL is returned.

struct page *find_lock_page(struct address_space *mapping, pgoff_t index)¶: locate, pin and lock a pagecache page

Parameters

struct address_space *mapping: the address_space to search
pgoff_t index: the page index

Description

Looks up the page cache entry at mapping & index. If there is a page cache page, it is returned locked and with an increased refcount.

Context

May sleep.

Return

A struct page or NULL if there is no page in the cache for this index.

struct page *find_or_create_page(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask)¶: locate or add a pagecache page

Parameters

struct address_space *mapping: the page’s address_space
pgoff_t index: the page’s index into the mapping
gfp_t gfp_mask: page allocation mode

Description

Looks up the page cache slot at mapping & offset. If there is a page cache page, it is returned locked and with an increased refcount.

If the page is not present, a new page is allocated using gfp_mask and added to the page cache and the VM’s LRU list. The page is returned locked and with an increased refcount.

On memory exhaustion, NULL is returned.

find_or_create_page() may sleep, even if gfp_flags specifies an atomic allocation!

struct page *grab_cache_page_nowait(struct address_space *mapping, pgoff_t index)¶: returns locked page at given index in given cache

Parameters

struct address_space *mapping: target address_space
pgoff_t index: the page index

Description

Returns locked page at given index in given cache, creating it if needed, but do not wait if the page is locked or to reclaim memory. This is intended for speculative data generators, where the data can be regenerated if the page couldn’t be grabbed. This routine should be safe to call while holding the lock for another page.

Clear __GFP_FS when allocating the page to avoid recursion into the fs and deadlock against the caller’s locked page.

pgoff_t folio_next_index(const struct folio *folio)¶: Get the index of the next folio.

Parameters

const struct folio *folio: The current folio.

Return

The index of the folio which follows this folio in the file.

loff_t folio_next_pos(const struct folio *folio)¶: Get the file position of the next folio.

Parameters

const struct folio *folio: The current folio.

Return

The position of the folio which follows this folio in the file.

struct page *folio_file_page(struct folio *folio, pgoff_t index)¶: The page for a particular index.

Parameters

struct folio *folio: The folio which contains this index.
pgoff_t index: The index we want to look up.

Description

Sometimes after looking up a folio in the page cache, we need to obtain the specific page for an index (eg a page fault).

Return

The page containing the file data for this index.

bool folio_contains(const struct folio *folio, pgoff_t index)¶: Does this folio contain this index?

Parameters

const struct folio *folio: The folio.
pgoff_t index: The page index within the file.

Context

The caller should have the folio locked and ensure e.g., shmem did not move this folio to the swap cache.

Return

true or false.

pgoff_t page_pgoff(const struct folio *folio, const struct page *page)¶: Calculate the logical page offset of this page.

Parameters

const struct folio *folio: The folio containing this page.
const struct page *page: The page which we need the offset of.

Description

For file pages, this is the offset from the beginning of the file in units of PAGE_SIZE. For anonymous pages, this is the offset from the beginning of the anon_vma in units of PAGE_SIZE. This will return nonsense for KSM pages.

Context

Caller must have a reference on the folio or otherwise prevent it from being split or freed.

Return

The offset in units of PAGE_SIZE.

loff_t folio_pos(const struct folio *folio)¶: Returns the byte position of this folio in its file.

Parameters

const struct folio *folio: The folio.

pgoff_t linear_page_delta(const struct vm_area_struct *vma, const unsigned long address)¶: Determine the relative page offset of address within vma.

Parameters

const struct vm_area_struct *vma: The VMA in which address resides.
const unsigned long address: The address whose relative page offset is required.

Description

The result is identical for both file-backed and anonymous mappings and simply determines how many pages address lies from vma->vm_start.

Return

The number of pages address is offset by within vma.

pgoff_t linear_page_index(const struct vm_area_struct *vma, const unsigned long address)¶: Determine the absolute page offset of address within vma.

Parameters

const struct vm_area_struct *vma: The VMA in which address resides.
const unsigned long address: The address whose absolute page offset is required.

Description

See the comment for vma_start_pgoff() for a description of what the page offset signifies.

Return

The absolute page offset of address within vma.

pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, const unsigned long address)¶: Determine the absolute anonymous page offset of address within vma.

Parameters

const struct vm_area_struct *vma: An anonymous or MAP_PRIVATE file-backed VMA in which address resides.
const unsigned long address: The address whose absolute page offset is required.

Description

This returns the anonymous page offset of address, which is the page offset the address possessed at the time the VMA was first faulted.

For anonymous mappings, this returns the same value as linear_page_index().

For MAP_PRIVATE file-backed mappings, this returns the anonymous page offset of address, which is the page offset the address possessed at the time the VMA was first faulted.

It is not valid to call this function for shared file-backed mappings.

Return

The absolute anonymous page offset of address within vma.

bool folio_trylock(struct folio *folio)¶: Attempt to lock a folio.

Parameters

struct folio *folio: The folio to attempt to lock.

Description

Sometimes it is undesirable to wait for a folio to be unlocked (eg when the locks are being taken in the wrong order, or if making progress through a batch of folios is more important than processing them in order). Usually folio_lock() is the correct function to call.

Context

Any context.

Return

Whether the lock was successfully acquired.

void folio_lock(struct folio *folio)¶: Lock this folio.

Parameters

struct folio *folio: The folio to lock.

Description

The folio lock protects against many things, probably more than it should. It is primarily held while a folio is being brought uptodate, either from its backing file or from swap. It is also held while a folio is being truncated from its address_space, so holding the lock is sufficient to keep folio->mapping stable.

The folio lock is also held while write() is modifying the page to provide POSIX atomicity guarantees (as long as the write does not cross a page boundary). Other modifications to the data in the folio do not hold the folio lock and can race with writes, eg DMA and stores to mapped pages.

Context

May sleep. If you need to acquire the locks of two or more folios, they must be in order of ascending index, if they are in the same address_space. If they are in different address_spaces, acquire the lock of the folio which belongs to the address_space which has the lowest address in memory first.

void lock_page(struct page *page)¶: Lock the folio containing this page.

Parameters

struct page *page: The page to lock.

Description

See folio_lock() for a description of what the lock protects. This is a legacy function and new code should probably use folio_lock() instead.

Context

May sleep. Pages in the same folio share a lock, so do not attempt to lock two pages which share a folio.

int folio_lock_killable(struct folio *folio)¶: Lock this folio, interruptible by a fatal signal.

Parameters

struct folio *folio: The folio to lock.

Description

Attempts to lock the folio, like folio_lock(), except that the sleep to acquire the lock is interruptible by a fatal signal.

Context

May sleep; see folio_lock().

Return

0 if the lock was acquired; -EINTR if a fatal signal was received.

bool filemap_range_needs_writeback(struct address_space *mapping, loff_t start_byte, loff_t end_byte)¶: check if range potentially needs writeback

Parameters

struct address_space *mapping: address space within which to check
loff_t start_byte: offset in bytes where the range starts
loff_t end_byte: offset in bytes where the range ends (inclusive)

Description

Find at least one page in the range supplied, usually used to check if direct writing in this range will trigger a writeback. Used by O_DIRECT read/write with IOCB_NOWAIT, to see if the caller needs to do filemap_write_and_wait_range() before proceeding.

Return

true if the caller should do filemap_write_and_wait_range() before doing O_DIRECT to a page in this range, false otherwise.

struct readahead_control¶: Describes a readahead request.

Definition:

struct readahead_control {
    struct file *file;
    struct address_space *mapping;
    struct file_ra_state *ra;
};

Members

file: The file, used primarily by network filesystems for authentication. May be NULL if invoked internally by the filesystem.
mapping: Readahead this filesystem object.
ra: File readahead state. May be NULL.

Description

A readahead request is for consecutive pages. Filesystems which implement the ->readahead method should call readahead_folio() or __readahead_batch() in a loop and attempt to start reads into each folio in the request.

Most of the fields in this struct are private and should be accessed by the functions below.

void page_cache_sync_readahead(struct address_space *mapping, struct file_ra_state *ra, struct file *file, pgoff_t index, unsigned long req_count)¶: generic file readahead

Parameters

struct address_space *mapping: address_space which holds the pagecache and I/O vectors
struct file_ra_state *ra: file_ra_state which holds the readahead state
struct file *file: Used by the filesystem for authentication.
pgoff_t index: Index of first page to be read.
unsigned long req_count: Total number of pages being read by the caller.

Description

page_cache_sync_readahead() should be called when a cache miss happened: it will submit the read. The readahead logic may decide to piggyback more pages onto the read request if access patterns suggest it will improve performance.

void page_cache_async_readahead(struct address_space *mapping, struct file_ra_state *ra, struct file *file, struct folio *folio, unsigned long req_count)¶: file readahead for marked pages

Parameters

struct address_space *mapping: address_space which holds the pagecache and I/O vectors
struct file_ra_state *ra: file_ra_state which holds the readahead state
struct file *file: Used by the filesystem for authentication.
struct folio *folio: The folio which triggered the readahead call.
unsigned long req_count: Total number of pages being read by the caller.

Description

page_cache_async_readahead() should be called when a page is used which is marked as PageReadahead; this is a marker to suggest that the application has used up enough of the readahead window that we should start pulling in more pages.

struct folio *readahead_folio(struct readahead_control *ractl)¶: Get the next folio to read.

Parameters

struct readahead_control *ractl: The current readahead request.

Context

The folio is locked. The caller should unlock the folio once all I/O to that folio has completed.

Return

A pointer to the next folio, or NULL if we are done.

struct folio *readahead_folio_last(struct readahead_control *ractl)¶: Get the next folio to read, from the tail.

Parameters

struct readahead_control *ractl: The current readahead request.

Description

Like readahead_folio(), but walks the range back-to-front. The folio is returned locked with its refcount dropped; the caller unlocks it once I/O completes. Compound folios are returned once, at their head index.

Context

The folio is locked.

Return

A pointer to the next folio, or NULL when done.

loff_t readahead_pos(const struct readahead_control *rac)¶: The byte offset into the file of this readahead request.

Parameters

const struct readahead_control *rac: The readahead request.

size_t readahead_length(const struct readahead_control *rac)¶: The number of bytes in this readahead request.

Parameters

const struct readahead_control *rac: The readahead request.

pgoff_t readahead_index(const struct readahead_control *rac)¶: The index of the first page in this readahead request.

Parameters

const struct readahead_control *rac: The readahead request.

unsigned int readahead_count(const struct readahead_control *rac)¶: The number of pages in this readahead request.

Parameters

const struct readahead_control *rac: The readahead request.

size_t readahead_batch_length(const struct readahead_control *rac)¶: The number of bytes in the current batch.

Parameters

const struct readahead_control *rac: The readahead request.

ssize_t folio_mkwrite_check_truncate(const struct folio *folio, const struct inode *inode)¶: check if folio was truncated

Parameters

const struct folio *folio: the folio to check
const struct inode *inode: the inode to check the folio against

Return

the number of bytes in the folio up to EOF, or -EFAULT if the folio was truncated.

unsigned int i_blocks_per_folio(const struct inode *inode, const struct folio *folio)¶: How many blocks fit in this folio.

Parameters

const struct inode *inode: The inode which contains the blocks.
const struct folio *folio: The folio.

Description

If the block size is larger than the size of this folio, return zero.

Context

The caller should hold a refcount on the folio to prevent it from being split.

Return

The number of filesystem blocks covered by this folio.

Memory pools¶

void mempool_exit(struct mempool *pool)¶: exit a mempool initialized with mempool_init()

Parameters

struct mempool *pool: pointer to the memory pool which was initialized with mempool_init().

Description

Free all reserved elements in pool and pool itself. This function only sleeps if the free_fn() function sleeps.

May be called on a zeroed but uninitialized mempool (i.e. allocated with kzalloc()).

void mempool_destroy(struct mempool *pool)¶: deallocate a memory pool

Parameters

struct mempool *pool: pointer to the memory pool which was allocated via mempool_create().

Description

Free all reserved elements in pool and pool itself. This function only sleeps if the free_fn() function sleeps.

int mempool_init(struct mempool *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data)¶: initialize a memory pool

Parameters

struct mempool *pool: pointer to the memory pool that should be initialized
int min_nr: the minimum number of elements guaranteed to be allocated for this pool.
mempool_alloc_t *alloc_fn: user-defined element-allocation function.
mempool_free_t *free_fn: user-defined element-freeing function.
void *pool_data: optional private data available to the user-defined functions.

Description

Like mempool_create(), but initializes the pool in (i.e. embedded in another structure).

Return

0 on success, negative error code otherwise.

struct mempool *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id)¶: create a memory pool

Parameters

int min_nr: the minimum number of elements guaranteed to be allocated for this pool.
mempool_alloc_t *alloc_fn: user-defined element-allocation function.
mempool_free_t *free_fn: user-defined element-freeing function.
void *pool_data: optional private data available to the user-defined functions.
gfp_t gfp_mask: memory allocation flags
int node_id: numa node to allocate on

Description

this function creates and allocates a guaranteed size, preallocated memory pool. The pool can be used from the mempool_alloc() and mempool_free() functions. This function might sleep. Both the alloc_fn() and the free_fn() functions might sleep - as long as the mempool_alloc() function is not called from IRQ contexts.

Return

pointer to the created memory pool object or NULL on error.

int mempool_resize(struct mempool *pool, int new_min_nr)¶: resize an existing memory pool

Parameters

struct mempool *pool: pointer to the memory pool which was allocated via mempool_create().
int new_min_nr: the new minimum number of elements guaranteed to be allocated for this pool.

Description

This function shrinks/grows the pool. In the case of growing, it cannot be guaranteed that the pool will be grown to the new size immediately, but new mempool_free() calls will refill it. This function may sleep.

Note, the caller must guarantee that no mempool_destroy is called while this function is running. mempool_alloc() & mempool_free() might be called (eg. from IRQ contexts) while this function executes.

Return

0 on success, negative error code otherwise.

int mempool_alloc_bulk(struct mempool *pool, void **elems, unsigned int count)¶: allocate multiple elements from a memory pool

Parameters

struct mempool *pool: pointer to the memory pool
void **elems: partially or fully populated elements array
unsigned int count: number of entries in elem that need to be allocated

Description

Allocate count elements into elems. This is done by first calling into the alloc_fn supplied at pool initialization time, and dipping into the reserved pool when alloc_fn fails to allocate an element.

On return all count elements in elems will be populated.

Return

Always 0. If it wasn’t for %$#^$ alloc tags, it would return void.

void *mempool_alloc(struct mempool *pool, gfp_t gfp_mask)¶: allocate an element from a memory pool

Parameters

struct mempool *pool: pointer to the memory pool
gfp_t gfp_mask: GFP_* flags. __GFP_ZERO is not supported.

Description

Allocate an element from pool. This is done by first calling into the alloc_fn supplied at pool initialization time, and dipping into the reserved pool when alloc_fn fails to allocate an element.

This function only sleeps if the alloc_fn callback sleeps, or when waiting for elements to become available in the pool.

Return

pointer to the allocated element or NULL when failing to allocate an element. Allocation failure can only happen when gfp_mask does not include __GFP_DIRECT_RECLAIM.

void *mempool_alloc_preallocated(struct mempool *pool)¶: allocate an element from preallocated elements belonging to a memory pool

Parameters

struct mempool *pool: pointer to the memory pool

Description

This function is similar to mempool_alloc(), but it only attempts allocating an element from the preallocated elements. It only takes a single spinlock_t and immediately returns if no preallocated elements are available.

Return

pointer to the allocated element or NULL if no elements are available.

unsigned int mempool_free_bulk(struct mempool *pool, void **elems, unsigned int count)¶: return elements to a mempool

Parameters

struct mempool *pool: pointer to the memory pool
void **elems: elements to return
unsigned int count: number of elements to return

Description

Returns a number of elements from the start of elem to pool if pool needs replenishing and sets their slots in elem to NULL. Other elements are left in elem.

Return

number of elements transferred to pool. Elements are always transferred from the beginning of elem, so the return value can be used as an offset into elem for the freeing the remaining elements in the caller.

void mempool_free(void *element, struct mempool *pool)¶: return an element to the pool.

Parameters

void *element: element to return
struct mempool *pool: pointer to the memory pool

Description

Returns element to pool if it needs replenishing, else frees it using the free_fn callback in pool.

This function only sleeps if the free_fn callback sleeps.

More Memory Management Functions¶

void zap_special_vma_range(struct vm_area_struct *vma, unsigned long address, unsigned long size)¶: zap all page table entries in a VMA which is not mm-managed

Parameters

struct vm_area_struct *vma: the vma covering the range to zap
unsigned long address: starting address of the range to zap
unsigned long size: number of bytes to zap

Description

This function does nothing when the provided address range is not fully contained in vma, or when vma is mm-managed.

int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, struct page **pages, unsigned long *num)¶: insert multiple pages into user vma, batching the pmd lock.

Parameters

struct vm_area_struct *vma: user vma to map to
unsigned long addr: target start user address of these pages
struct page **pages: source kernel pages
unsigned long *num: in: number of pages to map. out: number of pages that were not mapped. (0 means all pages were successfully mapped).

Description

Preferred over vm_insert_page() when inserting multiple pages.

In case of error, we may have mapped a subset of the provided pages. It is the caller’s responsibility to account for this case.

The same restrictions apply as in vm_insert_page().

int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, struct page *page)¶: insert single page into user vma

Parameters

struct vm_area_struct *vma: user vma to map to
unsigned long addr: target user address of this page
struct page *page: source kernel page

Description

This allows drivers to insert individual pages they’ve allocated into a user vma. The zeropage is supported in some VMAs, see vm_mixed_zeropage_allowed().

The page has to be a nice clean _individual_ kernel allocation. If you allocate a compound page, you need to have marked it as such (__GFP_COMP), or manually just split the page up yourself (see split_page()).

NOTE! Traditionally this was done with “remap_pfn_range()” which took an arbitrary page protection parameter. This doesn’t allow that. Your vma protection will have to be set up correctly, which means that if you want a shared writable mapping, you’d better ask for a shared writable mapping!

The page does not need to be reserved.

Usually this function is called from f_op->mmap() handler under mm->mmap_lock write-lock, so it can change vma->vm_flags. Caller must set VM_MIXEDMAP on vma if it wants to call this function from other places, for example from page-fault handler.

Return

0 on success, negative error code otherwise.

int vm_map_pages(struct vm_area_struct *vma, struct page **pages, unsigned long num)¶: maps range of kernel pages starts with non zero offset

Parameters

struct vm_area_struct *vma: user vma to map to
struct page **pages: pointer to array of source kernel pages
unsigned long num: number of pages in page array

Description

Maps an object consisting of num pages, catering for the user’s requested vm_pgoff

If we fail to insert any page into the vma, the function will return immediately leaving any previously inserted pages present. Callers from the mmap handler may immediately return the error as their caller will destroy the vma, removing any successfully inserted pages. Other callers should make their own arrangements for calling unmap_region().

Context

Process context. Called by mmap handlers.

Return

0 on success and error code otherwise.

int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages, unsigned long num)¶: map range of kernel pages starts with zero offset

Parameters

struct vm_area_struct *vma: user vma to map to
struct page **pages: pointer to array of source kernel pages
unsigned long num: number of pages in page array

Description

Similar to vm_map_pages(), except that it explicitly sets the offset to 0. This function is intended for the drivers that did not consider vm_pgoff.

Context

Process context. Called by mmap handlers.

Return

0 on success and error code otherwise.

vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, pgprot_t pgprot)¶: insert single pfn into user vma with specified pgprot

Parameters

struct vm_area_struct *vma: user vma to map to
unsigned long addr: target user address of this page
unsigned long pfn: source kernel pfn
pgprot_t pgprot: pgprot flags for the inserted page

Description

This is exactly like vmf_insert_pfn(), except that it allows drivers to override pgprot on a per-page basis.

This only makes sense for IO mappings, and it makes no sense for COW mappings. In general, using multiple vmas is preferable; vmf_insert_pfn_prot should only be used if using multiple VMAs is impractical.

pgprot typically only differs from vma->vm_page_prot when drivers set caching- and encryption bits different than those of vma->vm_page_prot, because the caching- or encryption mode may not be known at mmap() time.

This is ok as long as vma->vm_page_prot is not used by the core vm to set caching and encryption bits for those vmas (except for COW pages). This is ensured by core vm only modifying these page table entries using functions that don’t touch caching- or encryption bits, using pte_modify() if needed. (See for example mprotect()).

Also when new page-table entries are created, this is only done using the fault() callback, and never using the value of vma->vm_page_prot, except for page-table entries that point to anonymous pages as the result of COW.

Context

Process context. May allocate using GFP_KERNEL.

Return

vm_fault_t value.

vm_fault_t vmf_insert_pfn(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn)¶: insert single pfn into user vma

Parameters

struct vm_area_struct *vma: user vma to map to
unsigned long addr: target user address of this page
unsigned long pfn: source kernel pfn

Description

Similar to vm_insert_page, this allows drivers to insert individual pages they’ve allocated into a user vma. Same comments apply.

This function should only be called from a vm_ops->fault handler, and in that case the handler should return the result of this function.

vma cannot be a COW mapping.

As this is called only for pages that do not currently exist, we do not need to flush old virtual caches or the TLB.

Context

Process context. May allocate using GFP_KERNEL.

Return

vm_fault_t value.

int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, unsigned long size, pgprot_t prot)¶: remap kernel memory to userspace

Parameters

struct vm_area_struct *vma: user vma to map to
unsigned long addr: target page aligned user address to start at
unsigned long pfn: page frame number of kernel physical memory address
unsigned long size: size of mapping area
pgprot_t prot: page protection flags for this mapping

Note

this is only safe if the mm semaphore is held when called.

Return

0 on success, negative error code otherwise.

int vm_iomap_memory(struct vm_area_struct *vma, phys_addr_t start, unsigned long len)¶: remap memory to userspace

Parameters

struct vm_area_struct *vma: user vma to map to
phys_addr_t start: start of the physical memory to be mapped
unsigned long len: size of area

Description

This is a simplified io_remap_pfn_range() for common driver use. The driver just needs to give us the physical memory range to be mapped, we’ll figure out the rest from the vma information.

NOTE! Some drivers might want to tweak vma->vm_page_prot first to get whatever write-combining details or similar.

Return

0 on success, negative error code otherwise.

void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows)¶: Unmap pages from processes.

Parameters

struct address_space *mapping: The address space containing pages to be unmapped.
pgoff_t start: Index of first page to be unmapped.
pgoff_t nr: Number of pages to be unmapped. 0 to unmap to end of file.
bool even_cows: Whether to unmap even private COWed pages.

Description

Unmap the pages in this address space from any userspace process which has them mmaped. Generally, you want to remove COWed pages as well when a file is being truncated, but not when invalidating pages from the page cache.

void unmap_mapping_range(struct address_space *mapping, loff_t const holebegin, loff_t const holelen, int even_cows)¶: unmap the portion of all mmaps in the specified address_space corresponding to the specified byte range in the underlying file.

Parameters

struct address_space *mapping: the address space containing mmaps to be unmapped.
loff_t const holebegin: byte in first page to unmap, relative to the start of the underlying file. This will be rounded down to a PAGE_SIZE boundary. Note that this is different from truncate_pagecache(), which must keep the partial page. In contrast, we must get rid of partial pages.
loff_t const holelen: size of prospective hole in bytes. This will be rounded up to a PAGE_SIZE boundary. A holelen of zero truncates to the end of the file.
int even_cows: 1 when truncating a file, unmap even private COWed pages; but 0 when invalidating pagecache, don’t throw away private data.

int follow_pfnmap_start(struct follow_pfnmap_args *args)¶: Look up a pfn mapping at a user virtual address

Parameters

struct follow_pfnmap_args *args: Pointer to struct follow_pfnmap_args

Description

The caller needs to setup args->vma and args->address to point to the virtual address as the target of such lookup. On a successful return, the results will be put into other output fields.

After the caller finished using the fields, the caller must invoke another follow_pfnmap_end() to proper releases the locks and resources of such look up request.

During the start() and end() calls, the results in args will be valid as proper locks will be held. After the end() is called, all the fields in follow_pfnmap_args will be invalid to be further accessed. Further use of such information after end() may require proper synchronizations by the caller with page table updates, otherwise it can create a security bug.

If the PTE maps a refcounted page, callers are responsible to protect against invalidation with MMU notifiers; otherwise access to the PFN at a later point in time can trigger use-after-free.

Only IO mappings and raw PFN mappings are allowed. The mmap semaphore should be taken for read, and the mmap semaphore cannot be released before the end() is invoked.

This function must not be used to modify PTE content.

Return

zero on success, negative otherwise.

void follow_pfnmap_end(struct follow_pfnmap_args *args)¶: End a follow_pfnmap_start() process

Parameters

struct follow_pfnmap_args *args: Pointer to struct follow_pfnmap_args

Description

Must be used in pair of follow_pfnmap_start(). See the start() function above for more information.

int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write)¶: generic implementation for iomem mmap access

Parameters

struct vm_area_struct *vma: the vma to access
unsigned long addr: userspace address, not relative offset within vma
void *buf: buffer to read/write
int len: length of transfer
int write: set to FOLL_WRITE when writing, otherwise reading

Description

This is a generic implementation for vm_operations_struct.access for an iomem mapping. This callback is used by access_process_vm() when the vma is not page based.

unsigned long __get_pfnblock_flags_mask(const struct page *page, unsigned long pfn, unsigned long mask)¶: Return the requested group of flags for a pageblock_nr_pages block of pages

Parameters

const struct page *page: The page within the block of interest
unsigned long pfn: The target page frame number
unsigned long mask: mask of bits that the caller is interested in

Return

pageblock_bits flags

bool get_pfnblock_bit(const struct page *page, unsigned long pfn, enum pageblock_bits pb_bit)¶: Check if a standalone bit of a pageblock is set

Parameters

const struct page *page: The page within the block of interest
unsigned long pfn: The target page frame number
enum pageblock_bits pb_bit: pageblock bit to check

Return

true if the bit is set, otherwise false

enum migratetype get_pfnblock_migratetype(const struct page *page, unsigned long pfn)¶: Return the migratetype of a pageblock

Parameters

const struct page *page: The page within the block of interest
unsigned long pfn: The target page frame number

Return

The migratetype of the pageblock

Description

Use get_pfnblock_migratetype() if caller already has both page and pfn to save a call to page_to_pfn().

void __set_pfnblock_flags_mask(struct page *page, unsigned long pfn, unsigned long flags, unsigned long mask)¶: Set the requested group of flags for a pageblock_nr_pages block of pages

Parameters

struct page *page: The page within the block of interest
unsigned long pfn: The target page frame number
unsigned long flags: The flags to set
unsigned long mask: mask of bits that the caller is interested in

void set_pfnblock_bit(const struct page *page, unsigned long pfn, enum pageblock_bits pb_bit)¶: Set a standalone bit of a pageblock

Parameters

const struct page *page: The page within the block of interest
unsigned long pfn: The target page frame number
enum pageblock_bits pb_bit: pageblock bit to set

void clear_pfnblock_bit(const struct page *page, unsigned long pfn, enum pageblock_bits pb_bit)¶: Clear a standalone bit of a pageblock

Parameters

const struct page *page: The page within the block of interest
unsigned long pfn: The target page frame number
enum pageblock_bits pb_bit: pageblock bit to clear

void set_pageblock_migratetype(struct page *page, enum migratetype migratetype)¶: Set the migratetype of a pageblock

Parameters

struct page *page: The page within the block of interest
enum migratetype migratetype: migratetype to set

bool __move_freepages_block_isolate(struct zone *zone, struct page *page, bool isolate)¶: move free pages in block for page isolation

Parameters

struct zone *zone: the zone
struct page *page: the pageblock page
bool isolate: to isolate the given pageblock or unisolate it

Description

This is similar to move_freepages_block(), but handles the special case encountered in page isolation, where the block of interest might be part of a larger buddy spanning multiple pageblocks.

Unlike the regular page allocator path, which moves pages while stealing buddies off the freelist, page isolation is interested in arbitrary pfn ranges that may have overlapping buddies on both ends.

This function handles that. Straddling buddies are split into individual pageblocks. Only the block of interest is moved.

Returns true if pages could be moved, false otherwise.

void __putback_isolated_page(struct page *page, unsigned int order, int mt)¶: Return a now-isolated page back where we got it

Parameters

struct page *page: Page that was isolated
unsigned int order: Order of the isolated page
int mt: The page’s pageblock’s migratetype

Description

This function is meant to return a page pulled from the free lists via __isolate_free_page back to the free lists they were pulled from.

void __free_pages(struct page *page, unsigned int order)¶: Free pages allocated with alloc_pages().

Parameters

struct page *page: The page pointer returned from alloc_pages().
unsigned int order: The order of the allocation.

Description

This function can free multi-page allocations that are not compound pages. It does not check that the order passed in matches that of the allocation, so it is easy to leak memory. Freeing more memory than was allocated will probably emit a warning.

If the last reference to this page is speculative, it will be released by put_page() which only frees the first page of a non-compound allocation. To prevent the remaining pages from being leaked, we free the subsequent pages here. If you want to use the page’s reference count to decide when to free the allocation, you should allocate a compound page, and use put_page() instead of __free_pages().

Context

May be called in interrupt context or while holding a normal spinlock, but not in NMI context or while holding a raw spinlock.

void free_pages(unsigned long addr, unsigned int order)¶: Free pages allocated with __get_free_pages().

Parameters

unsigned long addr: The virtual address tied to a page returned from __get_free_pages().
unsigned int order: The order of the allocation.

Description

This function behaves the same as __free_pages(). Use this function to free pages when you only have a valid virtual address. If you have the page, call __free_pages() instead.

void *alloc_pages_exact(size_t size, gfp_t gfp_mask)¶: allocate an exact number physically-contiguous pages.

Parameters

size_t size: the number of bytes to allocate
gfp_t gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP

Description

This function is similar to alloc_pages(), except that it allocates the minimum number of pages to satisfy the request. alloc_pages() can only allocate memory in power-of-two pages.

This function is also limited by MAX_PAGE_ORDER.

Memory allocated by this function must be released by free_pages_exact().

Return

pointer to the allocated area or NULL in case of error.

void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask)¶: allocate an exact number of physically-contiguous pages on a node.

Parameters

int nid: the preferred node ID where memory should be allocated
size_t size: the number of bytes to allocate
gfp_t gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP

Description

Like alloc_pages_exact(), but try to allocate on node nid first before falling back.

Return

pointer to the allocated area or NULL in case of error.

void free_pages_exact(void *virt, size_t size)¶: release memory allocated via alloc_pages_exact()

Parameters

void *virt: the value returned by alloc_pages_exact.
size_t size: size of allocation, same value as passed to alloc_pages_exact().

Description

Release the memory allocated by a previous call to alloc_pages_exact.

unsigned long nr_free_zone_pages(int offset)¶: count number of pages beyond high watermark

Parameters

int offset: The zone index of the highest zone

Description

nr_free_zone_pages() counts the number of pages which are beyond the high watermark within all zones at or below a given zone index. For each zone, the number of pages is calculated as:

nr_free_zone_pages = managed_pages - high_pages

Return

number of pages beyond high watermark.

unsigned long nr_free_buffer_pages(void)¶: count number of pages beyond high watermark

Parameters

void: no arguments

Description

nr_free_buffer_pages() counts the number of pages which are beyond the high watermark within ZONE_DMA and ZONE_NORMAL.

Return

number of pages beyond high watermark within ZONE_DMA and ZONE_NORMAL.

int find_next_best_node_in(int node, nodemask_t *used_node_mask, const nodemask_t *candidates)¶: find the next node that should appear in a given node’s fallback list

Parameters

int node: node whose fallback list we’re appending
nodemask_t *used_node_mask: nodemask_t of already used nodes
const nodemask_t *candidates: nodemask_t of nodes eligible for selection

Description

We use a number of factors to determine which is the next node that should appear on a given node’s fallback list. The node should not have appeared already in node’s fallback list, and it should be the next closest node according to the distance array (which contains arbitrary distance values from each node to each node in the system), and should also prefer nodes with no CPUs, since presumably they’ll have very little allocation pressure on them otherwise.

Return

node id of the found node or NUMA_NO_NODE if no node is found.

void free_reserved_pages(struct page *page, unsigned int order)¶: free reserved pages

Parameters

struct page *page: First page to free.
unsigned int order: The page order to free.

Description

Free pages allocated through memblock during boot, letting the buddy manage them from now on.

page must be naturally aligned to the order and the order must not exceed MAX_PAGE_ORDER. All pages must be reserved.

void setup_per_zone_wmarks(void)¶: called when min_free_kbytes changes or when memory is hot-{added|removed}

Parameters

void: no arguments

Description

Ensures that the watermark[min,low,high] values for each zone are set correctly with respect to min_free_kbytes.

void __free_contig_range(unsigned long pfn, unsigned long nr_pages)¶: Free contiguous range of order-0 pages.

Parameters

unsigned long pfn: Page frame number of the first page in the range.
unsigned long nr_pages: Number of pages to free.

Description

For each order-0 struct page in the physically contiguous range, put a reference. Free any page who’s reference count falls to zero. The implementation is functionally equivalent to, but significantly faster than calling __free_page() for each struct page in a loop.

Memory allocated with alloc_pages(order>=1) then subsequently split to order-0 with split_page() is an example of appropriate contiguous pages that can be freed with this API.

Context

May be called in interrupt context or while holding a normal spinlock, but not in NMI context or while holding a raw spinlock.

int alloc_contig_frozen_range(unsigned long start, unsigned long end, acr_flags_t alloc_flags, gfp_t gfp_mask)¶

tries to allocate given range of frozen pages

Parameters

unsigned long start: start PFN to allocate
unsigned long end: one-past-the-last PFN to allocate
acr_flags_t alloc_flags: allocation information
gfp_t gfp_mask: GFP mask. Node/zone/placement hints are ignored; only some action and reclaim modifiers are supported. Reclaim modifiers control allocation behavior during compaction/migration/reclaim.

Description

The PFN range does not have to be pageblock aligned. The PFN range must belong to a single zone.

The first thing this routine does is attempt to MIGRATE_ISOLATE all pageblocks in the range. Once isolated, the pageblocks should not be modified by others.

All frozen pages which PFN is in [start, end) are allocated for the caller, and they could be freed with free_contig_frozen_range(), free_frozen_pages() also could be used to free compound frozen pages directly.

Return

zero on success or negative error code.

int alloc_contig_range(unsigned long start, unsigned long end, acr_flags_t alloc_flags, gfp_t gfp_mask)¶

tries to allocate given range of pages

Parameters

unsigned long start: start PFN to allocate
unsigned long end: one-past-the-last PFN to allocate
acr_flags_t alloc_flags: allocation information
gfp_t gfp_mask: GFP mask.

Description

This routine is a wrapper around alloc_contig_frozen_range(), it can’t be used to allocate compound pages, the refcount of each allocated page will be set to one.

All pages which PFN is in [start, end) are allocated for the caller, and should be freed with free_contig_range() or by manually calling __free_page() on each allocated page.

Return

zero on success or negative error code.

struct page *alloc_contig_frozen_pages(unsigned long nr_pages, gfp_t gfp_mask, int nid, nodemask_t *nodemask)¶

tries to find and allocate contiguous range of frozen pages

Parameters

unsigned long nr_pages: Number of contiguous pages to allocate
gfp_t gfp_mask: GFP mask. Node/zone/placement hints limit the search; only some action and reclaim modifiers are supported. Reclaim modifiers control allocation behavior during compaction/migration/reclaim.
int nid: Target node
nodemask_t *nodemask: Mask for other possible nodes

Description

This routine is a wrapper around alloc_contig_frozen_range(). It scans over zones on an applicable zonelist to find a contiguous pfn range which can then be tried for allocation with alloc_contig_frozen_range(). This routine is intended for allocation requests which can not be fulfilled with the buddy allocator.

The allocated memory is always aligned to a page boundary. If nr_pages is a power of two, then allocated range is also guaranteed to be aligned to same nr_pages (e.g. 1GB request would be aligned to 1GB).

Allocated frozen pages need be freed with free_contig_frozen_range(), or by manually calling free_frozen_pages() on each allocated frozen non-compound page, for compound frozen pages could be freed with free_frozen_pages() directly.

Return

pointer to contiguous frozen pages on success, or NULL if not successful.

struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, int nid, nodemask_t *nodemask)¶

tries to find and allocate contiguous range of pages

Parameters

unsigned long nr_pages: Number of contiguous pages to allocate
gfp_t gfp_mask: GFP mask.
int nid: Target node
nodemask_t *nodemask: Mask for other possible nodes

Description

This routine is a wrapper around alloc_contig_frozen_pages(), it can’t be used to allocate compound pages, the refcount of each allocated page will be set to one.

Allocated pages can be freed with free_contig_range() or by manually calling __free_page() on each allocated page.

Return

pointer to contiguous pages on success, or NULL if not successful.

void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)¶

free the contiguous range of frozen pages

Parameters

unsigned long pfn: start PFN to free
unsigned long nr_pages: Number of contiguous frozen pages to free

Description

This can be used to free the allocated compound/non-compound frozen pages.

void free_contig_range(unsigned long pfn, unsigned long nr_pages)¶

free the contiguous range of pages

Parameters

unsigned long pfn: start PFN to free
unsigned long nr_pages: Number of contiguous pages to free

Description

This can be only used to free the allocated non-compound pages.

struct page *alloc_pages_nolock(gfp_t gfp_flags, int nid, unsigned int order)¶: opportunistic reentrant allocation from any context

Parameters

gfp_t gfp_flags: GFP flags. Only __GFP_ACCOUNT, plus some flags that get set internally regardless (see gfp_nolock) are allowed.
int nid: node to allocate from
unsigned int order: allocation order size

Description

Allocates pages of a given order from the given node. This is safe to call from any context where RCU is watching (from atomic, NMI, and also reentrant allocator -> tracepoint -> alloc_pages_nolock_noprof). Allocation is best effort and to be expected to fail easily so nobody should rely on the success. Failures are not reported via warn_alloc(). See always fail conditions below.

Return

allocated page or NULL on failure. NULL does not mean EBUSY or EAGAIN. It means ENOMEM. There is no reason to call it again and expect !NULL.

int numa_nearest_node(int node, unsigned int state)¶: Find nearest node by state

Parameters

int node: Node id to start the search
unsigned int state: State to filter the search

Description

Lookup the closest node by distance if nid is not in state.

Return

this node if it is in state, otherwise the closest node by distance

int nearest_node_nodemask(int node, nodemask_t *mask)¶: Find the node in mask at the nearest distance from node.

Parameters

int node: a valid node ID to start the search from.
nodemask_t *mask: a pointer to a nodemask representing the allowed nodes.

Description

This function iterates over all nodes in mask and calculates the distance from the starting node, then it returns the node ID that is the closest to node, or MAX_NUMNODES if no node is found.

Note that node must be a valid node ID usable with node_distance(), providing an invalid node ID (e.g., NUMA_NO_NODE) may result in crashes or unexpected behavior.

bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, bool is_private_single_threaded)¶: check whether the folio can map prot numa

Parameters

struct folio *folio: The folio whose mapping considered for being made NUMA hintable
struct vm_area_struct *vma: The VMA that the folio belongs to.
bool is_private_single_threaded: Is this a single-threaded private VMA or not

Description

This function checks to see if the folio actually indicates that we need to make the mapping one which causes a NUMA hinting fault, as there are cases where it’s simply unnecessary, and the folio’s access time is adjusted for memory tiering if prot numa needed.

Return

True if the mapping of the folio needs to be changed, false otherwise.

struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid)¶: Allocate pages according to NUMA mempolicy.

Parameters

gfp_t gfp: GFP flags.
unsigned int order: Order of the page allocation.
struct mempolicy *pol: Pointer to the NUMA mempolicy.
pgoff_t ilx: Index for interleave mempolicy (also distinguishes alloc_pages()).
int nid: Preferred node (usually numa_node_id() but mpol may override it).

Return

The page on success or NULL if allocation fails.

struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr)¶: Allocate a folio for a VMA.

Parameters

gfp_t gfp: GFP flags.
int order: Order of the folio.
struct vm_area_struct *vma: Pointer to VMA.
unsigned long addr: Virtual address of the allocation. Must be inside vma.

Description

Allocate a folio for a specific address in vma, using the appropriate NUMA policy. The caller must hold the mmap_lock of the mm_struct of the VMA to prevent it from going away. Should be used for all allocations for folios that will be mapped into user space, excepting hugetlbfs, and excepting where direct use of folio_alloc_mpol() is more appropriate.

Return

The folio on success or NULL if allocation fails.

struct page *alloc_pages(gfp_t gfp, unsigned int order)¶: Allocate pages.

Parameters

gfp_t gfp: GFP flags.
unsigned int order: Power of two of number of pages to allocate.

Description

Allocate 1 << order contiguous pages. The physical address of the first page is naturally aligned (eg an order-3 allocation will be aligned to a multiple of 8 * PAGE_SIZE bytes). The NUMA policy of the current process is honoured when in process context.

Context

Can be called from any context, providing the appropriate GFP flags are used.

Return

The page on success or NULL if allocation fails.

int mpol_misplaced(struct folio *folio, struct vm_fault *vmf, unsigned long addr)¶: check whether current folio node is valid in policy

Parameters

struct folio *folio: folio to be checked
struct vm_fault *vmf: structure describing the fault
unsigned long addr: virtual address in vma for shared policy lookup and interleave policy

Description

Lookup current policy node id for vma,addr and “compare to” folio’s node id. Policy determination “mimics” alloc_page_vma(). Called from fault path where we know the vma and faulting address.

Return

NUMA_NO_NODE if the page is in a node that is valid for this policy, or a suitable node ID to allocate a replacement folio from.

void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)¶: initialize shared policy for inode

Parameters

struct shared_policy *sp: pointer to inode shared policy
struct mempolicy *mpol: struct mempolicy to install

Description

Install non-NULL mpol in inode’s shared policy rb-tree. On entry, the current task has a reference on a non-NULL mpol. This must be released on exit. This is called at get_inode() calls and we can use GFP_KERNEL.

int mpol_parse_str(char *str, struct mempolicy **mpol)¶: parse string to mempolicy, for tmpfs mpol mount option.

Parameters

char *str: string containing mempolicy to parse
struct mempolicy **mpol: pointer to struct mempolicy pointer, returned on success.

Description

Format of input:: <mode>[=<flags>][:<nodelist>]

Return

0 on success, else 1

void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)¶: format a mempolicy structure for printing

Parameters

char *buffer: to contain formatted mempolicy string
int maxlen: length of buffer
struct mempolicy *pol: pointer to mempolicy to be formatted

Description

Convert pol into a string. If buffer is too short, truncate the string. Recommend a maxlen of at least 51 for the longest mode, “weighted interleave”, plus the longest flag flags, “relative|balancing”, and to display at least a few node ids.

type softleaf_t¶: Describes a page table software leaf entry, abstracted from its architecture-specific encoding.

Description

Page table leaf entries are those which do not reference any descendent page tables but rather either reference a data page, are an empty (or ‘none’ entry), or contain a non-present entry.

If referencing another page table or a data page then the page table entry is pertinent to hardware - that is it tells the hardware how to decode the page table entry.

Otherwise it is a software-defined leaf page table entry, which this type describes. See leafops.h and specifically softleaf_type for a list of all possible kinds of software leaf entry.

A softleaf_t entry is abstracted from the hardware page table entry, so is not architecture-specific.

NOTE

While we transition from the confusing swp_entry_t type used for this: purpose, we simply alias this type. This will be removed once the transition is complete.

struct folio¶: Represents a contiguous set of bytes.

Definition:

struct folio {
    memdesc_flags_t flags;
    union {
        struct list_head lru;
        unsigned int mlock_count;
        struct dev_pagemap *pgmap;
    };
    struct address_space *mapping;
    union {
        pgoff_t index;
        unsigned long share;
    };
    union {
        void *private;
        swp_entry_t swap;
        unsigned long migrate_info;
    };
    atomic_t _mapcount;
    atomic_t _refcount;
#ifdef CONFIG_MEMCG;
    unsigned long memcg_data;
#elif defined(CONFIG_SLAB_OBJ_EXT);
    unsigned long _unused_slab_obj_exts;
#endif;
#if defined(WANT_PAGE_VIRTUAL);
    void *virtual;
#endif;
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS;
    int _last_cpupid;
#endif;
    atomic_t _large_mapcount;
    atomic_t _nr_pages_mapped;
#ifdef CONFIG_64BIT;
    atomic_t _entire_mapcount;
    atomic_t _pincount;
#endif ;
    mm_id_mapcount_t _mm_id_mapcount[2];
    union {
        mm_id_t _mm_id[2];
        unsigned long _mm_ids;
    };
#ifdef NR_PAGES_IN_LARGE_FOLIO;
    unsigned int _nr_pages;
#endif ;
    struct list_head _deferred_list;
#ifndef CONFIG_64BIT;
    atomic_t _entire_mapcount;
    atomic_t _pincount;
#endif ;
    void *_hugetlb_subpool;
    void *_hugetlb_cgroup;
    void *_hugetlb_cgroup_rsvd;
    void *_hugetlb_hwpoison;
};

Members

flags: Identical to the page flags.
{unnamed_union}: anonymous
lru: Least Recently Used list; tracks how recently this folio was used.
mlock_count: Number of times this folio has been pinned by mlock().
pgmap: Metadata for ZONE_DEVICE mappings
mapping: The file this page belongs to, or refers to the anon_vma for anonymous memory.
{unnamed_union}: anonymous
index: Offset within the file, in units of pages. For anonymous memory, this is the index from the beginning of the mmap.
share: number of DAX mappings that reference this folio. See dax_associate_entry.
{unnamed_union}: anonymous
private: Filesystem per-folio data (see folio_attach_private()).
swap: Used for swp_entry_t if folio_test_swapcache().
migrate_info: Stores migration state (anon_vma pointer and FOLIO_WAS_* markers).
_mapcount: Do not access this member directly. Use folio_mapcount() to find out how many times this folio is mapped by userspace.
_refcount: Do not access this member directly. Use folio_ref_count() to find how many references there are to this folio.
memcg_data: Memory Control Group data.
_unused_slab_obj_exts: Placeholder to match obj_exts in struct slab.
virtual: Virtual address in the kernel direct map.
_last_cpupid: IDs of last CPU and last process that accessed the folio.
_large_mapcount: Do not use directly, call folio_mapcount().
_nr_pages_mapped: Do not use outside of rmap and debug code.
_entire_mapcount: Do not use directly, call folio_entire_mapcount().
_pincount: Do not use directly, call folio_maybe_dma_pinned().
_mm_id_mapcount: Do not use outside of rmap code.
{unnamed_union}: anonymous
_mm_id: Do not use outside of rmap code.
_mm_ids: Do not use outside of rmap code.
_nr_pages: Do not use directly, call folio_nr_pages().
_deferred_list: Folios to be split under memory pressure.
_entire_mapcount: Do not use directly, call folio_entire_mapcount().
_pincount: Do not use directly, call folio_maybe_dma_pinned().
_hugetlb_subpool: Do not use directly, use accessor in hugetlb.h.
_hugetlb_cgroup: Do not use directly, use accessor in hugetlb_cgroup.h.
_hugetlb_cgroup_rsvd: Do not use directly, use accessor in hugetlb_cgroup.h.
_hugetlb_hwpoison: Do not use directly, call raw_hwp_list_head().

Description

A folio is a physically, virtually and logically contiguous set of bytes. It is a power-of-two in size, and it is aligned to that same power-of-two. It is at least as large as PAGE_SIZE. If it is in the page cache, it is at a file offset which is a multiple of that power-of-two. It may be mapped into userspace at an address which is at an arbitrary page offset, but its kernel virtual address is aligned to its size.

struct ptdesc¶: Memory descriptor for page tables.

Definition:

struct ptdesc {
    memdesc_flags_t pt_flags;
    union {
        struct rcu_head pt_rcu_head;
        struct list_head pt_list;
        struct {
            unsigned long _pt_pad_1;
            pgtable_t pmd_huge_pte;
        };
    };
    unsigned long __page_mapping;
    union {
        pgoff_t pt_index;
        struct mm_struct *pt_mm;
        atomic_t pt_frag_refcount;
#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING;
        atomic_t pt_share_count;
#endif;
    };
    union {
        unsigned long _pt_pad_2;
#if ALLOC_SPLIT_PTLOCKS;
        spinlock_t *ptl;
#else;
        spinlock_t ptl;
#endif;
    };
    unsigned int __page_type;
    atomic_t __page_refcount;
#ifdef CONFIG_MEMCG;
    unsigned long pt_memcg_data;
#endif;
};

Members

pt_flags: enum pt_flags plus zone/node/section.
{unnamed_union}: anonymous
pt_rcu_head: For freeing page table pages.
pt_list: List of used page tables. Used for s390 gmap shadow pages (which are not linked into the user page tables) and x86 pgds.
{unnamed_struct}: anonymous
_pt_pad_1: Padding that aliases with page’s compound head.
pmd_huge_pte: Protected by ptdesc->ptl, used for THPs.
__page_mapping: Aliases with page->mapping. Unused for page tables.
{unnamed_union}: anonymous
pt_index: Used for s390 gmap.
pt_mm: Used for x86 pgds.
pt_frag_refcount: For fragmented page table tracking. Powerpc only.
pt_share_count: Used for HugeTLB PMD page table share count.
{unnamed_union}: anonymous
_pt_pad_2: Padding to ensure proper alignment.
ptl: Lock for the page table.
ptl: Lock for the page table.
__page_type: Same as page->page_type. Unused for page tables.
__page_refcount: Same as page refcount.
pt_memcg_data: Memcg data. Tracked for page tables here.

Description

This struct overlays struct page for now. Do not modify without a good understanding of the issues.

type vm_fault_t¶: Return type for page fault handlers.

Description

Page fault handlers return a bitmask of VM_FAULT values.

enum vm_fault_reason¶: Page fault handlers return a bitmask of these values to tell the core VM what happened when handling the fault. Used to decide whether a process gets delivered SIGBUS or just gets major/minor fault counters bumped up.

Constants

VM_FAULT_OOM: Out Of Memory
VM_FAULT_SIGBUS: Bad access
VM_FAULT_MAJOR: Page read from storage
VM_FAULT_HWPOISON: Hit poisoned small page
VM_FAULT_HWPOISON_LARGE: Hit poisoned large page. Index encoded in upper bits
VM_FAULT_SIGSEGV: segmentation fault
VM_FAULT_NOPAGE: ->fault installed the pte, not return page
VM_FAULT_LOCKED: ->fault locked the returned page
VM_FAULT_RETRY: ->fault blocked, must retry
VM_FAULT_FALLBACK: huge page fault failed, fall back to small
VM_FAULT_DONE_COW: ->fault has fully handled COW
VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs fsync() to complete (for synchronous page faults in DAX)
VM_FAULT_COMPLETED: ->fault completed, meanwhile mmap lock released
VM_FAULT_HINDEX_MASK: mask HINDEX value

enum fault_flag¶: Fault flag definitions.

Constants

FAULT_FLAG_WRITE: Fault was a write fault.
FAULT_FLAG_MKWRITE: Fault was mkwrite of existing PTE.
FAULT_FLAG_ALLOW_RETRY: Allow to retry the fault if blocked.
FAULT_FLAG_RETRY_NOWAIT: Don’t drop mmap_lock and wait when retrying.
FAULT_FLAG_KILLABLE: The fault task is in SIGKILL killable region.
FAULT_FLAG_TRIED: The fault has been tried once.
FAULT_FLAG_USER: The fault originated in userspace.
FAULT_FLAG_REMOTE: The fault is not for current task/mm.
FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch.
FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals.
FAULT_FLAG_UNSHARE: The fault is an unsharing request to break COW in a COW mapping, making sure that an exclusive anon page is mapped after the fault.
FAULT_FLAG_ORIG_PTE_VALID: whether the fault has vmf->orig_pte cached. We should only access orig_pte if this flag set.
FAULT_FLAG_VMA_LOCK: The fault is handled under VMA lock.

Description

About FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_TRIED: we can specify whether we would allow page faults to retry by specifying these two fault flags correctly. Currently there can be three legal combinations:

ALLOW_RETRY and !TRIED: this means the page fault allows retry, and
this is the first try
ALLOW_RETRY and TRIED: this means the page fault allows retry, and
we’ve already tried at least once
!ALLOW_RETRY and !TRIED: this means the page fault does not allow retry

The unlisted combination (!ALLOW_RETRY && TRIED) is illegal and should never be used. Note that page faults can be allowed to retry for multiple times, in which case we’ll have an initial fault with flags (a) then later on continuous faults with flags (b). We should always try to detect pending signals before a retry to make sure the continuous page faults can still be interrupted if necessary.

The combination FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE is illegal. FAULT_FLAG_UNSHARE is ignored and treated like an ordinary read fault when applied to mappings that are not COW mappings.

int folio_is_file_lru(const struct folio *folio)¶: Should the folio be on a file LRU or anon LRU?

Parameters

const struct folio *folio: The folio to test.

Description

We would like to get this info without a page flag, but the state needs to survive until the folio is last deleted from the LRU, which could be as far down as __page_cache_release.

Return

An integer (not a boolean!) used to sort a folio onto the right LRU list and to account folios correctly. 1 if folio is a regular filesystem backed page cache folio or a lazily freed anonymous folio (e.g. via MADV_FREE). 0 if folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio.

int folio_flags_is_file_lru(const unsigned long *flags)¶: Should the folio be on a file LRU or anon LRU?

Parameters

const unsigned long *flags: The folio’s flags.

Description

Just like folio_is_file_lru but take the folio flags directly instead.

void __folio_clear_lru_flags(struct folio *folio)¶: Clear page lru flags before releasing a page.

Parameters

struct folio *folio: The folio that was on lru and now has a zero reference.

enum lru_list folio_lru_list(const struct folio *folio)¶: Which LRU list should a folio be on?

Parameters

const struct folio *folio: The folio to test.

Return

The LRU list a folio should be on, as an index into the array of LRU lists.

void lru_set_gen_flags(unsigned long *flags, int gen)¶: Set the LRU generation number to specified folio flags.

Parameters

unsigned long *flags: pointer to the folio flags
int gen: generation number, between 0 and (MAX_NR_GENS - 1), inclusive.

int lru_get_gen_flags(unsigned long flags)¶: Return the LRU generation number from folio flags.

Parameters

unsigned long flags: folio flags

Return

A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns -1 if the flags indicate the folio is off the list (e.g., isolated).

void lru_set_refs_flags(unsigned long *flags, unsigned int refs)¶: Set the LRU referenced count to folio flags.

Parameters

unsigned long *flags: pointer to the folio flags
unsigned int refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.

Description

For MGLRU, PG_referenced holds the first ref, and the extra bits hold the remaining refs. For classical LRU the extra bits are not used, so it can also be seen as the refs count never exceeds 1. In both cases, refs == 1 means PG_referenced is set and the extra bits are zero, and refs == 0 means PG_referenced and the extra bits are all unset.

int lru_get_refs_flags(unsigned long flags)¶: Return LRU referenced / access count from folio flags.

Parameters

unsigned long flags: folio flags

Description

Reads the LRU referenced count set by lru_set_refs_flags().

void folio_migrate_lru_refs(struct folio *new, const struct folio *old)¶: copy the reference state to a new folio

Parameters

struct folio *new: the destination folio
const struct folio *old: the source folio

Description

Transfer the reference state to new during migration: the MGLRU refs count, including PG_referenced, or just PG_referenced for the active/inactive LRU.

size_t num_pages_contiguous(struct page **pages, size_t nr_pages)¶: determine the number of contiguous pages that represent contiguous PFNs

Parameters

struct page **pages: an array of page pointers
size_t nr_pages: length of the array, at least 1

Description

Determine the number of contiguous pages that represent contiguous PFNs in pages, starting from the first page.

In some kernel configs contiguous PFNs will not have contiguous struct pages. In these configurations num_pages_contiguous() will return a num smaller than ideal number. The caller should continue to check for pfn contiguity after each call to num_pages_contiguous().

Returns the number of contiguous pages.

page_folio¶

page_folio (p)

Converts from page to folio.

Parameters

p: The page.

Description

Every page is part of a folio. This function cannot be called on a NULL pointer.

Context

No reference, nor lock is required on page. If the caller does not hold a reference, this call may race with a folio split, so it should re-check the folio still contains this page after gaining a reference on the folio.

Return

The folio which contains this page.

folio_page¶

folio_page (folio, n)

Return a page from a folio.

Parameters

folio: The folio.
n: The page number to return.

Description

n is relative to the start of the folio. This function does not check that the page number lies within folio; the caller is presumed to have a reference to the page.

bool folio_xor_flags_has_waiters(struct folio *folio, unsigned long mask)¶: Change some folio flags.

Parameters

struct folio *folio: The folio.
unsigned long mask: Bits set in this word will be changed.

Description

This must only be used for flags which are changed with the folio lock held. For example, it is unsafe to use for PG_dirty as that can be set without the folio lock held. It can also only be used on flags which are in the range 0-6 as some of the implementations only affect those bits.

Return

Whether there are tasks waiting on the folio.

bool folio_test_uptodate(const struct folio *folio)¶: Is this folio up to date?

Parameters

const struct folio *folio: The folio.

Description

The uptodate flag is set on a folio when every byte in the folio is at least as new as the corresponding bytes on storage. Anonymous and CoW folios are always uptodate. If the folio is not uptodate, some of the bytes in it may be; see the is_partially_uptodate() address_space operation.

bool folio_test_large(const struct folio *folio)¶: Does this folio contain more than one page?

Parameters

const struct folio *folio: The folio to test.

Return

True if the folio is larger than one page.

bool PageHuge(const struct page *page)¶: Determine if the page belongs to hugetlbfs

Parameters

const struct page *page: The page to test.

Context

Any context.

Return

True for hugetlbfs pages, false for anon pages or pages belonging to other filesystems.

bool page_has_movable_ops(const struct page *page)¶: test for a movable_ops page

Parameters

const struct page *page: The page to test.

Description

Test whether this is a movable_ops page. Such pages will stay that way until freed.

Returns true if this is a movable_ops page, otherwise false.

bool folio_has_attached_private(const struct folio *folio)¶: check if the folio has private data attached

Parameters

const struct folio *folio: The folio to check.

Description

Use this in code that may encounter swapcache or hugetlb folios but only wants to detect attached private data.

Return

true if the folio has private data attached.

int folio_has_private(const struct folio *folio)¶: Determine if folio has private stuff

Parameters

const struct folio *folio: The folio to be checked

Description

Determine if a folio has private stuff, indicating that release routines should be invoked upon it.

unsigned long folio_page_idx(const struct folio *folio, const struct page *page)¶: Return the number of a page in a folio.

Parameters

const struct folio *folio: The folio.
const struct page *page: The folio page.

Description

This function expects that the page is actually part of the folio. The returned number is relative to the start of the folio.

type vma_flag_t¶: specifies an individual VMA flag by bit number.

Description

This value is made type safe by sparse to avoid passing invalid flag values around.

bool fault_flag_allow_retry_first(enum fault_flag flags)¶: check ALLOW_RETRY the first time

Parameters

enum fault_flag flags: Fault flags.

Description

This is mostly used for places where we want to try to avoid taking the mmap_lock for too long a time when waiting for another condition to change, in which case we can try to be polite to release the mmap_lock in the first round to avoid potential starvation of other processes that would also want the mmap_lock.

Return

true if the page fault allows retry and this is the first attempt of the fault handling; false otherwise.

bool vma_flags_is_hugetlb(const vma_flags_t *flags)¶: Do the specified VMA flags indicate that the VMA is a hugetlb mapping?

Parameters

const vma_flags_t *flags: The VMA flags to test.

Return

true if the flags indicate a hugetlb mapping, false otherwise.

bool vma_is_hugetlb(const struct vm_area_struct *vma)¶: Is vma a hugetlb mapping?

Parameters

const struct vm_area_struct *vma: The VMA to test.

Return

true if vma is a hugetlb mapping, false otherwise.

bool vma_flags_is_mm_managed(const vma_flags_t *flags)¶: Do the specified VMA flags indicate that the contents of the VMA are managed by core mm?

Parameters

const vma_flags_t *flags: The VMA flags to test.

Description

Such a mapping is one established by a simple mmap() or brk() with no driver or other kernel component involved.

They are rmappable, and core mm services rmap operations including reclaim, as well as population, CoW and merge/split (if eligible).

Mappings which fail this test are those where this is not the case - e.g. MMIO, user-mapped kernel pages, udmabuf (pfnmap shmem), etc.

Return

true if the flags indicate an mm-managed mapping.

bool vma_is_mm_managed(const struct vm_area_struct *vma)¶: Are the contents of vma managed by core mm?

Parameters

const struct vm_area_struct *vma: The VMA to test.

Description

See vma_flags_is_mm_managed() for a description of this property.

Return

true if the VMA is mm-managed.

bool vma_flags_is_fixed_mapping(const vma_flags_t *flags)¶: Do the specified VMA flags indicate that this is a fixed mapping that cannot be expanded or merged?

Parameters

const vma_flags_t *flags: The VMA flags to test.

Description

Fixed mappings are those whose size is set at the point of mmap (for instance, a mapping of a fixed range of memory established by a driver), and thus cannot be expanded or merged.

Return

true if the flags indicate a fixed mapping.

bool vma_is_fixed_mapping(const struct vm_area_struct *vma)¶: Is this VMA a fixed mapping that cannot be expanded or merged?

Parameters

const struct vm_area_struct *vma: The VMA to test.

Description

See vma_flags_is_fixed_mapping() for a description of this property.

Return

true if the VMA maps a fixed mapping.

bool vma_flags_is_mm_backed(const vma_flags_t *flags)¶: Do the specified VMA flags imply the mapping is backed by core mm?

Parameters

const vma_flags_t *flags: The VMA flags to test.

Description

An mm-backed mapping is one managed by core mm (see vma_flags_is_mm_managed()) whose contents are established by core mm’s own fault path rather than a custom ->fault handler.

Mappings not managed by core mm are populated by their owner, and fixed mappings may be mm-managed but have their contents established by a custom ->fault handler, so neither are mm-backed. hugetlb mappings are fixed, but core mm owns them entirely, so they are.

These are the only mappings which can sensibly be merged, locked, dumped or be subject to uffd faulting, as all of these assume core mm and core mm alone manages their contents.

Return

true if the flags imply an mm-backed mapping, otherwise false.

bool vma_is_mm_backed(const struct vm_area_struct *vma)¶: Is vma backed by core mm?

Parameters

const struct vm_area_struct *vma: The VMA to test.

Description

See vma_flags_is_mm_backed() for details.

Return

true if the VMA is mm-backed, otherwise false.

bool vma_flags_can_merge(const vma_flags_t *flags)¶: Do the specified VMA flags permit the VMA to be merged with another?

Parameters

const vma_flags_t *flags: The VMA flags to test.

Return

true if the flags permit merging, false otherwise.

bool vma_can_merge(const struct vm_area_struct *vma)¶: Do vma’s flags permit it to be merged with another VMA?

Parameters

const struct vm_area_struct *vma: The VMA to test.

Return

true if the flags permit merging, otherwise false.

unsigned long vma_kernel_pagesize(struct vm_area_struct *vma)¶: Default page size granularity for this VMA.

Parameters

struct vm_area_struct *vma: The user mapping.

Description

The kernel page size specifies in which granularity VMA modifications can be performed. Folios in this VMA will be aligned to, and at least the size of the number of bytes returned by this function.

The default kernel page size is not affected by Transparent Huge Pages being in effect.

Return

The default page size granularity for this VMA.

unsigned int folio_order(const struct folio *folio)¶: The allocation order of a folio.

Parameters

const struct folio *folio: The folio.

Description

A folio is composed of 2^order pages. See get_order() for the definition of order.

Return

The order of the folio.

void folio_reset_order(struct folio *folio)¶: Reset the folio order and derived _nr_pages

Parameters

struct folio *folio: The folio.

Description

Reset the order and derived _nr_pages to 0. Must only be used in the process of splitting large folios.

int folio_mapcount(const struct folio *folio)¶: Number of mappings of this folio.

Parameters

const struct folio *folio: The folio.

Description

The folio mapcount corresponds to the number of present user page table entries that reference any part of a folio. Each such present user page table entry must be paired with exactly on folio reference.

For ordindary folios, each user page table entry (PTE/PMD/PUD/...) counts exactly once.

For hugetlb folios, each abstracted “hugetlb” user page table entry that references the entire folio counts exactly once, even when such special page table entries are comprised of multiple ordinary page table entries.

Will report 0 for pages which cannot be mapped into userspace, such as slab, page tables and similar.

Return

The number of times this folio is mapped.

bool folio_mapped(const struct folio *folio)¶: Is this folio mapped into userspace?

Parameters

const struct folio *folio: The folio.

Return

True if any page in this folio is referenced by user page tables.

unsigned int thp_order(struct page *page)¶: Order of a transparent huge page.

Parameters

struct page *page: Head page of a transparent huge page.

unsigned long thp_size(struct page *page)¶: Size of a transparent huge page.

Parameters

struct page *page: Head page of a transparent huge page.

Return

Number of bytes in this page.

void folio_get(struct folio *folio)¶: Increment the reference count on a folio.

Parameters

struct folio *folio: The folio.

Context

May be called in any context, as long as you know that you have a refcount on the folio. If you do not already have one, folio_try_get() may be the right interface for you to use.

void folio_put(struct folio *folio)¶: Decrement the reference count on a folio.

Parameters

struct folio *folio: The folio.

Description

If the folio’s reference count reaches zero, the memory will be released back to the page allocator and may be used by another allocation immediately. Do not access the memory or the struct folio after calling folio_put() unless you can be sure that it wasn’t the last reference.

Context

May be called in process or interrupt context, but not in NMI context. May be called while holding a spinlock.

void folio_put_refs(struct folio *folio, int refs)¶: Reduce the reference count on a folio.

Parameters

struct folio *folio: The folio.
int refs: The amount to subtract from the folio’s reference count.

Description

If the folio’s reference count reaches zero, the memory will be released back to the page allocator and may be used by another allocation immediately. Do not access the memory or the struct folio after calling folio_put_refs() unless you can be sure that these weren’t the last references.

Context

May be called in process or interrupt context, but not in NMI context. May be called while holding a spinlock.

void folios_put(struct folio_batch *folios)¶: Decrement the reference count on an array of folios.

Parameters

struct folio_batch *folios: The folios.

Description

Like folio_put(), but for a batch of folios. This is more efficient than writing the loop yourself as it will optimise the locks which need to be taken if the folios are freed. The folios batch is returned empty and ready to be reused for another batch; there is no need to reinitialise it.

Context

May be called in process or interrupt context, but not in NMI context. May be called while holding a spinlock.

bool vma_flags_is_cow_mapping(const vma_flags_t *flags)¶: Do these VMA flags imply a CoW mapping?

Parameters

const vma_flags_t *flags: The VMA flags to check.

Description

Mappings which could be CoW’d (subject to Copy-On-Write faults) are described as CoW mappings.

All mappings backed by anonymous folios (all anonymous mappings and most MAP_PRIVATE-file backed ranges) are CoW mappings.

All other mappings (including all MAP_SHARED mappings) are non-CoW.

The criteria are !VMA_SHARED_BIT, VMA_MAYWRITE_BIT.

VMA_MAYWRITE_BIT is checked instead of VMA_WRITE_BIT to account for both future mprotect() calls which can render a read-only mapping writable, and GUP with FOLL_FORCE (e.g. ptrace) which can CoW a read-only mapping.

No anonymous mapping can ever clear VMA_MAYWRITE_BIT.
Writes to anonymous mappings do not immediately result in CoW faults but may do so after the process is forked or if a read is followed by a write.
Writes to MAP_PRIVATE file-backed mappings result in CoW faults and may do so again after fork.
MAP_SHARED mappings of a file opened read-only are transformed into VMA_MAYSHARE_BIT, !VMA_SHARED_BIT, !VMA_MAYWRITE_BIT mappings, so remain non-CoW.
Drivers may clear VMA_MAYWRITE_BIT but do so at mmap() time and cannot mark themselves anonymous. Having cleared this flag it is not valid for them to leave the VMA_WRITE_BIT flag set.

As a consequence, the anonymous reverse mapping only tracks CoW mappings.

Return

true if the flags indicate a CoW mapping, otherwise false.

bool vma_is_cow_mapping(const struct vm_area_struct *vma)¶: Is this VMA a CoW mapping?

Parameters

const struct vm_area_struct *vma: The VMA to check.

Description

See vma_flags_is_cow_mapping() for details.

Return

true if the VMA is a CoW mapping, otherwise false.

bool vma_desc_is_cow_mapping(struct vm_area_desc *desc)¶: Is this VMA descriptor a CoW mapping?

Parameters

struct vm_area_desc *desc: The VMA descriptor to check.

Description

See vma_flags_is_cow_mapping() for details.

Return

true if the VMA descriptor describes a CoW mapping, otherwise false.

unsigned long folio_pfn(const struct folio *folio)¶: Return the Page Frame Number of a folio.

Parameters

const struct folio *folio: The folio.

Description

A folio may contain multiple pages. The pages have consecutive Page Frame Numbers.

Return

The Page Frame Number of the first page in the folio.

pte_t folio_mk_pte(const struct folio *folio, pgprot_t pgprot)¶: Create a PTE for this folio

Parameters

const struct folio *folio: The folio to create a PTE for
pgprot_t pgprot: The page protection bits to use

Description

Create a page table entry for the first page of this folio. This is suitable for passing to set_ptes().

Return[Page truncated to keep GeezerVison snappy.]

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.