GDAL
Typedefs | Enumerations | Functions
cpl_virtualmem.h File Reference

Virtual memory management. More...

#include <stddef.h>
#include "cpl_port.h"
#include "cpl_vsi.h"

Go to the source code of this file.

Typedefs

typedef struct CPLVirtualMem CPLVirtualMem
 Opaque type that represents a virtual memory mapping.
 
typedef void(* CPLVirtualMemCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)
 Callback triggered when a still unmapped page of virtual memory is accessed. More...
 
typedef void(* CPLVirtualMemUnCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)
 Callback triggered when a dirty mapped page is going to be freed. More...
 
typedef void(* CPLVirtualMemFreeUserData) (void *pUserData)
 Callback triggered when a virtual memory mapping is destroyed. More...
 

Enumerations

enum  CPLVirtualMemAccessMode { VIRTUALMEM_READONLY , VIRTUALMEM_READONLY_ENFORCED , VIRTUALMEM_READWRITE }
 Access mode of a virtual memory mapping. More...
 

Functions

size_t CPLGetPageSize (void)
 Return the size of a page of virtual memory. More...
 
CPLVirtualMem * CPLVirtualMemNew (size_t nSize, size_t nCacheSize, size_t nPageSizeHint, int bSingleThreadUsage, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemCachePageCbk pfnCachePage, CPLVirtualMemUnCachePageCbk pfnUnCachePage, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
 Create a new virtual memory mapping. More...
 
int CPLIsVirtualMemFileMapAvailable (void)
 Return if virtual memory mapping of a file is available. More...
 
CPLVirtualMem * CPLVirtualMemFileMapNew (VSILFILE *fp, vsi_l_offset nOffset, vsi_l_offset nLength, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
 Create a new virtual memory mapping from a file. More...
 
CPLVirtualMem * CPLVirtualMemDerivedNew (CPLVirtualMem *pVMemBase, vsi_l_offset nOffset, vsi_l_offset nSize, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
 Create a new virtual memory mapping derived from an other virtual memory mapping. More...
 
void CPLVirtualMemFree (CPLVirtualMem *ctxt)
 Free a virtual memory mapping. More...
 
void * CPLVirtualMemGetAddr (CPLVirtualMem *ctxt)
 Return the pointer to the start of a virtual memory mapping. More...
 
size_t CPLVirtualMemGetSize (CPLVirtualMem *ctxt)
 Return the size of the virtual memory mapping. More...
 
int CPLVirtualMemIsFileMapping (CPLVirtualMem *ctxt)
 Return if the virtual memory mapping is a direct file mapping. More...
 
CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode (CPLVirtualMem *ctxt)
 Return the access mode of the virtual memory mapping. More...
 
size_t CPLVirtualMemGetPageSize (CPLVirtualMem *ctxt)
 Return the page size associated to a virtual memory mapping. More...
 
int CPLVirtualMemIsAccessThreadSafe (CPLVirtualMem *ctxt)
 Return TRUE if this memory mapping can be accessed safely from concurrent threads. More...
 
void CPLVirtualMemDeclareThread (CPLVirtualMem *ctxt)
 Declare that a thread will access a virtual memory mapping. More...
 
void CPLVirtualMemUnDeclareThread (CPLVirtualMem *ctxt)
 Declare that a thread will stop accessing a virtual memory mapping. More...
 
void CPLVirtualMemPin (CPLVirtualMem *ctxt, void *pAddr, size_t nSize, int bWriteOp)
 Make sure that a region of virtual memory will be realized. More...
 
void CPLVirtualMemManagerTerminate (void)
 Cleanup any resource and handlers related to virtual memory. More...
 

Detailed Description

Virtual memory management.

This file provides mechanism to define virtual memory mappings, whose content is allocated transparently and filled on-the-fly. Those virtual memory mappings can be much larger than the available RAM, but only parts of the virtual memory mapping, in the limit of the allowed the cache size, will actually be physically allocated.

This exploits low-level mechanisms of the operating system (virtual memory allocation, page protection and handler of virtual memory exceptions).

It is also possible to create a virtual memory mapping from a file or part of a file.

The current implementation is Linux only.

Typedef Documentation

◆ CPLVirtualMemCachePageCbk

typedef void(* CPLVirtualMemCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)

Callback triggered when a still unmapped page of virtual memory is accessed.

The callback has the responsibility of filling the page with relevant values

Parameters
ctxtvirtual memory handle.
nOffsetoffset of the page in the memory mapping.
pPageToFilladdress of the page to fill. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr()
  • nOffset.
nToFillnumber of bytes of the page.
pUserDatauser data that was passed to CPLVirtualMemNew().

◆ CPLVirtualMemFreeUserData

typedef void(* CPLVirtualMemFreeUserData) (void *pUserData)

Callback triggered when a virtual memory mapping is destroyed.

Parameters
pUserDatauser data that was passed to CPLVirtualMemNew().

◆ CPLVirtualMemUnCachePageCbk

typedef void(* CPLVirtualMemUnCachePageCbk) (CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)

Callback triggered when a dirty mapped page is going to be freed.

(saturation of cache, or termination of the virtual memory mapping).

Parameters
ctxtvirtual memory handle.
nOffsetoffset of the page in the memory mapping.
pPageToBeEvictedaddress of the page that will be flushed. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr()
  • nOffset.
nToBeEvictednumber of bytes of the page.
pUserDatauser data that was passed to CPLVirtualMemNew().

Enumeration Type Documentation

◆ CPLVirtualMemAccessMode

Access mode of a virtual memory mapping.

Enumerator
VIRTUALMEM_READONLY 

The mapping is meant at being read-only, but writes will not be prevented. Note that any content written will be lost.

VIRTUALMEM_READONLY_ENFORCED 

The mapping is meant at being read-only, and this will be enforced through the operating system page protection mechanism.

VIRTUALMEM_READWRITE 

The mapping is meant at being read-write, and modified pages can be saved thanks to the pfnUnCachePage callback

Function Documentation

◆ CPLGetPageSize()

size_t CPLGetPageSize ( void  )

Return the size of a page of virtual memory.

Returns
the page size.
Since
GDAL 1.11

◆ CPLIsVirtualMemFileMapAvailable()

int CPLIsVirtualMemFileMapAvailable ( void  )

Return if virtual memory mapping of a file is available.

Returns
TRUE if virtual memory mapping of a file is available.
Since
GDAL 1.11

◆ CPLVirtualMemDeclareThread()

void CPLVirtualMemDeclareThread ( CPLVirtualMem *  ctxt)

Declare that a thread will access a virtual memory mapping.

This function must be called by a thread that wants to access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemUnDeclareThread().

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Since
GDAL 1.11

◆ CPLVirtualMemDerivedNew()

CPLVirtualMem * CPLVirtualMemDerivedNew ( CPLVirtualMem *  pVMemBase,
vsi_l_offset  nOffset,
vsi_l_offset  nSize,
CPLVirtualMemFreeUserData  pfnFreeUserData,
void *  pCbkUserData 
)

Create a new virtual memory mapping derived from an other virtual memory mapping.

This may be useful in case of creating mapping for pixel interleaved data.

The new mapping takes a reference on the base mapping.

Parameters
pVMemBaseBase virtual memory mapping
nOffsetOffset in the base virtual memory mapping from which to start the new mapping.
nSizeSize of the base virtual memory mapping to expose in the the new mapping.
pfnFreeUserDatacallback that is called when the object is destroyed.
pCbkUserDatauser data passed to pfnFreeUserData.
Returns
a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.
Since
GDAL 1.11

◆ CPLVirtualMemFileMapNew()

CPLVirtualMem * CPLVirtualMemFileMapNew ( VSILFILE fp,
vsi_l_offset  nOffset,
vsi_l_offset  nLength,
CPLVirtualMemAccessMode  eAccessMode,
CPLVirtualMemFreeUserData  pfnFreeUserData,
void *  pCbkUserData 
)

Create a new virtual memory mapping from a file.

The file must be a "real" file recognized by the operating system, and not a VSI extended virtual file.

In VIRTUALMEM_READWRITE mode, updates to the memory mapping will be written in the file.

On Linux AMD64 platforms, the maximum value for nLength is 128 TB. On Linux x86 platforms, the maximum value for nLength is 2 GB.

Supported on Linux only in GDAL <= 2.0, and all POSIX systems supporting mmap() in GDAL >= 2.1

Parameters
fpVirtual file handle.
nOffsetOffset in the file to start the mapping from.
nLengthLength of the portion of the file to map into memory.
eAccessModePermission to use for the virtual memory mapping. This must be consistent with how the file has been opened.
pfnFreeUserDatacallback that is called when the object is destroyed.
pCbkUserDatauser data passed to pfnFreeUserData.
Returns
a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.
Since
GDAL 1.11

◆ CPLVirtualMemFree()

void CPLVirtualMemFree ( CPLVirtualMem *  ctxt)

Free a virtual memory mapping.

The pointer returned by CPLVirtualMemGetAddr() will no longer be valid. If the virtual memory mapping was created with read/write permissions and that they are dirty (i.e. modified) pages, they will be flushed through the pfnUnCachePage callback before being freed.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Since
GDAL 1.11

◆ CPLVirtualMemGetAccessMode()

CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode ( CPLVirtualMem *  ctxt)

Return the access mode of the virtual memory mapping.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Returns
the access mode of the virtual memory mapping.
Since
GDAL 1.11

◆ CPLVirtualMemGetAddr()

void * CPLVirtualMemGetAddr ( CPLVirtualMem *  ctxt)

Return the pointer to the start of a virtual memory mapping.

The bytes in the range [p:p+CPLVirtualMemGetSize()-1] where p is the pointer returned by this function will be valid, until CPLVirtualMemFree() is called.

Note that if a range of bytes used as an argument of a system call (such as read() or write()) contains pages that have not been "realized", the system call will fail with EFAULT. CPLVirtualMemPin() can be used to work around this issue.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Returns
the pointer to the start of a virtual memory mapping.
Since
GDAL 1.11

◆ CPLVirtualMemGetPageSize()

size_t CPLVirtualMemGetPageSize ( CPLVirtualMem *  ctxt)

Return the page size associated to a virtual memory mapping.

The value returned will be at least CPLGetPageSize(), but potentially larger.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Returns
the page size
Since
GDAL 1.11

◆ CPLVirtualMemGetSize()

size_t CPLVirtualMemGetSize ( CPLVirtualMem *  ctxt)

Return the size of the virtual memory mapping.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Returns
the size of the virtual memory mapping.
Since
GDAL 1.11

◆ CPLVirtualMemIsAccessThreadSafe()

int CPLVirtualMemIsAccessThreadSafe ( CPLVirtualMem *  ctxt)

Return TRUE if this memory mapping can be accessed safely from concurrent threads.

The situation that can cause problems is when several threads try to access a page of the mapping that is not yet mapped.

The return value of this function depends on whether bSingleThreadUsage has been set of not in CPLVirtualMemNew() and/or the implementation.

On Linux, this will always return TRUE if bSingleThreadUsage = FALSE.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Returns
TRUE if this memory mapping can be accessed safely from concurrent threads.
Since
GDAL 1.11

◆ CPLVirtualMemIsFileMapping()

int CPLVirtualMemIsFileMapping ( CPLVirtualMem *  ctxt)

Return if the virtual memory mapping is a direct file mapping.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Returns
TRUE if the virtual memory mapping is a direct file mapping.
Since
GDAL 1.11

◆ CPLVirtualMemManagerTerminate()

void CPLVirtualMemManagerTerminate ( void  )

Cleanup any resource and handlers related to virtual memory.

This function must be called after the last CPLVirtualMem object has been freed.

Since
GDAL 2.0

◆ CPLVirtualMemNew()

CPLVirtualMem * CPLVirtualMemNew ( size_t  nSize,
size_t  nCacheSize,
size_t  nPageSizeHint,
int  bSingleThreadUsage,
CPLVirtualMemAccessMode  eAccessMode,
CPLVirtualMemCachePageCbk  pfnCachePage,
CPLVirtualMemUnCachePageCbk  pfnUnCachePage,
CPLVirtualMemFreeUserData  pfnFreeUserData,
void *  pCbkUserData 
)

Create a new virtual memory mapping.

This will reserve an area of virtual memory of size nSize, whose size might be potentially much larger than the physical memory available. Initially, no physical memory will be allocated. As soon as memory pages will be accessed, they will be allocated transparently and filled with the pfnCachePage callback. When the allowed cache size is reached, the least recently used pages will be unallocated.

On Linux AMD64 platforms, the maximum value for nSize is 128 TB. On Linux x86 platforms, the maximum value for nSize is 2 GB.

Only supported on Linux for now.

Note that on Linux, this function will install a SIGSEGV handler. The original handler will be restored by CPLVirtualMemManagerTerminate().

Parameters
nSizesize in bytes of the virtual memory mapping.
nCacheSizesize in bytes of the maximum memory that will be really allocated (must ideally fit into RAM).
nPageSizeHinthint for the page size. Must be a multiple of the system page size, returned by CPLGetPageSize(). Minimum value is generally 4096. Might be set to 0 to let the function determine a default page size.
bSingleThreadUsageset to TRUE if there will be no concurrent threads that will access the virtual memory mapping. This can optimize performance a bit.
eAccessModepermission to use for the virtual memory mapping.
pfnCachePagecallback triggered when a still unmapped page of virtual memory is accessed. The callback has the responsibility of filling the page with relevant values.
pfnUnCachePagecallback triggered when a dirty mapped page is going to be freed (saturation of cache, or termination of the virtual memory mapping). Might be NULL.
pfnFreeUserDatacallback that can be used to free pCbkUserData. Might be NULL
pCbkUserDatauser data passed to pfnCachePage and pfnUnCachePage.
Returns
a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.
Since
GDAL 1.11

◆ CPLVirtualMemPin()

void CPLVirtualMemPin ( CPLVirtualMem *  ctxt,
void *  pAddr,
size_t  nSize,
int  bWriteOp 
)

Make sure that a region of virtual memory will be realized.

Calling this function is not required, but might be useful when debugging a process with tools like gdb or valgrind that do not naturally like segmentation fault signals.

It is also needed when wanting to provide part of virtual memory mapping to a system call such as read() or write(). If read() or write() is called on a memory region not yet realized, the call will fail with EFAULT.

Parameters
ctxtcontext returned by CPLVirtualMemNew().
pAddrthe memory region to pin.
nSizethe size of the memory region.
bWriteOpset to TRUE if the memory are will be accessed in write mode.
Since
GDAL 1.11

◆ CPLVirtualMemUnDeclareThread()

void CPLVirtualMemUnDeclareThread ( CPLVirtualMem *  ctxt)

Declare that a thread will stop accessing a virtual memory mapping.

This function must be called by a thread that will no longer access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemDeclareThread().

Parameters
ctxtcontext returned by CPLVirtualMemNew().
Since
GDAL 1.11