Typedefs | Enumerations | Functions

cpl_virtualmem.h File Reference

Virtual memory management. More...

#include "cpl_port.h"
#include "cpl_vsi.h"

Go to the source code of this file.

Typedefs

typedef struct CPLVirtualMem CPLVirtualMem
 Opaque type that represents a virtual memory mapping.
typedef void(* CPLVirtualMemCachePageCbk )(CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)
 Callback triggered when a still unmapped page of virtual memory is accessed.
typedef void(* CPLVirtualMemUnCachePageCbk )(CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)
 Callback triggered when a dirty mapped page is going to be freed.
typedef void(* CPLVirtualMemFreeUserData )(void *pUserData)
 Callback triggered when a virtual memory mapping is destroyed.

Enumerations

enum  CPLVirtualMemAccessMode { VIRTUALMEM_READONLY, VIRTUALMEM_READONLY_ENFORCED, VIRTUALMEM_READWRITE }
 

Access mode of a virtual memory mapping.

More...

Functions

size_t CPLGetPageSize (void)
 Return the size of a page of virtual memory.
CPLVirtualMemCPLVirtualMemNew (size_t nSize, size_t nCacheSize, size_t nPageSizeHint, int bSingleThreadUsage, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemCachePageCbk pfnCachePage, CPLVirtualMemUnCachePageCbk pfnUnCachePage, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
 Create a new virtual memory mapping.
int CPLIsVirtualMemFileMapAvailable (void)
 Return if virtual memory mapping of a file is available.
CPLVirtualMemCPLVirtualMemFileMapNew (VSILFILE *fp, vsi_l_offset nOffset, vsi_l_offset nLength, CPLVirtualMemAccessMode eAccessMode, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
 Create a new virtual memory mapping from a file.
CPLVirtualMemCPLVirtualMemDerivedNew (CPLVirtualMem *pVMemBase, vsi_l_offset nOffset, vsi_l_offset nSize, CPLVirtualMemFreeUserData pfnFreeUserData, void *pCbkUserData)
 Create a new virtual memory mapping derived from an other virtual memory mapping.
void CPLVirtualMemFree (CPLVirtualMem *ctxt)
 Free a virtual memory mapping.
void * CPLVirtualMemGetAddr (CPLVirtualMem *ctxt)
 Return the pointer to the start of a virtual memory mapping.
size_t CPLVirtualMemGetSize (CPLVirtualMem *ctxt)
 Return the size of the virtual memory mapping.
int CPLVirtualMemIsFileMapping (CPLVirtualMem *ctxt)
 Return if the virtal memory mapping is a direct file mapping.
CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode (CPLVirtualMem *ctxt)
 Return the access mode of the virtual memory mapping.
size_t CPLVirtualMemGetPageSize (CPLVirtualMem *ctxt)
 Return the page size associated to a virtual memory mapping.
int CPLVirtualMemIsAccessThreadSafe (CPLVirtualMem *ctxt)
 Return TRUE if this memory mapping can be accessed safely from concurrent threads.
void CPLVirtualMemDeclareThread (CPLVirtualMem *ctxt)
 Declare that a thread will access a virtual memory mapping.
void CPLVirtualMemUnDeclareThread (CPLVirtualMem *ctxt)
 Declare that a thread will stop accessing a virtual memory mapping.
void CPLVirtualMemPin (CPLVirtualMem *ctxt, void *pAddr, size_t nSize, int bWriteOp)
 Make sure that a region of virtual memory will be realized.
void CPLVirtualMemManagerTerminate (void)
 Cleanup any resource and handlers related to virtual memory.

Detailed Description

Virtual memory management.

This file provides mechanism to define virtual memory mappings, whose content is allocated transparently and filled on-the-fly. Those virtual memory mappings can be much larger than the available RAM, but only parts of the virtual memory mapping, in the limit of the allowed the cache size, will actually be physically allocated.

This exploits low-level mechanisms of the operating system (virtual memory allocation, page protection and handler of virtual memory exceptions).

It is also possible to create a virtual memory mapping from a file or part of a file.

The current implementation is Linux only.


Typedef Documentation

typedef struct CPLVirtualMem CPLVirtualMem

Opaque type that represents a virtual memory mapping.

typedef void(* CPLVirtualMemCachePageCbk)(CPLVirtualMem *ctxt, size_t nOffset, void *pPageToFill, size_t nToFill, void *pUserData)

Callback triggered when a still unmapped page of virtual memory is accessed.

The callback has the responsibility of filling the page with relevant values

Parameters:
ctxt virtual memory handle.
nOffset offset of the page in the memory mapping.
pPageToFill address of the page to fill. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
nToFill number of bytes of the page.
pUserData user data that was passed to CPLVirtualMemNew().
typedef void(* CPLVirtualMemFreeUserData)(void *pUserData)

Callback triggered when a virtual memory mapping is destroyed.

Parameters:
pUserData user data that was passed to CPLVirtualMemNew().
typedef void(* CPLVirtualMemUnCachePageCbk)(CPLVirtualMem *ctxt, size_t nOffset, const void *pPageToBeEvicted, size_t nToBeEvicted, void *pUserData)

Callback triggered when a dirty mapped page is going to be freed.

(saturation of cache, or termination of the virtual memory mapping).

Parameters:
ctxt virtual memory handle.
nOffset offset of the page in the memory mapping.
pPageToBeEvicted address of the page that will be flushed. Note that the address might be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
nToBeEvicted number of bytes of the page.
pUserData user data that was passed to CPLVirtualMemNew().

Enumeration Type Documentation

Access mode of a virtual memory mapping.

Enumerator:
VIRTUALMEM_READONLY 

The mapping is meant at being read-only, but writes will not be prevented. Note that any content written will be lost.

VIRTUALMEM_READONLY_ENFORCED 

The mapping is meant at being read-only, and this will be enforced through the operating system page protection mechanism.

VIRTUALMEM_READWRITE 

The mapping is meant at being read-write, and modified pages can be saved thanks to the pfnUnCachePage callback


Function Documentation

size_t CPLGetPageSize ( void   ) 

Return the size of a page of virtual memory.

Returns:
the page size.
Since:
GDAL 2.0
int CPLIsVirtualMemFileMapAvailable ( void   ) 

Return if virtual memory mapping of a file is available.

Returns:
TRUE if virtual memory mapping of a file is available.
Since:
GDAL 2.0
void CPLVirtualMemDeclareThread ( CPLVirtualMem ctxt  ) 

Declare that a thread will access a virtual memory mapping.

This function must be called by a thread that wants to access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemUnDeclareThread().

Parameters:
ctxt context returned by CPLVirtualMemNew().
Since:
GDAL 2.0
CPLVirtualMem* CPLVirtualMemDerivedNew ( CPLVirtualMem pVMemBase,
vsi_l_offset  nOffset,
vsi_l_offset  nSize,
CPLVirtualMemFreeUserData  pfnFreeUserData,
void *  pCbkUserData 
)

Create a new virtual memory mapping derived from an other virtual memory mapping.

This may be usefull in case of creating mapping for pixel interleaved data.

The new mapping takes a reference on the base mapping.

Parameters:
pVMemBase Base virtual memory mapping
nOffset Offset in the base virtual memory mapping from which to start the new mapping.
nSize Size of the base virtual memory mapping to expose in the the new mapping.
pfnFreeUserData callback that is called when the object is destroyed.
pCbkUserData user data passed to pfnFreeUserData.
Returns:
a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.
Since:
GDAL 2.0
CPLVirtualMem* CPLVirtualMemFileMapNew ( VSILFILE *  fp,
vsi_l_offset  nOffset,
vsi_l_offset  nLength,
CPLVirtualMemAccessMode  eAccessMode,
CPLVirtualMemFreeUserData  pfnFreeUserData,
void *  pCbkUserData 
)

Create a new virtual memory mapping from a file.

The file must be a "real" file recognized by the operating system, and not a VSI extended virtual file.

In VIRTUALMEM_READWRITE mode, updates to the memory mapping will be written in the file.

On Linux AMD64 platforms, the maximum value for nLength is 128 TB. On Linux x86 platforms, the maximum value for nLength is 2 GB.

Only supported on Linux for now.

Parameters:
fp Virtual file handle.
nOffset Offset in the file to start the mapping from.
nLength Length of the portion of the file to map into memory.
eAccessMode Permission to use for the virtual memory mapping. This must be consistant with how the file has been opened.
pfnFreeUserData callback that is called when the object is destroyed.
pCbkUserData user data passed to pfnFreeUserData.
Returns:
a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.
Since:
GDAL 2.0
void CPLVirtualMemFree ( CPLVirtualMem ctxt  ) 

Free a virtual memory mapping.

The pointer returned by CPLVirtualMemGetAddr() will no longer be valid. If the virtual memory mapping was created with read/write permissions and that they are dirty (i.e. modified) pages, they will be flushed through the pfnUnCachePage callback before being freed.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Since:
GDAL 2.0
CPLVirtualMemAccessMode CPLVirtualMemGetAccessMode ( CPLVirtualMem ctxt  ) 

Return the access mode of the virtual memory mapping.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Returns:
the access mode of the virtual memory mapping.
Since:
GDAL 2.0
void* CPLVirtualMemGetAddr ( CPLVirtualMem ctxt  ) 

Return the pointer to the start of a virtual memory mapping.

The bytes in the range [p:p+CPLVirtualMemGetSize()-1] where p is the pointer returned by this function will be valid, until CPLVirtualMemFree() is called.

Note that if a range of bytes used as an argument of a system call (such as read() or write()) contains pages that have not been "realized", the system call will fail with EFAULT. CPLVirtualMemPin() can be used to work around this issue.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Returns:
the pointer to the start of a virtual memory mapping.
Since:
GDAL 2.0
size_t CPLVirtualMemGetPageSize ( CPLVirtualMem ctxt  ) 

Return the page size associated to a virtual memory mapping.

The value returned will be at least CPLGetPageSize(), but potentially larger.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Returns:
the page size
Since:
GDAL 2.0
size_t CPLVirtualMemGetSize ( CPLVirtualMem ctxt  ) 

Return the size of the virtual memory mapping.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Returns:
the size of the virtual memory mapping.
Since:
GDAL 2.0
int CPLVirtualMemIsAccessThreadSafe ( CPLVirtualMem ctxt  ) 

Return TRUE if this memory mapping can be accessed safely from concurrent threads.

The situation that can cause problems is when several threads try to access a page of the mapping that is not yet mapped.

The return value of this function depends on whether bSingleThreadUsage has been set of not in CPLVirtualMemNew() and/or the implementation.

On Linux, this will always return TRUE if bSingleThreadUsage = FALSE.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Returns:
TRUE if this memory mapping can be accessed safely from concurrent threads.
Since:
GDAL 2.0
int CPLVirtualMemIsFileMapping ( CPLVirtualMem ctxt  ) 

Return if the virtal memory mapping is a direct file mapping.

Parameters:
ctxt context returned by CPLVirtualMemNew().
Returns:
TRUE if the virtal memory mapping is a direct file mapping.
Since:
GDAL 2.0
void CPLVirtualMemManagerTerminate ( void   ) 

Cleanup any resource and handlers related to virtual memory.

This function must be called after the last CPLVirtualMem object has been freed.

Since:
GDAL 2.0
CPLVirtualMem* CPLVirtualMemNew ( size_t  nSize,
size_t  nCacheSize,
size_t  nPageSizeHint,
int  bSingleThreadUsage,
CPLVirtualMemAccessMode  eAccessMode,
CPLVirtualMemCachePageCbk  pfnCachePage,
CPLVirtualMemUnCachePageCbk  pfnUnCachePage,
CPLVirtualMemFreeUserData  pfnFreeUserData,
void *  pCbkUserData 
)

Create a new virtual memory mapping.

This will reserve an area of virtual memory of size nSize, whose size might be potentially much larger than the physical memory available. Initially, no physical memory will be allocated. As soon as memory pages will be accessed, they will be allocated transparently and filled with the pfnCachePage callback. When the allowed cache size is reached, the least recently used pages will be unallocated.

On Linux AMD64 platforms, the maximum value for nSize is 128 TB. On Linux x86 platforms, the maximum value for nSize is 2 GB.

Only supported on Linux for now.

Note that on Linux, this function will install a SIGSEGV handler. The original handler will be restored by CPLVirtualMemManagerTerminate().

Parameters:
nSize size in bytes of the virtual memory mapping.
nCacheSize size in bytes of the maximum memory that will be really allocated (must ideally fit into RAM).
nPageSizeHint hint for the page size. Must be a multiple of the system page size, returned by CPLGetPageSize(). Minimum value is generally 4096. Might be set to 0 to let the function determine a default page size.
bSingleThreadUsage set to TRUE if there will be no concurrent threads that will access the virtual memory mapping. This can optimize performance a bit.
eAccessMode permission to use for the virtual memory mapping.
pfnCachePage callback triggered when a still unmapped page of virtual memory is accessed. The callback has the responsibility of filling the page with relevant values.
pfnUnCachePage callback triggered when a dirty mapped page is going to be freed (saturation of cache, or termination of the virtual memory mapping). Might be NULL.
pfnFreeUserData callback that can be used to free pCbkUserData. Might be NULL
pCbkUserData user data passed to pfnCachePage and pfnUnCachePage.
Returns:
a virtual memory object that must be freed by CPLVirtualMemFree(), or NULL in case of failure.
Since:
GDAL 2.0
void CPLVirtualMemPin ( CPLVirtualMem ctxt,
void *  pAddr,
size_t  nSize,
int  bWriteOp 
)

Make sure that a region of virtual memory will be realized.

Calling this function is not required, but might be usefull when debugging a process with tools like gdb or valgrind that do not naturally like segmentation fault signals.

It is also needed when wanting to provide part of virtual memory mapping to a system call such as read() or write(). If read() or write() is called on a memory region not yet realized, the call will fail with EFAULT.

Parameters:
ctxt context returned by CPLVirtualMemNew().
pAddr the memory region to pin.
nSize the size of the memory region.
bWriteOp set to TRUE if the memory are will be accessed in write mode.
Since:
GDAL 2.0
void CPLVirtualMemUnDeclareThread ( CPLVirtualMem ctxt  ) 

Declare that a thread will stop accessing a virtual memory mapping.

This function must be called by a thread that will no longer access the content of a virtual memory mapping, except if the virtual memory mapping has been created with bSingleThreadUsage = TRUE.

This function must be paired with CPLVirtualMemDeclareThread().

Parameters:
ctxt context returned by CPLVirtualMemNew().
Since:
GDAL 2.0

Generated for GDAL by doxygen 1.7.1.