Vector driver in Python implementation tutorial

Added in version 3.1.

Introduction

Since GDAL 3.1, the capability of writing read-only vector drivers in Python has been added. It is strongly advised to read the Vector driver implementation tutorial first, which will give the general principles of how a vector driver works.

This capability does not require the use of the GDAL/OGR SWIG Python bindings (but a vector Python driver may use them.)

Note: per project policies, this is considered as an "experimental" feature and the GDAL project will not accept such Python drivers to be included in the GDAL repository. Drivers aiming at inclusion in GDAL master should priorly be ported to C++. The rationale for this is that:

the correctness of the Python code can mostly be checked at runtime, whereas C++ benefits from static analysis (at compile time, and other checkers).
Python code is executed under the Python Global Interpreter Lock, which makes them not scale.
Not all builds of GDAL have Python available.

Linking mechanism to a Python interpreter

See Linking mechanism to a Python interpreter

Driver location

Driver filenames must start with gdal_ or ogr_ and have the .py extension. They will be searched in the following directies:

the directory pointed by the GDAL_PYTHON_DRIVER_PATH configuration option (there may be several paths separated by : on Unix or ; on Windows)
if not defined, the directory pointed by the GDAL_DRIVER_PATH configuration option.
if not defined, in the directory (hardcoded at compilation time on Unix builds) where native plugins are located.

GDAL does not try to manage Python dependencies that are imported by the driver .py script. It is up to the user to make sure its current Python environment has all required dependencies installed.

Import section

Drivers must have the following import section to load the base classes.

from gdal_python_driver import BaseDriver, BaseDataset, BaseLayer

The gdal_python_driver module is created dynamically by GDAL and is not present on the file system.

Metadata section

In the first 1000 lines of the .py file, a number of required and optional KEY=VALUE driver directives must be defined. They are parsed by C++ code, without using the Python interpreter, so it is vital to respect the following constraints:

each declaration must be on a single line, and start with # gdal: DRIVER_ (space character between sharp character and gdal, and between colon character and DRIVER_)
the value must be a literal value of type string (except for # gdal: DRIVER_SUPPORTED_API_VERSION which can accept an array of integers), without expressions, function calls, escape sequences, etc.
strings may be single or double-quoted

The following directives must be declared:

# gdal: DRIVER_NAME = "NAME": the short name of the driver
# gdal: DRIVER_SUPPORTED_API_VERSION = [1]: the API version(s) supported by the driver. Must include 1, which is the only currently supported version in GDAL 3.1
# gdal: DRIVER_DCAP_VECTOR = "YES": declares a vector driver
# gdal: DRIVER_DMD_LONGNAME = "a longer name of the driver"

Additional directives:

# gdal: DRIVER_DMD_EXTENSIONS = "ext1 ext2": list of extension(s) recognized by the driver, without the dot, and separated by space
# gdal: DRIVER_DMD_HELPTOPIC = "https://example.com/my_help.html": URL to a help page for the driver
# gdal: DRIVER_DMD_OPENOPTIONLIST = "<OpenOptionList><Option name='OPT1' type='boolean' description='bla' default='NO'/></OpenOptionList>" where the XML is an OptionOptionList.
and all other metadata items found in gdal.h starting with GDAL_DMD_ or GDAL_DCAP by creating an item name which starts with # gdal: DRIVER_ and the value of the GDAL_DMD_ or GDAL_DCAP metadata item. For example #define GDAL_DMD_CONNECTION_PREFIX "DMD_CONNECTION_PREFIX" becomes # gdal: DRIVER_DMD_CONNECTION_PREFIX

Example:

# gdal: DRIVER_NAME = "DUMMY"
# gdal: DRIVER_SUPPORTED_API_VERSION = [1]
# gdal: DRIVER_DCAP_VECTOR = "YES"
# gdal: DRIVER_DMD_LONGNAME = "my dummy plugin"
# gdal: DRIVER_DMD_EXTENSIONS = "foo bar"
# gdal: DRIVER_DMD_HELPTOPIC = "https://example.com/my_help.html"

Driver class

The entry point .py script must contains a single class that inherits from gdal_python_driver.BaseDriver.

That class must define the following methods:

identify(self, filename, first_bytes, open_flags, open_options={})

Parameters:

filename (str) -- File name, or more generally, connection string.
first_bytes (binary) -- First bytes of the file (if it is a file). At least 1024 (if the file has at least 1024 bytes), or more if a native driver in the driver probe sequence has requested more previously.
open_flags (int) -- Open flags. To be ignored for now.
open_options (dict) -- Open options.

Returns:

True if the file is recognized by the driver, False if not, or -1 if that cannot be known from the first bytes.

open(self, filename, first_bytes, open_flags, open_options={})

Parameters:

filename (str) -- File name, or more generally, connection string.
first_bytes (binary) -- First bytes of the file (if it is a file). At least 1024 (if the file has at least 1024 bytes), or more if a native driver in the driver probe sequence has requested more previously.
open_flags (int) -- Open flags. To be ignored for now.
open_options (dict) -- Open options.

Returns:

an object deriving from gdal_python_driver.BaseDataset or None

Example:

# Required: class deriving from BaseDriver
class Driver(BaseDriver):

    def identify(self, filename, first_bytes, open_flags, open_options={}):
        return filename == 'DUMMY:'

    # Required
    def open(self, filename, first_bytes, open_flags, open_options={}):
        if not self.identify(filename, first_bytes, open_flags):
            return None
        return Dataset(filename)

Dataset class

The Driver.open() method on success should return an object from a class that inherits from gdal_python_driver.BaseDataset.

Layers

The role of this object is to store vector layers. There are two implementation options. If the number of layers is small or they are fast to construct, then the __init__ method can defined a layers attribute that is a sequence of objects from a class that inherits from gdal_python_driver.BaseLayer.

Example:

class Dataset(BaseDataset):

    def __init__(self, filename):
        self.layers = [Layer(filename)]

Otherwise, the following two methods should be defined:

layer_count(self)

Returns:: the number of layers

layer(self, idx)

Parameters:: idx (int) -- Index of the layer to return. Normally between 0 and self.layer_count() - 1, but calling code might pass any value. In case of invalid index, None should be returned.
Returns:: an object deriving from gdal_python_driver.BaseLayer or None. The C++ code will take care of caching that object, and this method will only be called once for a given idx value.

Example:

class Dataset(BaseDataset):

    def layer_count(self):
        return 1

    def layer(self, idx):
        return [Layer(self.filename)] if idx = 0 else None

Metadata

The dataset may define a metadata dictionary, in __init__ of key: value of type string, for the default metadata domain. Alternatively, the following method may be implemented.

metadata(self, domain)

Parameters:: domain (str) -- metadata domain. Empty string for the default one
Returns:: None, or a dictionary of key:value pairs of type string;

Other methods

The following method may be optionally implemented:

close(self): Called at the destruction of the C++ peer GDALDataset object. Useful to close database connections for example.

Layer class

The Dataset object will instantiate one or several objects from a class that inherits from gdal_python_driver.BaseLayer.

Metadata, and other definitions

The following attributes are required and must defined at __init__ time:

name: Layer name, of type string. If not set, a name method must be defined.

fields

Sequence of field definitions (may be empty). Each field is a dictionary with the following properties:

name: Required

type: A integer value of type ogr.OFT_ (from the SWIG Python bindings), or one of the following string values: String, Integer, Integer16, Integer64, Boolean, Real, Float, Binary, Date, Time, DateTime

If that attribute is not set, a fields method must be defined and return such a sequence.

geometry_fields

Sequence of geometry field definitions (may be empty). Each field is a dictionary with the following properties:

name: Required. May be empty

type: Required. A integer value of type ogr.wkb_ (from the SWIG Python bindings), or one of the following string values: Unknown, Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometryCollections or all other values returned by OGRGeometryTypeToName()

srs: The SRS attached to the geometry field as a string that can be ingested by OGRSpatialReference::SetFromUserInput(), such as a PROJ string, WKT string, or AUTHORITY:CODE.

If that attribute is not set, a geometry_fields method must be defined and return such a sequence.

The following attributes are optional:

fid_name: Feature ID column name, of type string. May be empty string. If not set, a fid_name method may be defined.

metadata: A dictionary of key: value strings, corresponding to metadata of the default metadata domain. Alternatively, a metadata method that accepts a domain argument may be defined.

iterator_honour_attribute_filter: Can be set to True if the feature iterator takes into account the attribute_filter attribute that can be set on the layer.

iterator_honour_spatial_filter: Can be set to True if the feature iterator takes into account the spatial_filter attribute that can be set on the layer.

feature_count_honour_attribute_filter: Can be set to True if the feature_count method takes into account the attribute_filter attribute that can be set on the layer.

feature_count_honour_spatial_filter: Can be set to True if the feature_count method takes into account the spatial_filter attribute that can be set on the layer.

Feature iterator

The Layer class must implement the iterator interface, so typically with a __iter__ method.

The resulting iterator must produce dictionaries for each feature's content. The keys allowed in the returned dictionary are:

id: Strongly recommended. The value must be an integer to be recognized as a FID.

type: Required. The value must be the string "OGRFeature"

fields: Required. The value must be either a dictionary whose keys are field names; or None

geometry_fields

Required. the value must be a dictionary whose keys are geometry field names (possibly the empty string for unnamed geometry columns); or None.

The value of each key must be either a geometry encoded as a WKT string; a geometry encoded as ISO WKB as a bytes-like object; or None.

style: Optional. The value must be a string conforming to the Feature Style Specification.

Filtering

By default, any attribute or spatial filter set by the user of the OGR API will be evaluated by the generic C++ side of the driver, by iterating over all features of the layer.

If the iterator_honour_attribute_filter (resp. iterator_honour_spatial_filter) attribute of the layer object is set to True, the attribute filter (resp. spatial filter) must be honoured by the feature iterator method.

The attribute filter is set in the attribute_filter attribute of the layer object. It is a string conforming to OGR SQL. When the attribute filter is changed by the OGR API, the attribute_filter_changed optional method is called (see below paragraph about optional methods). An implementation of attribute_filter_changed may decide to fallback on evaluation by the generic C++ side of the driver by calling the SetAttributeFilter method (see below passthrough example)

The geometry filter is set in the spatial_filter attribute of the layer object. It is a string encoding as ISO WKT. It is the responsibility of the user of the OGR API to express it in the CRS of the layer. When the attribute filter is changed by the OGR API, the spatial_filter_changed optional method is called (see below paragraph about optional methods). An implementation of spatial_filter_changed may decide to fallback on evaluation by the generic C++ side of the driver by calling the SetSpatialFilter method (see below passthrough example)

Optional methods

The following methods may be optionally implemented:

extent(self, force_computation)

Returns:: the list [xmin,ymin,xmax,ymax] with the spatial extent of the layer.

feature_count(self, force_computation)

Returns:: the number of features of the layer.

If self.feature_count_honour_attribute_filter or self.feature_count_honour_spatial_filter are set to True, the attribute filter and/or spatial filter must be honoured by this method.

feature_by_id(self, fid)

Parameters:: fid (int) -- feature ID
Returns:: a feature object in one of the formats of the __next__ method described above, or None if no object matches fid

attribute_filter_changed(self): This method is called whenever self.attribute_filter has been changed. It is the opportunity for the driver to potentially change the value of self.iterator_honour_attribute_filter or feature_count_honour_attribute_filter attributes.

spatial_filter_changed(self): This method is called whenever self.spatial_filter has been changed (its value is a geometry encoded in WKT) It is the opportunity for the driver to potentially change the value of self.iterator_honour_spatial_filter or feature_count_honour_spatial_filter attributes.

test_capability(self, cap)

Parameters:: string (cap) -- potential values are BaseLayer.FastGetExtent, BaseLayer.FastSpatialFilter, BaseLayer.FastFeatureCount, BaseLayer.RandomRead, BaseLayer.StringsAsUTF8 or other strings supported by OGRLayer::TestCapability()
Returns:: True if the capability is supported, False otherwise.

Full example

The following example is a passthrough driver that forwards the calls to the SWIG Python GDAL API. It has no practical use, and is just intended to show case most possible uses of the API. A real-world driver will only use part of the API demonstrated. For example, the passthrough driver implements attribute and spatial filters in a completely dummy way, by calling back the C++ part of the driver. The iterator_honour_attribute_filter and iterator_honour_spatial_filter attributes, and the attribute_filter_changed and spatial_filter_changed method implementations, could have omitted with the same result.

The connection strings recognized by the drivers are PASSHTROUGH:connection_string_supported_by_non_python_drivers. Note that the prefixing by the driver name is absolutely not a requirement, but something specific to this particular driver which is a bit artificial (without the prefix, the connection string would go directly to the native driver). The CityJSON driver mentioned in the Other examples paragraph does not need it.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# This code is in the public domain, so as to serve as a template for
# real-world plugins.
# or, at the choice of the licensee,
# Copyright 2019 Even Rouault
# SPDX-License-Identifier: MIT

# gdal: DRIVER_NAME = "PASSTHROUGH"
# API version(s) supported. Must include 1 currently
# gdal: DRIVER_SUPPORTED_API_VERSION = [1]
# gdal: DRIVER_DCAP_VECTOR = "YES"
# gdal: DRIVER_DMD_LONGNAME = "Passthrough driver"
# gdal: DRIVER_DMD_CONNECTION_PREFIX = "PASSTHROUGH:"

from osgeo import gdal, ogr

from gdal_python_driver import BaseDriver, BaseDataset, BaseLayer

class Layer(BaseLayer):

    def __init__(self, gdal_layer):
        self.gdal_layer = gdal_layer
        self.name = gdal_layer.GetName()
        self.fid_name = gdal_layer.GetFIDColumn()
        self.metadata = gdal_layer.GetMetadata_Dict()
        self.iterator_honour_attribute_filter = True
        self.iterator_honour_spatial_filter = True
        self.feature_count_honour_attribute_filter = True
        self.feature_count_honour_spatial_filter = True

    def fields(self):
        res = []
        layer_defn = self.gdal_layer.GetLayerDefn()
        for i in range(layer_defn.GetFieldCount()):
            ogr_field_def = layer_defn.GetFieldDefn(i)
            field_def = {"name": ogr_field_def.GetName(),
                         "type": ogr_field_def.GetType()}
            res.append(field_def)
        return res

    def geometry_fields(self):
        res = []
        layer_defn = self.gdal_layer.GetLayerDefn()
        for i in range(layer_defn.GetGeomFieldCount()):
            ogr_field_def = layer_defn.GetGeomFieldDefn(i)
            field_def = {"name": ogr_field_def.GetName(),
                         "type": ogr_field_def.GetType()}
            srs = ogr_field_def.GetSpatialRef()
            if srs:
                field_def["srs"] = srs.ExportToWkt()
            res.append(field_def)
        return res

    def test_capability(self, cap):
        if cap in (BaseLayer.FastGetExtent, BaseLayer.StringsAsUTF8,
                BaseLayer.RandomRead, BaseLayer.FastFeatureCount):
            return self.gdal_layer.TestCapability(cap)
        return False

    def extent(self, force_computation):
        # Impedance mismatch between SWIG GetExtent() and the Python
        # driver API
        minx, maxx, miny, maxy = self.gdal_layer.GetExtent(force_computation)
        return [minx, miny, maxx, maxy]

    def feature_count(self, force_computation):
        # Dummy implementation: we call back the generic C++ implementation
        return self.gdal_layer.GetFeatureCount(True)

    def attribute_filter_changed(self):
        # Dummy implementation: we call back the generic C++ implementation
        if self.attribute_filter:
            self.gdal_layer.SetAttributeFilter(str(self.attribute_filter))
        else:
            self.gdal_layer.SetAttributeFilter(None)

    def spatial_filter_changed(self):
        # Dummy implementation: we call back the generic C++ implementation
        # the 'inf' test is just for a test_ogrsf oddity
        if self.spatial_filter and 'inf' not in self.spatial_filter:
            self.gdal_layer.SetSpatialFilter(
                ogr.CreateGeometryFromWkt(self.spatial_filter))
        else:
            self.gdal_layer.SetSpatialFilter(None)

    def _translate_feature(self, ogr_f):
        fields = {}
        layer_defn = ogr_f.GetDefnRef()
        for i in range(ogr_f.GetFieldCount()):
            if ogr_f.IsFieldSet(i):
                fields[layer_defn.GetFieldDefn(i).GetName()] = ogr_f.GetField(i)
        geom_fields = {}
        for i in range(ogr_f.GetGeomFieldCount()):
            g = ogr_f.GetGeomFieldRef(i)
            if g:
                geom_fields[layer_defn.GetGeomFieldDefn(
                    i).GetName()] = g.ExportToIsoWKb()
        return {'id': ogr_f.GetFID(),
                'type': 'OGRFeature',
                'style': ogr_f.GetStyleString(),
                'fields': fields,
                'geometry_fields': geom_fields}

    def __iter__(self):
        for f in self.gdal_layer:
            yield self._translate_feature(f)

    def feature_by_id(self, fid):
        ogr_f = self.gdal_layer.GetFeature(fid)
        if not ogr_f:
            return None
        return self._translate_feature(ogr_f)

class Dataset(BaseDataset):

    def __init__(self, gdal_ds):
        self.gdal_ds = gdal_ds
        self.layers = [Layer(gdal_ds.GetLayer(idx))
                    for idx in range(gdal_ds.GetLayerCount())]
        self.metadata = gdal_ds.GetMetadata_Dict()

    def close(self):
        del self.gdal_ds
        self.gdal_ds = None


class Driver(BaseDriver):

    def _identify(self, filename):
        prefix = 'PASSTHROUGH:'
        if not filename.startswith(prefix):
            return None
        return gdal.OpenEx(filename[len(prefix):], gdal.OF_VECTOR)

    def identify(self, filename, first_bytes, open_flags, open_options={}):
        return self._identify(filename) is not None

    def open(self, filename, first_bytes, open_flags, open_options={}):
        gdal_ds = self._identify(filename)
        if not gdal_ds:
            return None
        return Dataset(gdal_ds)

Other examples

Other examples, including a CityJSON driver, may be found at https://github.com/OSGeo/gdal/tree/master/examples/pydrivers