Why can shapely/geos parse this 'invalid' Well Known Binary?
Question: I am trying to parse [Well Known Binary](https://en.wikipedia.org/wiki/Well-
known_text#Well-known_binary) a binary encoding of geometry objects used in
Geographic Information Systems (GIS). I am using [this spec from
ESRI](http://edndoc.esri.com/arcsde/9.1/general_topics/wkb_representation.htm)
(same results [here from
esri](http://webhelp.esri.com/arcgisserver/9.3/dotnet/index.htm#geodatabases/the_ogc_103951442.htm)).
I have input data from [Osmosis](http://wiki.openstreetmap.org/wiki/Osmosis) a
tool to parse OpenStreetMap data, specifically the [pgsimp-dump
format](http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.43#--write-
pgsimp-dump_.28--wsd.29) which gives the hex represenation of the binary.

The ESRI docs say that there should only be 21 bytes for a `Point`, 1 byte for
byte order, 4 for uint32 for typeid, and 8 for double x and 8 for double y.

An example from osmosis is this (hex) example:
`0101000020E6100000DB81DF2B5F7822C0DFBB7262B4744A40`, which is 25 bytes long.

[Shapely](http://toblerity.org/shapely/) a python programme to parse WKB
(etc), which is based on the popular C library
[GEOS](http://trac.osgeo.org/geos/) **is** able to parse this string:

    
    
    >>> import shapely.wkb
    >>> shapely.wkb.loads("0101000020E6100000DB81DF2B5F7822C0DFBB7262B4744A40", hex=True)
    <shapely.geometry.point.Point object at 0x7f221f2581d0>
    

When I ask Shapely to parse from then convert to WKB I get a 21 bytes.

    
    
    >>> shapely.wkb.loads("0101000020E6100000DB81DF2B5F7822C0DFBB7262B4744A40", hex=True).wkb.encode("hex").upper()
    '0101000000DB81DF2B5F7822C0DFBB7262B4744A40'
    

The difference is the 4 bytes in the middle, which appear 3 bytes into the
uint32 for the typeif=d

    
    
    01010000**20E61000**00DB81DF2B5F7822C0DFBB7262B4744A40
    

Why can shapely/geos parse this WKB when it's invalid WKB? What do these bytes
mean?


Answer: GEOS / Shapely use an Extended variant of WKT/WKB called EWKT / EWKB, which is
[documented](https://github.com/postgis/postgis/blob/2.1.0/doc/ZMSgeoms.txt)
by PostGIS. If you have access to PostGIS, you can see what's going on here:

    
    
    SELECT ST_AsEWKT('0101000020E6100000DB81DF2B5F7822C0DFBB7262B4744A40'::geometry);
    

Returns the EWKT `SRID=4326;POINT(-9.2351011 52.9117549)`. So the extra data
was the spatial reference identifier, or SRID. Specifically
[EPSG:4326](http://epsg.io/4326) for WGS 84.

Shapely [does not support
SRIDs](https://github.com/Toblerity/Shapely/pull/132), however there are a few
hacks, e.g.:

    
    
    from shapely import geos
    geos.WKBWriter.defaults['include_srid'] = True
    

should now make `wkb` or `wkb_hex` output the EWKB, which includes the SRID.
The default is `False`, which would output ISO WKB for 2D geometries (but not
for 3D).

So it seems your objective is to convert EWKB to ISO WKB, which you can do
with GEOS / Shapely for 2D geometries only. If you have 3D (Z or M) or 4D (ZM)
geometries, then only PostGIS is able to do this conversion.

