More

Compensating lat/long offsets after conversions

Compensating lat/long offsets after conversions


In my application I am doing conversions between two different formats of data. First format is a simple one containing NE and SW coordinates for a given rectangular area:

area = { "sw": { "lat": -1234.5678, "lng": -8765.4321 }, "ne": { "lat": 1234.5678, "lng": 8765.4321 } }

The second format needs to be ETSI EN 302 931 V1.1.1 compliant, which, instead of SW/NE pairs, has a center coordinate pair and the distances to the rectangle's long and short sides. To convert, I measure the distances to the long/short sides using the haversine formula

var R = 6371; // km var φ1 = lat1.toRadians(); var φ2 = lat2.toRadians(); var Δφ = (lat2-lat1).toRadians(); var Δλ = (lon2-lon1).toRadians(); var a = Math.sin(Δφ/2) * Math.sin(Δφ/2) + Math.cos(φ1) * Math.cos(φ2) * Math.sin(Δλ/2) * Math.sin(Δλ/2); var c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a)); var d = R * c;

and find the center using simple mid-point calculations:

var centerLat = (this.ne.lat + this.sw.lat) / 2; var centerLng = (this.ne.lng + this.sw.lng) / 2;

When converting the ETSI compliant object back to the simple format I'm using this formula to calculate coordinates after moving a distance(to long/short) in a certain direction in order to recreate the SW/NE pairs. The problem is that the resultant coordinate pairs are slightly off from the original ones with offsets up to 1000s (e.g. 0.00667).

My question is if there is any way to compensate or remove the errors. Also any methodology or formula suggestions are welcome.


Vincenty's formula (ellipsoid based) is more accurate than haversine (sphere based).

Also, lat and long are usually expressed in degree, but your coordinates are not in 0-180, therefore you could be in another system than expected.


Finding the center is not as simple as you think. take an example square in EPSG:4326:

Transform it into World Mercator, and the center is somewhere else:

In Lambert conformal conical, it is not yet a rectangle:

And same for azimutal equidistant:

So be careful if you think of a "simple" rectangle and its center point. The world is not a plane! My examples are exaggerated, but you will find offsets in every projected CRS if you just take the average of the unprojected latlon values as center point.