Welcome to an in-depth exploration of precision drone mapping, where we delve into the practical application of Edition 2 of the American Society for Photogrammetry and Remote Sensing (ASPRS) Positional Accuracy Standards while quality testing DJI P1 and L2 sensors and our workflows. In this post, we analyze the performance of two drone mapping sensor payloads: the DJI P1 full-frame photogrammetry camera equipped with a 35mm lens; and the DJI L2, the company’s latest first-party LiDAR sensor.
We are an end user of these sensors and do not sell any equipment. Our goal is not to claim nor verify the absolute accuracy of each sensor but to understand real-world results using our local government Real-Time Virtual Reference Network (RTVRN) to collect geo-referenced position data for control, checkpoints and data tagging.
With this in mind, we propose to establish an accuracy floor for like conditions, which could further be improved upon when establishing control and checkpoints with conventional survey methods such as a total station survey. At the same time, we expect that accuracy will be reduced in more complicated terrain.
Setting the Stage: ASPRS Positional Accuracy Standards
In the intricate world of geospatial data acquisition, precision is not just a virtue; it’s an absolute necessity. Flawed measurements or systematic errors can distort entire datasets, rendering insights unreliable and jeopardizing critical decisions.
Recognizing this inherent need, ASPRS stands as a respected voice in establishing positional accuracy standards that serve as a framework for evaluating the accuracy of geospatial data.
In the following scenarios, we focus on the recent second edition of the ASPRS Positional Accuracy Standards, shaping our exploration of the DJI Zenmuse P1 and Zenmuse L2 in real-world testing. The second edition offers modernization of accuracy statistics with sensor-agnostic measures and recognizes the crucial role of ground truthing by incorporating survey checkpoint accuracy in the new framework.
Our analysis closely follows the ASPRS testing and reporting of product accuracy approach as outlined in the excellent LIDAR Magazine article by Dr. Qassim Abdullah.
Let’s get into the details!
Area of interest for testing DJI P1 and L2 sensors
We conveniently conducted our testing at the Town of Buena Vista, Colorado drone training park. This is a one-of-a-kind facility with many UAS training obstacles situated upon a gently sloping meadow. There is no dense vegetation so consider this exercise conducted on a controlled soft surface environment. In this project, we mapped approximately 25 acres.
Payload 1, Photogrammetry sensor: DJI P1 with 35mm Full Frame Prime Lens
Testing Methodology:
Establishing a project base point: While typically we tie a project to a known point, or acquire a new base point using an OPUS or PPP analysis of a static GNSS observation, in this exercise we simply connected our Emlid RS2 GNSS base to a local government RTVRS network and established a new base point with a 5-minute average RTK Fix – using the NAD83(2011) datum NTRIP mount point.
Using this newly acquired base point as our project benchmark, we collected 5 ground control points around the area of interest (each noted as GCP-x in the image above) using an Emlid RS2 GNSS rover equipped with a LoRa radio link to the GNSS base. Points were collected in State Plane survey feet, along with the residual errors for northing, easting and elevation.
Additionally, we collected 5 checkpoints with the same methodology. ASPRS calls for 30 checkpoint to confirm to their ‘tested to meet’ accuracy designation. In this quality check, we’re following the more practical ‘produced to meet’ designation.
Mapping data was collected with the DJI Matrice 300 RTK (M300) UAS and the Zenmuse P1 full-frame 35mm camera payload flying at 328ft AGL (100m), 22 mph (10m/s), 80/70 overlap and the gimbal in the nadir position. The flight mission was executed with the M300 connected to the GNSS base via local NTRIP and photos geo-tagged with 100% RTK FIX positioning. Orthophoto and digital elevation model (DEM) mapping products were produced in DJI Terra version 4.0.1 using the 5 GCPs to calibrate results to both horizontal and vertical control components.
Rather than relying on DJI’s accuracy report, checkpoint target coordinates were extracted from the produced orthophoto and DEM in GIS and subsequently compared to points collected with the Emlid RS2 rover. Product fit to checkpoints calculated as the RMSE for horizontal, vertical, and 3D.
Additionally, as newly required by the ASPRS 2nd edition, the horizontal and vertical RMSE was calculated for the five checkpoint RTK surveys along with the RMSE of the GNSS base point.
With these two components, the overall project RMSE calculated by considering both data acquisition residuals and checkpoint accuracy residuals.
Knowing that our calculations could include pixel picking error, we additionally considered the horizontal and vertical RMSE as calculated in the DJI Terra quality report to establish a ‘max’ worst case scenario using the higher RSME values between DJI and our manual calculations. Using our manual methodology along with the OEM report offers redundancy in the verification process.
Result: The P1 with ground control achieved an overall RMSE of less than 0.2 feet.
Detailed results (best viewed on desktop):
Payload 2, LiDAR sensor: DJI L2
Testing Methodology:
Project base point acquisition – the same project base point as above: (Emlid RS2 GNSS base, measured with a 5-minute average RTK Fix to the local government RTK network – NAD83(2011) datum mount point.)
5 Control points around the area of interest (same as above, collected with an Emlid RS2 rover with LoRa link to the GNSS base.)
Data collection with DJI Matrice 300 RTK (M300) and the Zenmuse L2 LiDAR sensor, 240hz, repetitive scan mode, 200 ft AGL, 22mph (10m/s). The M300 connected to the GNSS base via local NTRIP. 100% RTK positional FIX throughout the data collection. Mapping product produced in DJI Terra 4.0.1 using 5 checkpoints to calibrate results in the vertical direction only.
Checkpoint coordinates were manually extracted from the produced point cloud by selecting the closest pixel to the center of each ground target (without interpolation), and compared to checkpoint coordinates collected with the Emlid RS2 rover. ASPRS product fit to checkpoints was used to calculate as the RMSE for horizontal, vertical, and 3D.
As above, the horizontal and vertical RMSE was calculated for the five checkpoints and the base point.
The overall project RMSE was calculated by considering both data acquisition residuals and checkpoint accuracy residuals, per edition 2 of ASPRS Positional Accuracy standards.
Result: The L2 with ground control achieved an overall RMSE of less than 0.2 feet.
Detailed results (best viewed on desktop):
Bonus Scenerio: P1 RTK Only (no ground control)
Finally, we conducted a second test with the P1 data. This time we re-processed the same imagery with our company’s TopoVision™ post-processing pipeline which bolts onto the open-source photogrammetry engine from OpenDroneMap (ODM). ODM natively works only in the WGS84 datum and doesn’t handle datum transformations, hence we rely on our internally developed process and scripts to unlock reliable State Plane deliverables.
Applying ground control points in State Plane coordinates is clunky in ODM, so for this scenario we demonstrate that by using RTK-only, we can still achieve sub quarter inch 3D RMSE with the M300 and P1. This is usually more than adequate for GIS deliverables!
Result: The P1 with RTK-only achieved an overall RMSE of less than 0.25 feet.
Detailed results (best viewed on desktop):
Insights
Testing DJI P1 and L2 sensors In the controlled conditions of this area of interest, where dense vegetation is not a factor, the DJI Terra results using ground control were not discernable between the two sensors. Furthermore, the DJI P1 sensor still performed with under a quarter inch RMSE using RTK and no ground control, offering TopoVision™ as a formidable tool for many use cases like GIS data collection.
While we demonstrated the viability of TopoVision™, DJI Terra offers a better workflow for ground control and checkpoint verification of survey or engineering deliverables. That said, the OEM’s reporting tools should not be trusted alone, or without considering the impact of checkpoint errors as per the Edition 2 ASPRS standard. Furthermore, we found that when working in state plane coordinates, the DJI Terra point cloud quality report incorrectly stated units in meters rather than survey feet, requiring close attention to detail and a keen understanding of the end-to-end process.
Our main takeaway: Perform your own independent testing and verification of any OEM software reports before making their quality declarations your own.
Elevate Your Precision Mapping Game: ASPRS as Your Guide
As we conclude our exploration, the key takeaway is clear – precision mapping is not just a goal; it’s an achievable standard with the right technology and adherence to ASPRS guidelines. Whether you’re leveraging the imaging prowess of DJI P1 photogrammetry or the DJI L2-generated point cloud, ASPRS Positional Accuracy Standards can act as a compass guiding you towards mapping excellence. Depending on the terrain, we believe either product is more than capable of producing a common deliverable such as 1 ft topographic contours with the required accuracy.
Stay tuned for more real-world testing, standards exploration, and practical insights as we continue to unravel the secrets behind precision mapping. Precision isn’t just a destination; it’s a journey guided by standards that set the bar high.
Have questions, suggestions or perhaps you’ve found a major flaw in our approach? Hit the contact button and let us know! #MapLocal #TopoMatters
