Introduction
If readers
would like to contribute an article for the Guest Contributor page heading
please contact me, my email address appears on the About Me page heading. The 0nly two things I ask is that the
article should be hill related and importantly I should not end up in court
through its publication! Otherwise the
choice of subject matter is down to the Guest Contributor.
About
the Author; Joe Nuttall
 |
Over recent times LIDAR has
revolutionised hill lists with a multitude of reclassifications taking place
due to its accuracy, and a number of people are responsible for these
analyses. Joe has written a programme
that in itself is also revolutionising LIDAR analysis for those interested in
hill classification. His explanation of
this is given below with a link to the dropbox files containing a spreadsheet
with over 29,600 hills.
|
Myrddyn has asked me to say a few words about the summit analysis program I've been working on recently. With the lockdown currently in place it might provide some of you with a welcome distraction.
I wrote a program in 2016 to analyse LIDAR. It was partially successful - it found a new Nuttall (Graig Gwaun Taf positioned at SO 005 207) - but as the data for the UK is very patchy, the program could not identify every col and summit which was covered by LIDAR.
At the end of 2017 the Environment Agency announced that they were embarking on a plan to cover all England, so when this data started to become available in May 2019 I wanted to revisit my program.
The idea was to fill in the gaps in the LIDAR with an alternative height map (DEM), which would allow it to identify all summits and cols with LIDAR. As an additional goal I wanted a high enough quality DEM for the gaps so that it might spot possible new summits outside of the LIDAR.
To test the usefulness of the possible DEMs to plug the gaps I looked at Fuar-tholl Mor on Maoile Lunndaidh/Creag Toll a' Choin (NH 127 460), a three ring contour top which is only shown on the 1:25,000 map. Contours on Open Cycle Map, which come from the Shuttle Radar Topography Mission (SRTM), look very good but Fuar-tholl Mor is completely missing (I downloaded the data from NASA to verify it was missing and not a casualty of a smoothing algorithm). Next I looked to the OS who provide a free product OS Terrain 50 Grid, which is a nationwide 50m grid of heights - but this is not only extremely low resolution, it also has major errors - it completely cuts off the summit of Garnedd Ugain/Crib-y-Ddysgl (SH 610 551)! The alternative OS Terrain 50 Contours, however, is very good and clearly shows Fuar-tholl Mor. It appears to be a slightly angular version of the contours on 1:25,000 maps, but occasionally some small contours are missing. This is a vector product, so I used Python scripts in QGIS to interpolate height data from the contours.
An unexpected hurdle was the apparently simple task of downloading the LIDAR, as there are multiple different sources and formats and download methods, some of them very convoluted, this is an irritating and long winded task. It's also quite large - I downloaded about 1 TB of data - so I had to splash out on an external hard drive.
The program takes the interpolated OS height data, overlays the LIDAR height data, then looks for summits and cols. It matches the summits in the DoBIH, and then outputs data for all 29,693 tops... enough to keep anyone busy! The difficulty was getting the algorithm to process data in O (n logn) time so that it didn't grind to a halt with the large data set, and to process the data in sections and then combine the results so that it didn't run out of memory.
To allow you to check borderline tops it lists tops with drop >=20m (reduced to 10m for land over 600m). Although most of the Nuttalls have no LIDAR, I was surprised that the OS contour interpolation was good enough that all of them are picked up (and most ex-Nuttalls) except for Harrop Pike (NY 500 077).
In addition it draws a height-shaded map of Great Britain, with LIDAR coverage shaded blue, marking all summits and cols and the ridge-routes between. This was originally just for debugging purposes, but I find the map fascinating.
The program passed my main test case - it needed to spot the col for Carnedd y Ddelw (summit at SH 708 705), right on the edge of the LIDAR coverage. Aled Williams spotted this col back in 2016 where my original program had missed it!
You can't take the data as is, of course - the LIDAR needs looking at. For example the col given might be a bridge, or the summit rocks might have been missed. For the non-LIDAR summits all it tells you is that you might want to double-check the map. But the large number of recent changes to the DoBIH marked ‘‘JNSA’’ show that the data is proving to be quite useful.
At the end of 2017 the Environment Agency announced that they were embarking on a plan to cover all England, so when this data started to become available in May 2019 I wanted to revisit my program.
The idea was to fill in the gaps in the LIDAR with an alternative height map (DEM), which would allow it to identify all summits and cols with LIDAR. As an additional goal I wanted a high enough quality DEM for the gaps so that it might spot possible new summits outside of the LIDAR.
To test the usefulness of the possible DEMs to plug the gaps I looked at Fuar-tholl Mor on Maoile Lunndaidh/Creag Toll a' Choin (NH 127 460), a three ring contour top which is only shown on the 1:25,000 map. Contours on Open Cycle Map, which come from the Shuttle Radar Topography Mission (SRTM), look very good but Fuar-tholl Mor is completely missing (I downloaded the data from NASA to verify it was missing and not a casualty of a smoothing algorithm). Next I looked to the OS who provide a free product OS Terrain 50 Grid, which is a nationwide 50m grid of heights - but this is not only extremely low resolution, it also has major errors - it completely cuts off the summit of Garnedd Ugain/Crib-y-Ddysgl (SH 610 551)! The alternative OS Terrain 50 Contours, however, is very good and clearly shows Fuar-tholl Mor. It appears to be a slightly angular version of the contours on 1:25,000 maps, but occasionally some small contours are missing. This is a vector product, so I used Python scripts in QGIS to interpolate height data from the contours.
An unexpected hurdle was the apparently simple task of downloading the LIDAR, as there are multiple different sources and formats and download methods, some of them very convoluted, this is an irritating and long winded task. It's also quite large - I downloaded about 1 TB of data - so I had to splash out on an external hard drive.
The program takes the interpolated OS height data, overlays the LIDAR height data, then looks for summits and cols. It matches the summits in the DoBIH, and then outputs data for all 29,693 tops... enough to keep anyone busy! The difficulty was getting the algorithm to process data in O (n logn) time so that it didn't grind to a halt with the large data set, and to process the data in sections and then combine the results so that it didn't run out of memory.
To allow you to check borderline tops it lists tops with drop >=20m (reduced to 10m for land over 600m). Although most of the Nuttalls have no LIDAR, I was surprised that the OS contour interpolation was good enough that all of them are picked up (and most ex-Nuttalls) except for Harrop Pike (NY 500 077).
In addition it draws a height-shaded map of Great Britain, with LIDAR coverage shaded blue, marking all summits and cols and the ridge-routes between. This was originally just for debugging purposes, but I find the map fascinating.
The program passed my main test case - it needed to spot the col for Carnedd y Ddelw (summit at SH 708 705), right on the edge of the LIDAR coverage. Aled Williams spotted this col back in 2016 where my original program had missed it!
 |
| Zoomed in section, showing the summits and cols and ridges and the patchwork of LIDAR around Carnedd y Ddelw |
You can't take the data as is, of course - the LIDAR needs looking at. For example the col given might be a bridge, or the summit rocks might have been missed. For the non-LIDAR summits all it tells you is that you might want to double-check the map. But the large number of recent changes to the DoBIH marked ‘‘JNSA’’ show that the data is proving to be quite useful.
Link to dropbox files:
You'll need to download the files - the preview doesn't work
Cheers
Joe
Cheers
Joe
Fascinating, thanks for taking the time to explain the process.
ReplyDelete