Discussion: Processing digital imagery "point clouds"

Spatial Services, the government mapping agency in NSW, is progressively releasing state-wide topo data on-line for free download under a Creative Commons licence. The data sets are in 3 main forms - LIDAR point cloud ( typically 1m spacing, but some 2m), DEM as 1m, 2m and/or 5m mesh, and digital imagery "point cloud". They have indicated that LIDAR data collection will continue along the eastern coastal part of the state, but west of the main range most of the new data will be gathered by digital imagery at nominal 0.4m pixel size and available in laz format. Processing the DEM and LIDAR point cloud data for orienteering base maps is not an issue as the available tools handle these well (within the limits of the data density.)

So far the DEM data from the processed digital imagery has been presented as 5m mesh gridded data (ESRI format) in 2km x 2km tiles, and the digital imagery as unclassified las 1.2 or binary laz format "point cloud" tiles. The data sets usually have a common specification across a 1:100k map sheet, covering approx 2700 sq km. Within the map sheet area there may be areas suitable for orienteering maps, and I would like to be able to maximise and optimise the quality of the data in these potential map areas. Does anyone have any experience in handling data in this format and recommendations on suitable tools and processes?

There is at least 150000 sq km of digital imagery data already released in the central part of the State - included in this coverage are potentially more areas like Sappa Bulga, the granite complex used for the 2007 JWOC carnival. Even if the Spatial Services data is not high enough quality for the final base map of a complex area like Sappa Bulga it will be very useful for preliminary assessment and selection of map areas.

Can you provide a sample or link of the data?

As sample of the data with a recent high quality map for comparison:
(Hill End - Aus champs 2017 - map on Routegadget - one of the last major maps in Australia from old-style photogrammetry

Elevation data on-line for Australia (ELVIS = Elevation Information System):
http://elevation.fsdf.org.au/

instructions and hints on opening page

zoom in to area around Sydney and then go to Hill End 190km NW of Sydney, in the Orange 1:100k map sheet:

Hill End lat -33.0336 long 149.41449

using the "select an area" button on the right half of the screen

zoom to Selected bounds: eg 149.4097° west, -33.0313° north, 149.4178° east, -33.0362° south

available files listed"

"point cloud"
Orange201308-PHO3-C0-AHD_7246342_55_0002_0002.zip 252MB
- metadata is displayed if you click on the file name, and the extent of the tile displayed on the map pane
- tick the box to start the download process - an email will be sent to you with the download link - note size of single 2km x 2km file

5m DEM
Orange201308-PHO3-AHD_7246342_55_0002_0002_5m.zip 388kB

Point this at Victoria.

Invis,
the business model for open-file topo data, and actual implementation, has certainly changed in NSW over the last 2 years - I hope for your sake that Victoria gets its act together too.

I have also been working with similar data. From that site there are two sets of data available. A dtm in .asc format and a 'point cloud' in .las format. The 2 x 2 km tile point cloud I downloaded and imported into OCAD 11. It had close to 20 million points (all 'unclassified') and when I generated a slope gradient image it looked to me like a DSM - in open areas it showed the ground and in forested areas it showed the tree tops ( the area in question is mostly open). Looking at the slope gradient image in the open areas it looked pretty good quality - much better than contours I have seen generated from the what I presume was the 5m grid dtm (from the .asc file??)

Importing the .asc file into OCAD allowed me to generate a slope gradient image (much lower res than than the dsm described above) but for some reason I could not generate contours from that in OCAD 11. Then I calculated the difference between the the dsm (from the .las data) and the dtm (from the .asc) and was able to generate vegetation info from the result.

My feeling is that if we could separate the point cloud data into dsm and dtm (first and last return ) we could get quite a good base map with both contours and vegetation.

I followed the description to Red Hill and started the download of the las+zip, this is running on AWS with extremely low priority/bandwidth.

I am getting about 10-15 kB/s (on an 80 Mbit/s fiber connection) so I'm guessing this throttling is designed to discourage people from outside Australia?

The stated limit of max 50 GB/day is obviously impossible to reach, I would get about 5 GB if nothing failed during a 24 hour period.

Re. robplow's problems with the point cloud: This seems like totally raw data, without even ground classification, but my LAStools based pipeline should be able to get both contours and some vegetation data tomorrow.

Terje: I got 2MB/s to a US connection with only 2MB (16Mb)/s available right now (darned family), so could get the 50GB in a day—particularly when the kids go to sleep. That said: if the problem really is some kind of throttling, and you're doing it in AWS, perhaps you could move the work to the AWS Sydney datacenter?

I'm curious about your veg processing method(s) with no classifications. Sounds great! Is there a document somewhere, like another thread?

I did get the data downloaded, it really should have been stored as LAZ instead of ZIP'ed LAS which needs 2.5 times as much space!

It is quite nice in that each point carries RGB image data as well, but the density is somewhat less that I'd like, and the vegetation is so thick that it is hard to find good ground points below it.

Terje,
I'll pass your suggestion on about using laz format rather than zipped las on to Spatial Services - I noticed the better compression in the sample data set I sent you.
Do you know if laz format causes problems for any GIS system, or is it now effectively an industry standard?

As far as I know laz does not cause problems, not using it does.

People call LAZ "zipped LAS", but a lot of people assume it's just a zipped up LAS file, and it's not. It's a special format designed to compactly compress LAS data. ESRI muddied the waters further by introducing a proprietary format zLAS, which they say is better for streaming lidar data, but seems intended to lock up data for ESRI products, and impose a hassle on non-ESRI software and users.

If your software needs LAS or text format, it's easy to make with LASTools (even the free version).

I find LAZ is about 1/7 the size of LAS, which is huge when you consider download times and hard drive storage issues. I'm in the market for a 2-4 TB external drive right now.

As both Jagge and cedarcreek notes, LAZ is an effectively fully open and compatible compressed format for LAS data. If you also need faster spatial queries, then LAZ has a companion LAX format which provides indexing for LAZ data, without locking everything up in ESRI's blatant attempt to "embrace & extend" an open standard in order to make it proprietary again.

I have completely stopped all storage of LAS files, when I get some I immediately convert to LAZ and delete the originals.

For my LAStools processing I also use LAY (LAS Layers) files to hold temporary data as I'm adding it, i.e. classifications, heights above the ground etc.