Register | Login
Attackpoint - performance and training tools for orienteering athletes

Discussion: Lidar mapping batch process

in: Orienteering; General

Apr 7, 2017 3:04 PM # 
Rob H:
Has anyone had success recently with the lastools-based batch system put together by Terje Mathison?
Described here: http://tmsw.no/mapping/basemap_generation.html

A few years old now. I am trying to evaluate it against other approaches (such as Karttapullautin) as an aid in basemap production for a couple of current projects. It is partially working, but fails to generate the classified tiles. The end of the process produces the following:

Classified tiles generated, ready to generate contours, slope/cliff/dem and vegetation
Usage: gen-dxf-iso.pl lazfile|file_with_list_of_laz_files [output_file]
Usage: gen-slope-asc.pl lazfile|file_with_list_of_laz_files [output_file]
Could Not Find C:\data\orientering\tiles_classified\tile*.asc
Usage: gen-laz-veg.pl lazfile|file_with_list_of_laz_files output_file
Usage: veg2dxf.pl giffile|file_with_list_of_gif_files
Could Not Find C:\data\orientering\tiles_classified\tile*.gif
Could Not Find C:\data\orientering\tiles_classified\tile*.gfw

I have tried with a few different laz files from different areas and projects with the same result. It says the classified tiles were generated, but the output folder is empty. Any insights appreciated. cheers,

Rob
ONB
Advertisement  
Apr 9, 2017 8:32 AM # 
TheInvisibleLog:
Interested here as well. So commenting to keep it in the current discussion list.
Apr 9, 2017 12:58 PM # 
Terje Mathisen:
I have been reworking this process significantly, mostly in order to handle new higher density lidar scans: The new adaptive process use 256x256 tiles (with a 32 m buffer), but then splits those tiles into 128x128 or even 64x64 when the original scans are very high density.

In your case I'm guessing this is a PATH issue, i.e. in addition to the LAStools binaries you must also make sure that all the .pl files are either in the local directory, or somewhere in the PATH, and you must also check that Windows is setup to treat .pl files as executable.
Apr 9, 2017 2:25 PM # 
Rob H:
Thanks Terje. I was assuming you had probably worked further on this system since 2013.
I had all the files, scripts etc in the same directory, so I don't think it's that. I will check the .pl executable issue.

Right now we still have 30-40 cm snow in the woods here, but in a week or 2 it will be mapping season again...

Rob
Apr 9, 2017 3:17 PM # 
Jagge:
Maybe changes in lastools command line opting is making it fail. At least clip option changed at one point and made KP fail before I made it try the other commas too if first try failed.
Apr 10, 2017 3:22 PM # 
Rob H:
Thanks for the feedback ideas. Not sure what to try next....I am running Windows 7, the pearl scripts are all correctly associated on the system as far as I can see. I terms of possible PATH issues, I added .pl to the path environment variable, but this made no difference. The batch process 'ran' (as before), but produced no classified tiles. I was using a single 1 km2 laz tile for this test.
Apr 10, 2017 4:05 PM # 
Terje Mathisen:
A single LAZ tile?

My batch process used to depend on first having lastile split the input area into multiple (in both x & y direction) tiles, but since the default was 250x250 m that should have been OK.

Do you get a tiles_raw directory with a bunch og tile_xxx_yyy.laz files?

Ditto for tiles_classified?
Apr 10, 2017 4:32 PM # 
JimBaker:
Is the link above the correct place to download the latest tool set?
Apr 10, 2017 7:23 PM # 
Terje Mathisen:
@JimBaker: Yes, I just checked: The zip posted there is from Nov 15th 2016, it should contain a consistent set of files from my previous pipeline.

The new version is not complete, most crucially it is completely missing a ground point search, i.e. it depends on having already classified input files like we're currently getting in Norway.
Apr 11, 2017 1:23 AM # 
Rob H:
Terje,
I get the tiles_raw directory with 250x250m tiles. They all seem fine when opened and examined etc.
The classified tiles directory is created, but is empty. No files.

Tried this with laz files from several areas and sources and always the same result.
Apr 11, 2017 1:08 PM # 
Terje Mathisen:
Rob H: If you will setup something like TeamViewer on your PC and text me the link and a suggested time (From now until 22:00 Oslo time) I can try to connect to your machine and then we can figure out together why it isn't working for you.

My cell phone is +47-91728852, or you can email me at terje.mathisen (at) tmsw.no
Apr 11, 2017 3:07 PM # 
Rob H:
Thanks I appreciate the offer. I will be in touch. Assuming we figure it out, I/we can post the learnings back here.
Apr 11, 2017 6:38 PM # 
Terje Mathisen:
Rob & I were able to connect and we found the problem in ~10 minutes:

It turned out that when Rob had downloaded the lastools.zip package a year ago either his download was broken or he accidentally omitted one of the critical binaries from that package when he unpacked it, i.e. las2txt.exe.

Both my batch pipeline and Kartapullautin uses las2txt to convert a binary las/laz file into text so it becomes easy to parse, and in Rob's case with the program missing he never got any output data.

I guess I should invest in a little up front checking in my script to make sure that all the critical parts are available!
Apr 12, 2017 12:59 AM # 
Rob H:
Big thanks to Terje for helping troubleshoot this. So - the learning is to check/refresh your lastools package when trying this batch process. It's a cool system... many output files to manage, so if you have a big project this may become challenging? But I especially like the fact that the output vegetation data is in vector form, and so easily manipulated in whatever mapping package you prefer.
Apr 13, 2017 5:35 AM # 
Jagge:
Maybe Rob H wjen trying karttapullautin moved las2txt.exe to an other folder (insturctions sugest that, even if it s not necessary, it only needs to be in path). And that's why it was missing.
Apr 13, 2017 7:49 PM # 
jtorranc:
But I especially like the fact that the output vegetation data is in vector form, and so easily manipulated in whatever mapping package you prefer.

Is it now? That's a new feature since I last updated my copy of Terje's package plus Lastools. I'll have to go grab the latest version for next time I make a basemap. Maybe by then ground point detection will be working for the version he's now working on.

BTW, Terje, I used your tools to make the basemaps for both areas that will be used in the QOC-hosted US Classic Champs in November. Also a new map that I fieldchecked last summer in Gatineau Park, north of Ottawa.
Apr 16, 2017 2:55 PM # 
Terje Mathisen:
@jtorranc: Great to hear that you're getting some use out of my sw, after the race I'd appreciate a (link to) a copy of the finished map!

I wrote my raster-to-vector vegetation conversion sw quite early, but the first versions were not stable enough/required too much "hand-holding" for me to include any of them in the mapping.zip package.

The current version is still not quite general, in particular it depends on knowing that the initial raster vegetation bitmap will always use a 2x2m cell size.

To get from raster to vector I first process the image and create individual 2m long line segments along all cell boundaries with different colors on each side, with a virtual white boundary around the current tile.

This list of segments is indexed twice, by each of the end points.

Next I join those 2m horizontal & vertical line segments together into polylines, but only for junctions where exactly two segments meet, so at the end I have a bunch of polylines and a set of junction points where 3 or 4 of the lines meet.

I.e. foreach (list of (x,y) junction coordinates) do
if (nr of lines meeting here == 2) then
remove both lines from the index
merge them into a single polyline
stuff the endpoints of that polyline back into the index
endif
enddo
Repeat until there are no more junctions to merge.

At this stage I also record the color (vegetation type) used for each patch.

The next step is to low-pass filter each polyline while keeping the endpoints fixed, i.e. I round off all 90 degree corners so that for instance a stair-step pattern will turn into a 45 degree slope.

Finally I go through the list of vegetation patches and determine which polylines enclose each patch so that I can output a single area object for each of them. At this stage I must also determine if I have any holes or islands, i.e. a green patch in the middle of an open yellow area, and if so, record this as a hole in the surrounding vegetation type. (BTW, this last part required several rounds of back & forth with the OCAD developers in order to come up with an encoding which follows the DXF standard and which OCAD can understand.)
Apr 16, 2017 4:19 PM # 
JimBaker:
What about compatibility with Open Orienteering Mapper, which I suspect will become the predominate O mapping tool (due to a number of advantages)?
Apr 16, 2017 9:02 PM # 
Terje Mathisen:
That already works on the development version of OOM, I'm advocating giving OOM + open lidar and either local community vector data or OpenStreetmap for the topo stuff in order to have a totally free mapping solution.
Apr 16, 2017 10:26 PM # 
gudeso:
Off topic, did they solve the problems of importing shp files?
Apr 17, 2017 7:01 AM # 
Terje Mathisen:
The GDAL module which is what allows dxf area and areas with holes to be imported should also handle Shape.
Apr 20, 2017 8:09 PM # 
jtorranc:
@Terje - I'll have to find a link for the map in Gatineau Park, which was used last fall. Not sure there's RouteGadget but maybe it's in someone's DOMA. In the meantime, I neglected to mention I also used your software to make a basemap of a state park here in MD, which we just held an event on without doing any field checking beyond what I recorded with my GPS during two visits to the park to design the courses - this was practical since the map is lousy with charcoal burning platforms, which show up beautifully in the slope raster your software produces. You can see the map, at least the portion used this past Sunday (there's a lot more of the park, mostly to the west and north) at http://qocweb.org/routes/cgi-bin/reitti.pl?act=map...
Apr 20, 2017 10:04 PM # 
MChub:
@jtorranc : This one, I suppose?
Apr 21, 2017 12:10 AM # 
JimBaker:
I've started wondering if the near future of much O mapping is lidar plus GPS trails, no other field work. Were the boulders captured using lidar? How accurate and complete were they? (I've noticed the cliffs on my list basemap are quite complete.)
Apr 21, 2017 12:50 AM # 
jjcote:
The map that jtorranc made worked without fieldwork largely because of the nature of the vegetation, which was a mix of incredibly open hardwood forest with almost unlimited visibility, and well-bounded mountain laurel, which showed up well enough to be pretty accurate on the map. (There was also greenbriar in spots, which wasn't mapped, and it would have been a bit better if it was, but the courses generally stayed away from it.) And then there was rock all over the place which you pretty much ignored.
Apr 22, 2017 10:19 PM # 
JimBaker:
It still could prove useful to create navigation sport maps without thousands ‚Äčof dollars or hundreds of hours of investment. If I know what a map includes and what it doesn't, I can generally adapt. Variety in terrain is useful. Our current cost model makes that hard.
Apr 23, 2017 2:10 AM # 
jjcote:
Oh, definitely useful. It wasn't good enough for a championship, but it was as good as a lot of local meet maps created by volunteers that I've run on. It probably wouldn't work well in some terrain, but in a lot of places it would be just fine, I think.
Apr 23, 2017 2:30 AM # 
JimBaker:
I'm amazed at how good the contours, cliffs and vegetation (yellow and light green, but not vertical light green) are for the area I'm mapping using a lidar base. Very little change needed; no change would be adequate. Trails and power line were easy to put on by GPS. It's the boulders that are proving time consuming. If I just punted those, the club would have a nearly instant map. I'll probably soldier on, but the effort for a feature that's mostly too common to be terribly useful is making me think whether I should just map most of the boulder areas as boulder field, and get a map out with far less effort. Given how much free lidar Colorado has, I wonder how many nearly instant maps it could have. I'd be happy to orienteer in a new area even if the boulders were missing.
Apr 23, 2017 3:46 AM # 
Terje Mathisen:
@JimBaker: If you have access to high-density lidar you might be able to locate many of those boulders as small dot knolls on the base map my sw generates, I have seen that previously (from Colorado), as well as boulder fields that turn up as green stripes because the lidar looks like low brush:

A glance at orto photos might suffice to convert a bunch of these to the proper object type.
Apr 23, 2017 11:56 AM # 
JimBaker:
I'll try running your tools. I recall that you processed the lidar for my current project, and it showed a few boulders (out of hundreds). Aerial photos do show more of the rock, but not how high of course, and thus show a mix of mappable and not.
Apr 23, 2017 12:34 PM # 
Terje Mathisen:
The main problem is that boulders normally have exactly the same lidar footprint as a small round bush, and such a bush is almost certainly too insignificant to show on the map. :-(

Very large boulders however can show up as ground points which I then convert to a dot knoll since the area is too small for a form line contour.

I have considered doing multiple lasground_new runs, with hand-tweaked parameters to avoid interpolating the ground underneath boulders, but this will also lead to all dense bushes turn up at the same time. It is possible that lidar intensity would be able to differentiate between a naked rock surface and leaves, but not if the rock is (partly) moss-covered.
Apr 23, 2017 12:49 PM # 
JimBaker:
Do bushes not have a sufficient subsequent reflection beneath them? I thought that vegetation was distinguished by multiple reflections, the last being the ground? Are bushes that dense? (In much Colorado terrain, there are few dense bushes more than knee high except in swamps, though there can be deadfall or logging, so perhaps different heuristics would work?)
Apr 23, 2017 5:45 PM # 
Terje Mathisen:
Small bushes (maybe somewhat species-dependent?) try to keep all the leaves on the surface, so with maybe 2 pulses per sq meter you only get a couple of hits, and seldom any secondary returns.

Get up to 5 (or 10-20) ppm and you can start to delineate the actual rock surface, while even very dense vegetation will tend to get a few secondary returns and/or develop a somewhat fuzzy surface.
Apr 23, 2017 7:16 PM # 
JimBaker:
My data has ten per square meter, so maybe something could be done? There's an enormous amount of Colorado lidar with seven to ten per sq m, so this could help us enormously.
Apr 23, 2017 7:26 PM # 
jjcote:
Assume it's all rock and no bushes. The fieldwork would be a lot easier if you were just deleting bushes than if you had to locate and add rock. (For that terrain, with few bushes.)
Apr 23, 2017 8:48 PM # 
Terje Mathisen:
Jim, do you have a link to a typical sample of the ~10 p lidar? I can try to see if a manual tweak of the lasground_new parameters make it capable of locating these boulders as very small but steep dot knolls.
Apr 23, 2017 10:55 PM # 
JimBaker:
north central ridge sample

Data at
http://opentopo.sdsc.edu/result?id=pc1492987682697

Metadata at
http://opentopo.sdsc.edu/result?id=pc1492987682697...

southern ridge sample

Data at
http://opentopo.sdsc.edu/result?id=pc1492987499874

Metadata at
http://opentopo.sdsc.edu/result?id=pc1492987499874...
Apr 24, 2017 1:11 AM # 
cedarcreek:
NVDI on 4-band aerials also might help find big boulders.
Apr 24, 2017 1:45 AM # 
Terje Mathisen:
I've checked the first area with default settings, I can see the boulders easily on Google Earth, and at least some of them turn up as vegetation points in the 10 points per sq m dense lidar.

For this particular type of open forest I would probably just use orto photos to identify the exact placement of each boulder, after first jogging through the area with a GPS watch and mark each interesting block with a split time.
Apr 24, 2017 8:48 AM # 
Jagge:
Boulder are patches without ground classified points but instead last returns 0~6m above ground. So if you plot out those you get pretty good guestimate layer for boulder mapping. And with trying to

For instant orienteering run I sometimes with Karttapullautin plot those last returns with buffer and then override them with ground points with bluffer and draw remaining as purple with black outline.

http://routegadget.net//kartta/?zoom=15&mapurl=htt...

With appropriate buffers based on cloud density (and detected boulder size expectations) it result isn't that bad if one is prepared to see some bushes mapped with purple. Not sure does it make any sense here with the settings I used here.

Please login to add a message.