in: eddie; eddie > 2005-09-18;
| # Posted 2005-09-19 23:19:53 | |
| Spike: | What is "Lidar"?
And congrats on the win. |
| Advertisement | |
| # Posted 2005-09-20 06:58:50 | |
| eddie: | Thanks!
There's a pretty good explanation of lidar for elevation mapping here. Lidar is used for other things too like aerosol concentration mapping and upper atmosphere wind measurement, but the concept here is just like those laser range-finding thingys. Its essentially radar but with light wavelengths rather than radio wavelengths. I started playing with this stuff a couple of years ago back home in Louisiana. The army corps of engineers had begun a big lidar elevation survey of all of south Louisiana for flood management. My dad's property was in one of the first areas to be mapped. Here is that lidar analysis I did of some family property in Louisiana: Chenal lidar data This is all extremely low relief stuff. In some of the images I have colored the trees green and the water blue, so that should give you a reference for the height differences. In the colored ones the bare earth heigh differences are a max of ~2m with details at about the 0.2m level easily visible. Areas with moire pattern are generally sugar cane field stubble in rows. The image coupee_dem_image.jpg shows the USGS 30m posting DEM data, which you can compare to chenal_lidar_image.jpg which is a dumbed down to 5m posting version of the lidar data. I think the raw lidar was 1m postings. Note all the striations in the contour features inside the arc of the old Mississipi river oxbow. There is a lake in this oxbov called "False River" Our property is a long skinny section leading from Bayou Chenal (old Miss river bottom) at a sandbar which is now an island in the bayou, towards the center of the oxbow. The low rises and drops of the old riverbeds and mudbars inside the oxbow can be seen as VERY subtle rises and drops as you go towards the back. By subtle I mean a couple of feet. The changes in veg are more obvious as it goes from regular hardwoods on the "high" ridges to cypress swamp in the "low" areas. Also in the coupee DEM image you can see the mississipi river levees along the shore. These are typically ~7-10m high. Here is that lidar data of the earthworks at Port Hudson. Port Hudson lidar This is the Port Hudson civil war battlefield, which is across the river in some high ground that you can just see in the upper right of coupee_dem_image.jpg. Its all forested now, but in the bare-earth lidar image you can actually see the old earthworks which are under 1m high after all this time. Lidar data like this might be a very cool way to find old earthworks and foundations - an archeological tool. This would be a great place for an O-map! Very intricate reentrant structure and a nice meandering stream at the bottom. It would only be useable in winter though - would be too thick and hot in the summer. Here are a couple of rendered images of the Mckeldin lidar data. Ted and I spent some time trying to figure out why this clearing didn't match the O-map we were using. The lidar places it nicely. mckeldin_render1.gif mckeldin_render2.gif The first image is looking N. The second is facing SW and the elevation has been vertically exaggerated by a factor of 2. The red cross marks the spot where Ted and I were standing scratching our heads for a while :) These make it look like an alpine meadow high in the Rockies, but its just an abandoned, overgrown farm field in central Maryland. These blocks are 600m on a side. Here is a piece of the lidar-derrived basemap in progress. The uper right purple square is the area in the rendered blocks. One thing about the lidar data that suprised me was the ability to see very indistict trails. The shallow depression of the trail surface below the surrounding forest floor makes them really stand out in a gradient image of the lidar data. Keep in mind that all of this is seen through the vegetation cover (probably leaf-off). And you get both vegetation height and density, as well as the bare earth elevation - everything already rectified and georeferenced - all in one pass. |
| # Posted 2005-09-20 19:12:05 | |
| jtorranc: | My two cents - I think this topic ought to graduate to the general discussion list sometime soon. Though I can understand a desire to delay publication of a work in progress, this is way cool and the potential utility to orienteers everywhere is huge. |
| # Posted 2005-09-20 19:20:39 | |
| feet: | Yep, I agree: repost on the main discussion board. Do you have a sense of how broadly-available lidar data is? We've been using post-2000 vintage MassGIS 1m aerial photo data for CSU park-o mapping, but it has no elevation data incorporated so we're still on USGS-old-school-style data for that, so this looks very useful if it's available for MA. |
| # Posted 2005-09-20 19:31:34 | |
| eddie: | I've been delaying and delaying because I had ambitions of writing a "white paper" on the subject, but that is looking less and less likely. I've collected quite a bit of field info now, including a bunch of photos of features at Oregon Ridge and comparrison of 2m vs 1m posting lidar data.
My general impression is that the contours from both 2m and 1m are adequate for basemap. Fine detail info (ditches, pits, streams, trails) are very, very good with 1m data, but in the 2m data its a little questionable. I believe the problem lies in the fact that many O features are in the 1-2m size range, which is Nyquist at 1m but undersampled at 2m. So I'd recommend getting 1m or better posting data if possible, but 2m posting is adequate. 5m and higher is probably useless for O basemaps. I've looked at the USGS 10m posting DEMs of the 7.5 minute quads and they contain about the same level of information as the quads themselves. 5m would be twice that resolution but still largely inadequate. Its possible that data in the 3-5m range might be useable to some degree. Certainly if it was a choice between that and a USGS topo it would be better than the topo. One of the best things, at least for us living in a coastal watershed, is that many goverment agencies are starting to collect this 1-2m res data and we are able to get it for free. The 2m data of Mckeldin is part of a county-wide Chesapeake watershed survey. The 1m Oregon Ridge data we got was a special case of a hydrology study of the stream that runs through that park by a local university (UMBC). They graciously provided the data to us under a standard "non-commercial use" basis. The Louisiana data was a Corps of Engineers project and was available via free download from LSU. Many aerial survey companies will now fly contracted areas just like they would for aerial photos. Greg Lennon (QOC) has been doing a great job of ringing up these guys and securing the data for us - a chore I am very bad at and hate doing myself. In just a couple of months I'm already starting to drown under the ammount of data and projects we have ready to go. We are still waiting on the Baltimore County 2m data, which I am very anxious to get because I want to map more of the Loch Raven watershed near my house. Francis wants to do more work on the Walker's O-map farther south along the shore as well. Best of all you can combine a lower-res rectified orthophoto from say the NAPP or some other source (even Google maps) with the lidar data to add other info to the base. While these photos are at too large a scale for steroplotter derived O-contours, they still have enough res for adding features to the base (if you are careful about it - mind the projection/slope effects and shadowing). So far though, there seems to be plenty of info in the lidar data by itself. |
| # Posted 2005-09-20 21:56:56 | |
| jtorranc: | A further comment on that last paragraph - I can't see any reason why it would have to be a lower-res rectified orthophoto. The 1 meter resolution colour infrared DOQQ we got from USGS or GIS Data Depot or wherever was invaluable in mapping vegetation and other features when we made our map of the University of Maryland College Park Campus. Not that the colour infrared photography is available everywhere in the US but even the plain colour and the black and white DOQQs contain a lot of useful information and some kind of DOQQ has been available everywhere I've ever looked for it (admittedly this means just MD and VA). |
| # Posted 2005-09-20 22:05:52 | |
| eddie: | Yeah, those are the NAPP photos. The entire US gets covered every 7 years or so. I believe those are 1:40000 scale film photos, scanned at 1m/pix, then rectified using the 10m or 30m DEM data. I think the rule of thumb for sterophoto O basemaps is photos no more than 0.5x the scale of the final O-map. (i.e. 1:20000 photos for a 1:10000 O-map).
One of my goals is to see if having precise elevations every 1m is equivalent to having 1m resolution stereophotos taken at say 1:20000 scale, then pushed through a manual (or computer) stereoplotter for making contours (noting that resolution is not the same as scale, and scan/image resolution is not necessarily the same as the true optical resolution of an imaging system). And if 1m lidar turns out to be not good enough, what do we really need for O maps? Its likely that it will come down to getting two samples of the smallest features you'd like to see on a basemap. If you want to resolve two 1m sized objects right next to each other you need samples every 0.5m, but you don't necessarily need to resolve features that small to make a basemap for a 5m, 1:10000 O map. The 1m lidar is looking good so far. Eventually will need a real professional to verify things, as I am most definately not a skilled fieldchecker. |
| # Posted 2005-09-20 23:45:49 | |
| eddie: | It may be interesting to note that there are several of these lidar instruments in space. In particular, there is one aboard the Mars Global Surveyor spacecraft currently orbiting Mars. The MOLA (Mars Orbiter Laser Altimeter) is the instrument. There are some nice global elevation maps of Mars made from orbit. Data was sampled at 300m horizontal, 37 cm vertical res. with a 130m spot diameter at the surface. The entire northern polar region is a lowland, leading to speculation that there may have been an ocean there. This is further reinforced through the use of a rainbow color table which uses blues for low and reds for high. :)
In case you were wondering, here's a picture of the nerds who built it. |
| # Posted 2005-09-21 00:00:47 | |
| glennon: | Admittedly our LIDAR work has all focused on terrains in Maryland and Virginia so far, but if that MOLA group can just increase their spot density in the future, we could start thinking about the first extra-planetary orienteering basemap, eh? ;-)
|
| # Posted 2005-09-21 00:04:31 | |
| feet: | What's the symbol for 'the void of interplanetary space'? 'Open land' doesn't quite seem to cover it. |
| # Posted 2005-09-21 00:14:13 | |
| eddie: | :) I've thought about doing a photo-O on mars. There are lots of nice pics on the rover pages. Thes machines are truly amazing. They have now been operating for a year longer than their design life. Spirit is now sitting on top of a hill nearly 4km away from its landing site and still going strong.
Also Cassini in orbit around Saturn has a radar which is gathering some nice data on the moon Titan. So far its all intensity/scatter info, but I presume they are getting elevations from the radar data as well. And of course there were the magellan radar altimetry maps of Venus, back in the day. |
| # Posted 2005-09-21 20:44:51 | |
| JDW: | Take a look at this presentation from the recent International Conference on Orienteering Mapping in Japan.
|
| # Posted 2005-09-21 21:19:26 | |
| eddie: | I added info from the vegetation-height part of the lidar data to the base last night in the 4 sections already done. New version replaces the old one posted here.
Also added some of the fieldchecking edits we made Sunday. *almost* everything on here is derived directly from the lidar data. Some things were so subtle I couldn't add them without prior knowledge from field-checking that they were actually there. The only place I really did this was the trail in the S reentrant and some of the rock features. All of the clearings were obvious in the lidar. The two fenclines were added based on the difference in tree height which made a sharp line there. Ted and I had found pieces of wire along those lines, then going back to the lidar I was able to place those lines pretty accurately. Ted can verify them based on the bearings he took Sunday. I also used the purple "marked route" symbol to indicate very shallow lines I could see in the unsharp-masked lidar that are too shallow to be mapped as ditches (as verified on our field trip Sunday), but still may be useful to the mapper. The purple makes them stand out and easy to "delete" later. In some cases these are simply the lowest point in a reentrant bottom. The big clearing in the upper right of the base is the one in the mckeldin_render1.gif image. Note the two single trees I added last night. You can see all of those little shrubs in the lidar data, but those two stood out so they got added. It looks like they are the tallest individuals in the clearing. Will have to check to see if the others are worth adding. We were in this clearing Sunday and you can definately see all these little shrubs. |
| # Posted 2005-09-21 21:56:12 | |
| jtorranc: | I presume based on "some of the fieldchecking edits" that there is more trail network to add. There certainly seem to be more dead ends/sections of trails going from nowhere to nowhere than I would expect. I'll look forward to seeing what a 'final' version with everything that can be located based on the LIDAR data rather than by conventional field checking in relation to features in the LIDAR data looks like. |
| # Posted 2005-09-21 22:04:03 | |
| eddie: | Yeah, the reason for the incomplete trail sections is that I only add what I can see in the lidar data. In the case of that trail in the south, I had some knowledge of where the trail actually went in-between segments and could then use the subtle scratch in the lidar data to add what would otherwise be too weak to use ffrom the data only. Some of that trail I had put in as "ditch" in the original basemap. The difference between ditch and trail in the lidar is often difficult to tell. If its raised like a roadbed, then obviously its not a ditch :) |
| # Posted 2005-09-21 23:00:25 | |
| eddie: | Thanks for the .ppt John. This is basically what I had in mind to do for a white paper, but with a few more "how-to" details. He has the same comparrisons with photos in terrain. Also it was nice to see he draws the same conclusions: 2.5m posting not quite good enough, 1m posting VG. His pro and con lists also make sense based on what I've seen.
He's using what he calls "hillside shading" to increase the contrast of subtle but sharp features like ditches and roads. Essentially its a surface rendering with oblique lighting. I've used two other image types for that: image gradient (an image of "steepness") unsharp mask (high-pass filtering) All of the above are methods of increasing the contrast of subtle but sharp features in the elevation data. I've found that the unsharp gives the best contrast. The gradient suffers from the fact that its directional (some combination of x and y gradients), so features that are sharp *along* the gradient vector are invisible. There are probably mathematical versions of a spatial gradient that deal with this better. The unsharp uses a 2d spatial boxcar so in a sense is omnidirection. Both work though. So I make the contours directly from the bare-earth lidar elevations at 5m (2.5m or even 1m is better for a base because its easier to see the subtle things, but for Mckeldin we are using this for a meet this weekend so I've been producing only the 5m version). Then I use two templates: The first is the unsharp mask. The second is the vegetation height (which is the lidar first-return minus the last-return (really the filtered bare-earth data). I'm also toying with other products - masks of spatially averaged vegetation height (which is how I produced the 2-color renderd images). Same with veg point density. I was originally thinking of using automatic edge-finders for this stuff, but I think in the end that involves more hand-edits to create a useabble base than simply using the templates by hand to begin with. The gradient image might also be useful to ID cliffs - can just do a threshold above some slope. I write the contour vectors directly to ocad using an ocad5 writer I wrote in IDL. The reason its 5 is because I wrote it several years ago. The newer ocad formats are more flexible but its real work to make a new writer. Could also write to DXF and then import. Anyways, ocad6 will read the ocad5 and then save to convert it. Presently the contours are x,y pair vectors. I'm working on code to make the conversion from vectors to bezier curves to save space in the ocad file and make the contours smoother and easier to edit in the final product. Another area I'm working on is the thing he calls "contour dithering" in the .ppt presentation. When you get into a flat area, pixel-to-pixel noise or real surface (forest floor) roughness can cause a contour to jigggle around individual pixels or even make small loops and funny shapes near the main contour level. You can defeat some of this with spatial smoothing before making the contours, but then you lose sharpness in other areas. There are other ways around this. Can for example vary the smoothing kernel based on the slope or original lidar sampling density (which varies across the dataset). |
| # Posted 2005-09-22 19:46:17 | |
| eddie: | New version of the lidar-derived basemap has replaced the older ones linked in this thread (reload if you've already looked at an earlier version). I finished two more 600m square blocks on the right. These two right-hand purple square sections are entirely lidar derived - no fieldchecking or other sources of information so far. Also I changed the symbol sizes from 15000 to 10000 so Ted can easily include this in the map for this weekend. I haven't had a chance to adjust things for overlapping clarity due to the resize yet. Later today I'll post a figure with the unsharp and veg templates side-by-side with the base at the same scale. |
| # Posted 2005-09-22 23:15:00 | |
| Spike: | Eddie, could you give us a link to the USGS topo of the same area. It'd be interesting to compare the two. |
| # Posted 2005-09-23 00:00:57 | |
| eddie: | USGS Topo
Google maps image |
| # Posted 2005-09-23 02:13:26 | |
| eddie: | ok, I've posted a comparison figure showing the 3 lidar component "products" and a section of the basemap. I've rotated it 90 degrees to make scrolling a little more natural, so N is right and W is up. The panels are as follows:
1) Unsharp masked version of bare-earth lidar return 2) my OCAD basemap 3) Vegetation height version of lidar data ("first-return") 4) raw bare-earth lidar image - contours are made from this The bottom panel is stretched from 50m(black) to 150m (white). All images here are 1.2m per pix. The originals are 1m/pix but they are reduced slightly in this figure. The lidar data itself was on 2m postings. So 1) is a filtered version of 4), but 3) is a different return in the lidar data (above ground returns). Can you find the things I missed in the top panel? :) There are a couple of strong features that I missed, and some more subtle features that I left out on purpose but maybe should have put in to be consistent across the basemap. Its amazing how sensitive this is to vertical variation (15cm vert resolution). For example, there is a faint raised (white) streak going diagonally across the upper right - crossing a reentrant that ends sharply where I have placed a "spring" symbol. I put some segments of indistinct trail along that line. There is a similar white streak near the center of the image going more vertical on which I have not put a trail, but is equally strong. I should probably put trail on both or neither. The reason I'm a little leary about this one is I actually took a look at a feature like this on another part of the map last weekend. The feature was there - a very old roadbed - but if I hadn't known it was there ahead of time I never would have seen it in the forest. It was *extremely* indistinct. A decision about a feature like this should be left to the fieldchecker, so everything I think might be real in the data should go on my basemap. You'll notice some purple dashed lines. I use this whenever I think a feature is probably too shallow to bother with or isn't a feature at all, like say the bottom of a reentrant. But sometimes knowing where the bottom of a reentrant is on the map could be useful to the fieldchecker. Having it in the basemap OCAD file like this makes it easy to "hide" these subtle things if you don't want them, or also to "turn them on" in a final OCAD map to have them placed accurately if they turn out to be useable features. Having a smaller contour interval on the basemap would help with this too as these features would then show up better. But the basemap here is being used directly for a race this weekend, thus the 5m interval :) No peeking! There's another subtle thing you can see in the top panel that is of note. If you look very closely you'll see lots of faint diagonal stripes from upper left to lower right. These are caused by a small vertical offset between data in adjacent flight tracks. The scanned data in adjacent tracks overlaps, just like the 60% sidelap in stereophoto flight tracks. Think of it as a constant error in the GPS measurement for one track relative to the other. When the data is georeferenced, these overlapping areas are interleaved and you end up with one scan low, one scan high, one scan low, etc. which causes the stripes. The data I have are sorted by UTM easting. In principle if you know which data are from which track you can correct this vertical error and get rid of the faint stripes. I want to try this soon, as this seems to set the noise floor in all the datasets I have in-hand. But of course its already well below the threshold for features on an O-map so it doesn't matter much. There is one segment of indistinct trail on my basemap which follows one of these by accident :) Can you find it? So yeah, it can lead to errors if the basemapper stays up too late. |
| # Posted 2005-09-23 23:02:06 | |
| jtorranc: | I wish I was going to be at McKeldin on Sunday to see the new area myself.
When you have a moment, assuming the data itself can be obtained free, how much in the way of time (computer and human) is it taking you to produce a section of basemap? Also, what, if any, costs for software licenses were involved? |
| # Posted 2005-09-25 00:41:31 | |
| eddie: | For this particular base, its taking me about 5.5 hrs/km ^2 of human time to add the by-hand parts (the unsharp mask and veg templates).
The contours are generated by code, and it takes something like 2 mins to: Read the x,y,z ascii data file, grid this into an image, triangulate, smooth, contour. The area I'm wking with is about 7.5 km^2. It then takes another 1 min to convert the x,y contour vectors into an ocad5 file. It takes about 10s to make the unsharp-masked version, chop it into bite-sized pieces (in this case, 600m x 600m squares), and write them out as bmp files. Another 1 min to read, convert, chop and write the first returns (veg) bmp templates. So in total, less than 5 mins of compute time (Mac G5, 1.2 Ghz), most of that being file I/O. I'd have to add to that something like 30 mins just for "optimization" - chosing the desired area from a larger lidar dataset, picking a smoothing kernel and otehr logistics. So figure a total of 30-60 mins compute overhead for a given project. This is tiny compared to the by-hand work of course. If the dataet was 1m posting rather than 2m it could take more time since there might be more visible detail to add. On the other hand it will be easier to see the weak detail than in the 2m data, so this is probably a wash. Plus any benefit will pay off in time saved for the fieldchecker I would guess. So based on this, the time formula for a project is: T = 1.0 + (5.5 * area) where area is in km^2 and T is in hours. The data we've been using so far is free. The software I'm using (IDL) is something I use at work. Its a general purpose image/array processing software. The routines I'm using to work with this data are pretty general and could be easily written in C or something else, and I'm sure there are free packages out there that already do all of these things. Also I read somewhere that its possible to make compiled apps from IDL source that can be run without an IDL license, but I've never done that before - might be worth a try. IDL itself is rather expensive. I don't own a license myself. Its really good stuff though. Every image on this thread was made using IDL (except for the OCAD of course). |
| # Posted 2005-09-25 09:36:47 | |
| eddie: | I made some figures showing how the lidar data is taken along the flight tracks and then stored as x,y,z triplets for each lidar return. I had thought the data was sorted by UTM coord, but it appears that within each file block it is stored in the order that it was recorded, making it somewhat easier to extract as a function of flight track. This should allow for the removal of the "cornrows" as discussed above. Anyways, here are three figures:
fight_tracks.gif fight_tracks_zoom.gif fight_tracks_trace.gif The first shows one file block worth of data (these are the files we got from MD DNR - I combined 4 adjacent files to get our coverage for this basemap. This is one of those 4). The three colors represent the three overlapping flight tracks. Green arrows indicate the direction of the aircraft's flight. Each "dot" on these plots represents a single lidar return (reflection). The swaths appear to be about 1km wide. The laser was scanned side to size as the plane moved forward, making a zig-zag pattern on the ground. Any small changes in the plane's direction or tilt of the wings results in the wavy edges of the lidar swath. Any changes in the plane's speed along the direction of travel or up-down tilt of the nose results in a change of the spacing of the scans...they bunch up if the plane slows down, resulting in a higher scan density in that area. So turbulence and pilot input effect the scan density and location, given a fixed laser pulse rate and mirror scan rate. The second figure is a close-up of a small part in the lower left of the first plot. Its a little easier to see the varying point density here. As I understand it, DNR asks for some average point density to meet their goals, and the plane's flying height, the laser pulse rate, and the scan width are set accordingly. You don't get precisely 2m posting density everywhere. Some areas have much more than 2m and some areas have less, but its some kind of an average that defines the spec. Note also that with a 60% sidelap every area is covered by at least two adjacent flight tracks, and the scans tend to interleave randomly resulting in higher along-track sampling than that of a single track alone. The third figure is zoomed-in farther still, and this time I have drawn a red line connecting sequential points (the points are represented by white crosses here). Note that sometimes the points appear to "back up," while other times they are in a nice, evenly spaced sequence. This particular dataset is the first-return set. For each laser pulse there can be multiple reflections - one hitting say the high branches of a tree, the next hitting a smaller tree just beyond and lower than the first, and a final echo off the cold, hard ground farther still. If there's no tree in the way the beam has to go farther to hit the first-return in that case - which is the ground. This difference in distance to the nearest first-return gives the appearance of the sequential scan slowing down and speeding up, even though the laser is being pulsed at a constant rate and the scan mirror is sweeping at a constant rate and the plane is in level flight. You might notice that sometimes there is no reflection at all - there are gaps in the data. In particular, in the first figure you can see that the river and a couple of lakes show black patches where there was no recorded return on one or both adjacent tracks. Sometimes there is a reflection from the water, but sometimes not. I can immagine that this might depend on the angle that the laser hits the water. The surface would act like a mirror. If the surface is normal to or nearly normal to you'll get a strong reflection, but if it hits at a shallow angle most of it will reflect off at the angle it came in (away from the receiver), with only a small ammount of backscatter. If the water is wavey you might also get a stronger reflection. But the receiver has some signal strength threshold at which it triggers a return, and below that level the pulse is just lost. Some materials on the surface might even absorb the particular wavelength of light the lidar is using, although this is probably rare (as an analogy, think of a radar beam hitting a stealth fighter). Since the lidar has all the information it needs to know about exactly where the aircraft is (DGPS) and how it is orientend relative to the surface of the earth (inertial frame given aircraft telemetry), then the time it takes for the light pulse to bounce back gives the precise location of whatever it hit, be it the top of a tree or the bottom of a 1.5m depression. Ground control points (GCPs) can be established at known locations to help get everything referenced properly in post-processing. I presume this is what leads to the small cross-track vertical errors that show up as "cornrows." These are much smaller than the vertical resolution of the data itself though, which tends to be 15-20 cm...pretty amazing. |
| # Posted 2005-09-28 07:47:07 | |
| glennon: | An initial page of links to background information about LIDAR with the most relevance to creating orienteering basemaps is now online at http://www.lidarbasemaps.org/
Feel free to praise or pan any of the linked sources, or even better, as you come across things you feel are particularly useful, let me know so I can update the page. |
You must be logged in to add a message
