Discussion: Downloading huge lidar file

I've been given access to an enormous lidar file that covers terrain that i am really excited to get mapped. The file is on the order of 75 gigs. Gswede (who has been generating KP maps for me) and i can't really figure out how to download and manage something that huge. Any suggestions?

What's the format? Las, Laz, or something else?

Happy to download and tile for you into manageable chunks, if you like, assuming it’s LA[SZ]. Shoot me an e-mail if so.

But in general, you could fire up an AWS instance of a size that works, and do it there.

You have to download more RAM first

Ram download

So, Hugh and Igor, they're las files in a zip folder. We're told they are tiled already, but just downloading them is the problem.

My internet is not great and will drop the download with a network error after an hour or so. I could try going to a cafe with fast internet nearby, but also not a fan of spiking cases at the university two blocks away.

LAS-in-ZIP is a horrible format, barely better than naked LAS files.

What you want is either individual LAZ files, or LAZ files stored (_not compressed_) into a ZIP file. If you can reach those files from AWS (or Google) as suggested by hughmac, then you can run unzip + las2las to convert them to individual LAZ files which you would then download using wget or sftp, with retry enabled.

If you can access the ZIP file through HTTP(S), you should also be able to download/extract only a selection of the contained files.

Not sure what's the easiest way on your system but here's a method in Linux on the command line that looks legit:
https://stackoverflow.com/questions/8543214/is-it-...

Just trying to download the whole zip using wget with retry as Terje suggested might just work. It would resume from where it got interrupted. Then unzipping should work too with enough disk available.

You can also continue failed downloads in Chrome, though it’s not automatic like wget.

Alright, I've got some stuff to try out then.

I agree Terje. I was surprised they decided to put LAS files in a zip folder when they have their own compressed format.

Since I'm trying my hand at adulting, I got a 1TB external hard drive. So I should have plenty of space to work with the files when downloaded.

If you get me the link, I can download, extract to a directory, convert to LAZ, and upload to Dropbox as individual files (the whole folder), and share the Dropbox folder with you. Dropbox will keep trying till it’s all synced.

An alternative a county in Florida did for me - send them a USB stick / thumb drive and have them load the file and send back to you. Matching each tile to the right part of the county was a small challenge but we got that figured out.

The converted LAZ files as a whole were 1/2 the size of the single ZIP file. :) All 540 files done, tested a KP run on a tile, and imported into OOM with some OSM data for quick visual georeference match (yay, already UTM 12N), and shared via Dropbox. HPC cluster spare cycles FTW. 6 minutes to extract, convert, and push all to Dropbox.

Woooo!

Looks like a really cool dataset, and location, Boris! Looking forward to seeing the results.

Good luck everyone, I have had some frustrating experiences myself when creating maps of areas I was (going to) visit in the US, like some areas where you can only download LAS files of entire counties (or something similar?), anyway many, many GB even though I was only interested in maybe 10x10 km.

For both coastal California and in Boulder, Colorado the process was much nicer:

https://tmsw.no/qr/show_map.php?user=terjem&ma... (Santa Cruz MTB)
https://tmsw.no/qr/show_map.php?user=terjem&ma... (Flatiron hike/run)

I think I’d have doubled (or more!) the contour interval on that Flatiron map! :) Wow! Amazing terrain.

Love Santa Cruz, spent my senior undergrad year there, and lived another few years there, then up on the ridge, in redwoods, and live oak forests. Miss CA nature. And sushi.

Terje, if you're ever back in Boulder, I can try to find you a good copy of RMOC's "Devil's Thumb" map from that area (we're not allowed to have events there anymore...). http://omaps.worldofo.com/?id=2113

The place is all nasty with cactus when you go off-trail anyway. The one time I set courses there, I threw away my boots when I was done.

@bbrooks: Thanks, I don't know if I'll have have any professional reason to get to Boulder again, I've switched jobs this summer due to Covid. California (Santa Cruz and/or Monterey/Pacific Grove/Asilomar) is still likely though. Probably not until next November...