Register | Login
Attackpoint - performance and training tools for orienteering athletes

Discussion: Downloading huge lidar file

in: Orienteering; General

Nov 19, 2020 11:05 PM # 
BorisGr:
I've been given access to an enormous lidar file that covers terrain that i am really excited to get mapped. The file is on the order of 75 gigs. Gswede (who has been generating KP maps for me) and i can't really figure out how to download and manage something that huge. Any suggestions?
Advertisement  
Nov 19, 2020 11:11 PM # 
igor_:
What's the format? Las, Laz, or something else?
Nov 19, 2020 11:31 PM # 
hughmac⁴:
Happy to download and tile for you into manageable chunks, if you like, assuming it’s LA[SZ]. Shoot me an e-mail if so.

But in general, you could fire up an AWS instance of a size that works, and do it there.
Nov 20, 2020 2:51 AM # 
Rhombus:
You have to download more RAM first
Nov 20, 2020 3:20 AM # 
Gswede:
Ram download

So, Hugh and Igor, they're las files in a zip folder. We're told they are tiled already, but just downloading them is the problem.

My internet is not great and will drop the download with a network error after an hour or so. I could try going to a cafe with fast internet nearby, but also not a fan of spiking cases at the university two blocks away.
Nov 20, 2020 9:26 AM # 
Terje Mathisen:
LAS-in-ZIP is a horrible format, barely better than naked LAS files.

What you want is either individual LAZ files, or LAZ files stored (_not compressed_) into a ZIP file. If you can reach those files from AWS (or Google) as suggested by hughmac, then you can run unzip + las2las to convert them to individual LAZ files which you would then download using wget or sftp, with retry enabled.
Nov 20, 2020 10:18 AM # 
Jan_B:
If you can access the ZIP file through HTTP(S), you should also be able to download/extract only a selection of the contained files.

Not sure what's the easiest way on your system but here's a method in Linux on the command line that looks legit:
https://stackoverflow.com/questions/8543214/is-it-...
Nov 20, 2020 10:36 AM # 
igor_:
Just trying to download the whole zip using wget with retry as Terje suggested might just work. It would resume from where it got interrupted. Then unzipping should work too with enough disk available.
Nov 20, 2020 1:11 PM # 
ken:
You can also continue failed downloads in Chrome, though it’s not automatic like wget.
Nov 20, 2020 2:07 PM # 
Gswede:
Alright, I've got some stuff to try out then.

I agree Terje. I was surprised they decided to put LAS files in a zip folder when they have their own compressed format.

Since I'm trying my hand at adulting, I got a 1TB external hard drive. So I should have plenty of space to work with the files when downloaded.
Nov 20, 2020 2:59 PM # 
hughmac⁴:
If you get me the link, I can download, extract to a directory, convert to LAZ, and upload to Dropbox as individual files (the whole folder), and share the Dropbox folder with you. Dropbox will keep trying till it’s all synced.
Nov 20, 2020 4:25 PM # 
gordhun:
An alternative a county in Florida did for me - send them a USB stick / thumb drive and have them load the file and send back to you. Matching each tile to the right part of the county was a small challenge but we got that figured out.
Nov 20, 2020 9:13 PM # 
hughmac⁴:
The converted LAZ files as a whole were 1/2 the size of the single ZIP file. :) All 540 files done, tested a KP run on a tile, and imported into OOM with some OSM data for quick visual georeference match (yay, already UTM 12N), and shared via Dropbox. HPC cluster spare cycles FTW. 6 minutes to extract, convert, and push all to Dropbox.
Nov 20, 2020 9:23 PM # 
BorisGr:
Woooo!
Nov 20, 2020 9:27 PM # 
hughmac⁴:
Looks like a really cool dataset, and location, Boris! Looking forward to seeing the results.
Nov 20, 2020 9:47 PM # 
Terje Mathisen:
Good luck everyone, I have had some frustrating experiences myself when creating maps of areas I was (going to) visit in the US, like some areas where you can only download LAS files of entire counties (or something similar?), anyway many, many GB even though I was only interested in maybe 10x10 km.

For both coastal California and in Boulder, Colorado the process was much nicer:

https://tmsw.no/qr/show_map.php?user=terjem&ma... (Santa Cruz MTB)
https://tmsw.no/qr/show_map.php?user=terjem&ma... (Flatiron hike/run)
Nov 20, 2020 10:19 PM # 
hughmac⁴:
I think I’d have doubled (or more!) the contour interval on that Flatiron map! :) Wow! Amazing terrain.

Love Santa Cruz, spent my senior undergrad year there, and lived another few years there, then up on the ridge, in redwoods, and live oak forests. Miss CA nature. And sushi.
Nov 20, 2020 10:33 PM # 
bbrooke:
Terje, if you're ever back in Boulder, I can try to find you a good copy of RMOC's "Devil's Thumb" map from that area (we're not allowed to have events there anymore...). http://omaps.worldofo.com/?id=2113
Nov 21, 2020 12:44 AM # 
jjcote:
The place is all nasty with cactus when you go off-trail anyway. The one time I set courses there, I threw away my boots when I was done.
Nov 21, 2020 9:22 PM # 
Terje Mathisen:
@bbrooks: Thanks, I don't know if I'll have have any professional reason to get to Boulder again, I've switched jobs this summer due to Covid. California (Santa Cruz and/or Monterey/Pacific Grove/Asilomar) is still likely though. Probably not until next November...

Please login to add a message.