Skiing is much slower, so much warmer. And you have more use of your hands to cover your face on downhills. Kudos for getting out there!
In my XC skiing days (45-55 years ago), I really had only one bad experience with getting cold.
It was a race in southern VT, started at an inn that might have been called the Hermitage Inn, or something else. Short section over to the base of the Haystack ski area. Up one of the downhill trails to the top. North along the ridge to Mt. Snow. Down the southernmost downhill trail, and then back to the inn on a trail that was pretty straight and gently downhill. And it was a cold day.
The problem was that from the top of Mt Snow to the finish you were moving pretty fast, sometimes real fast (I could ski downhill on my XC skis just fine). And I didn't have any sort of windproof pants or underwear or whatever it's called.
Finished, got back in the inn for something warm to drink (I think the field was maybe 20-30 skiers), and then after about 10 minutes a certain body part had started to warm up, and it started hurting, and then really hurting. I was there with a friend and we were both just dying.
It passed, of course. But skiing can get cold. (As I remember racing downhill could, had to wear face masks on the really cold days, otherwise just too painful.)
It is fortunate you didn't get frostbite on that, but I guess you have superior blood flow.
I guess if you had anticipated the problem, you could have put a face mask in your pants.