There is very little I am afraid of but riding roller skis on suburban hilly road comes close
It can be scarey at times. Fortunately the roads in this particular area are wider than normal and don't have curbs. Presumably because there are no sidewalks. First rule of rollerskiing comes straight from safe driving class -- always have an "out". So be prepared to ski off into the grass, and start running when you hit it if necessary. Or worse case, sit down, like Alex did a few years back. Not pleasant but better than the alternative.
It's like hitting someone else's back wheel with your front wheel when riding a bike. If you've practiced ahead of time and know how to react, you don't have to go down. But if you haven't your natural reflex will put you on the pavement.
One of my most exciting spots is in the Cornell Plantations, which is a big rollerski hangout due to the combinations of hills, nice pavement, very little traffic, and 20 mph speed limit. There's a short blind downhill that has a drop that you can't really see all of until you've already committed. Sometimes I start down that hill and then discover the gardner has stretched a garden hose across the road. First time I encountered that my fight-or-flight went into overdrive searching for a reasonable solution. Speed was increasing quickly and options were decreasing. Eventually I just took a deep breath and jumped over it. Turned out to be no big deal but it's still a bit exciting every time it happens.