Get off, stand by your bike, and let him try to push his bike around you. It's simply not two people + two bikes wide without serious invasion of personal space. Well, not two people of my girth anyway ;(
That's exactly what I did, and he was invading my personal space while he ranted at me. I seriously think that if I had been on the outside he'd have shoved me in. Some people are just nuts.
All the world old is queer save thee and me ...