Looks like you almost nailed it - it is easier in the other direction, I often miss the bit where you climb up the cliff from the way you came.
Yeah I had a look from the other direction and I'm guessing you can climb up to the pagoda from the foreshore. It's a pretty nice loop to do on a lunch break.
Yep, just squeeze left of the shed from memory, then a bit of a scrabble and you are up. Its the drop/climb up the crag line from the "farm" that I can never find coming anti-clockwise.