I've done it before by just continuing on through that farm place. If I recall correctly it used to be public access down into that valley.
I considered it but wasn't confident enough on my own. It's a shame if there's no public access because it seems that side of the mountain has the best path up.