This is a pita for sure if you are low level and trying to do this. Best bet if you have invis is to approach the pond from a direction that he will not spawn (see other locs given while invised. He seems to love to spawn on you in the middle of a bunch of angry mobs while you are invisible. The center of the pond is also apparently a possible spawn point but there are no other mobs there to enter the struggle and wipe you out.
Good luck.