There also seems to be hidden effect on the buff.
I was in SoL, when I noticed my friend could harvest items about a second faster than me. We spent about 5 minutes looking at buffs, and it seemed he had nothing to make it faster than me, except the harvesting cloak buff.
On the last 2 shinies in zone, we did a little test. I harvested it with the buff, and without the buff, and it seemed to shave a second off my time when I had the harvesting cloak buff on.
I want to run another test, because I was just counting myself and might have messed up.