You move the arrow head by changing the coordinates within the polygon itself, instead of moving the whole arrow head
(To illustrate: Let's say your arrowhead item is a piece of paper. The polygon within is a triangle you scribble on the paper. When you want to move the triangle, you can (a) move the paper, or (b) keep the paper where it is, but scribble the triangle in a different part of the paper)
Your RotateHandleItem is affected by the zoom. Now you use coordinates from within that object to determine the polygon for the arrow head. That way, you go over coordinate system boundaries without correctly mapping the coordinates.
Easier way: Assign the arrow head a fixed polygon, and use the setPos method to move it to the correct position.