In case anyone else has this problem, adding this to the RowLayout does the trick:
Rectangle { height: rowLayout.childrenRect.height }
The Rectangle by default has 0 width, and the RowLayout's childrenRect.height property has the value we're after, i.e. the maximum height of its children, visible or not. Bit of a hack, obviously.