Thats true for all operations as there is always some management overhead for the actual task to compute.
And with increasing units, who do the compute work (cores, shader, whatever), the management overhead rises.
In general tasks are never only compute. You always need to feed the units with...