Repartitioning and top-level cells

Richard wanted to see how the numbers of top-levels cells impacted the various repartitioning schemes, so I've ran a number of tests and created some animated GIFs that show how the updates-time plot varies. First for the standard repartition type

costs/costs

The red dots are the default top-level cells, 12x12x12. Each plot is based on a single run of SPH EAGLE 50 on 6 nodes of COSMA6. So the same nodes are used in a plot, but can vary between plots.

Now for an edge cut using costs

none/costs

Now time bins as an edge cut

none/time

Now edge time bins with vertex costs

costs/time

Now particle counts only

counts/none

And finally particle counts with costs edges

counts/costs

They all agree that 36x36x36 is the optimal number, at least for the small steps. There are interesing effects in the linear domain for the edge cuts, when it looks like the balance can get quite bad, but not just as a direct consequence of the number of top-level cells, as it can improve again. Must be down to some issue in the geometry of the particles, as the edge cut seems to keep the same numbers of cells per node.

Some of the rebuild steps look a little different than I expected, it seems to be the case that more cells doesn't necessarily mean slower, for instance in costs/costsmoving from 12x12x12 to 24x24x24 shows a speed. It is only once you get to 48x48x48 that you see a penalty. Others have more close variations. Not sure I understand this.