Having too many connections can be bad. Let's say we have 64 ranks. Let's say that we want the ranks to be grouped in 8. Then, when rank 3 want to send a message to rank 55, it is required to get the intermediate ranks. rank 3 -> intermediate(rank 3) -> intermediate(rank 55) -> rank 55 rank 3 sends to rank 0 rank 0 sends to rank 48 rank 3 sends to rank 0 rank 0 sends to rank 48 rank 48 sends to rank 55 rank 48 sends to rank 55 To route messages with Ray, you need to provide the option -route-messages. You can also provide the type using -connection-type == Connection types == - complete (same as not using -route-messages) - group (as described above, not really good because it is not balanced) - random (the graph has random n*log2(n)/2 edges) - debruijn (better than random, but needs a power of something) - kautz (better than debruijn, but needs n ranks where n=(k+1)*k^(d-1) where k and d are two integers.