Verified Commit 9e3ecd39 authored by lelutin's avatar lelutin
Browse files

cross-ref cumin to direct-ssh setup and expand on effects of using batch 

Using cumin's batch size is still a possibility to avoid issues, but it
is preferred to configure yourself for direct ssh connections and avoid
using the batch size if not necessary.

if direct-ssh connection is not possible, then using the batch size hack
is still possible. using it does have some side-effects that one should
be aware of though.

small correction in the text after my tests today: the limitation is
imposed by the MaxStartups setting, not MaxSessions.
parent e5bf6970

howto/cumin.md

+23 −7
Original line number Diff line number Diff line
@@ -165,16 +165,32 @@ change the path to cumin in the bash function. 


## Avoiding spurious connection errors by limiting batch size



If your ssh configuration makes all hosts use the same `ProxyJump`, you might be

seeing some strange and fluctuating percentage of hosts will error out on

connection. This is because you are hitting your ssh proxy's max connections per

IP. This could be caused by the server's `MaxSessions` which defaults to 10.



In order to avoid weird issues, you should limit your batch size to 10 or lower,

e.g.:

If you use cumin to run ad-hoc commands on many hosts at once, you'll most

probably want to look into setting yourself up for [direct

connection](doc/ssh-jump-host#when-not-to-use-the-jump-host) to the hosts,

instead of passing through a jump host.



Without the above-mentioned setup, you'll quickly hit a problem where hosts give

you seemingly random ssh connection errors for a variable percentage of the host

list. This is because you are hitting ssh server limitations imposed on you on

the jump host. The ssh server uses the default value for its `MaxStartups`

option, which means once you have 10 simultaneous open connections you'll start

seeing connections dropped with a 30% chance.



Again, it's recommended in this case to set yourself up for direct ssh

connection to all of the hosts. But if you are not in a position where this is

possible and you still need to go through the jump host, you can avoid weird

issues by limiting your batch size to 10 or lower, e.g.:



    cumin -b 10 'F:os.distro.codename=bookworm' 'apt update'



Note however that doing this will have the following effects:



* execution of the command on all hosts will be much slower

* if some hosts see command failures, cumin will stop processing your requested

  commands after reaching the batch size. so your command will possibly only run

  on 10 of all of the hosts.



## Example commands



This will run the `uptime` command on all hosts: