If you want to disable all type of connection delay, you have to disable GSSAPIAuthentication too if active. GSSAPI is a IETF standard for doing strong encrypted authentication in network based applications. It is also supported by openssh but it can also introduce delays. Some distributions left this mode active on default, so you have to change like this in /etc/ssh/sshd_config:
GSSAPIAuthentication no