The first setting I try for SMTP is "localhost" as the host with no additional authorization.
Second, I'll set the host to "mail.domain.com" with "domain" being your web site's domain, again with no additional authorization.
Third, I'd keep the host at "mail.domain.com" and enable "AUTH". This requires that an actual mailbox be set up on your host with a username and password to login.
Fourth, I would disable AUTH and use the same account I just set up in the "Authorize by Checking Email" section.
Last, if it still doesn't work, try enabling both AUTH and "Authorize by Checking Email".
For "Test Mail Settings" to run the proper test, your Administrator Email and/or Bounce To Email must be something@yourdomain.com and your Remote Test Address should be an email address that doesn't exist on the same server.