Ran into a strange problem recently where an Exchange 2016 server could not send mail to Office 365 via hybrid mail flow. What made this situation particularly strange is that other Exchange servers in the environment had no problem sending messages over the hybrid connection. On the problem server, messages would get stuck in the queue and eventually time out.
The queues were filled with retries such as these.
451 4.4.0 Primary target IP address responded with: "421 4.2.1 Unable to connect." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or deliver failed to all alternate hosts.
This message tells us that the server was unable to connect to Office 365. Unfortunately, it does not give us much detail beyond that. For that level of detail, we need to enable logging on the SMTP send connector used to send mail to Office 365.
Turn up logging on the SMTP Send Connector
To enable logging on a send connector, log into the Exchange Admin Center (EAC) and select the Mail Flow tab and Send Connectors sub-tab. Double click the send connector named Outbound to Office 365 and select Verbose under the General tab. Click Save.
To perform this same action through the Exchange Management Shell (EMS) type the following command.
C:\> Set-SendConnector -Identity "Outbound to Office 365" -ProtocolLoggingLevel Verbose
Note: Protocol logging can take some time before it starts creating log files. You can jump-start this process by restarting the Microsoft Exchange Transport service. Keep in mind this will disrupt mail flow on that server while the service restarts. The default location for SMTP send logs in Exchange 2016 is %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpSend.
While we waited for logging to generate some entries we also confirmed that we could successfully make a connection from the problem server to Office 365. For this task, we confirmed that we could telnet over port 25 to Office 365 and send an email message. This confirmed two things. First that this server was not being blocked on outbound port 25. Second that this server could resolve and reach Office 365 servers.
[Read more…] about Hybrid mail flow: TLS negotiation failed with error NoCredentials