Ran into a strange problem recently where an Exchange 2016 server could not send mail to Office 365 via hybrid mail flow. What made this situation particularly strange is that other Exchange servers in the environment had no problem sending messages over the hybrid connection. On the problem server, messages would get stuck in the queue and eventually time out.
The queues were filled with retries such as these.
451 4.4.0 Primary target IP address responded with: "421 4.2.1 Unable to connect." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or deliver failed to all alternate hosts.
This message tells us that the server was unable to connect to Office 365. Unfortunately, it does not give us much detail beyond that. For that level of detail, we need to enable logging on the SMTP send connector used to send mail to Office 365.
Turn up logging on the SMTP Send Connector
To enable logging on a send connector, log into the Exchange Admin Center (EAC) and select the Mail Flow tab and Send Connectors sub-tab. Double click the send connector named Outbound to Office 365 and select Verbose under the General tab. Click Save.
To perform this same action through the Exchange Management Shell (EMS) type the following command.
C:\> Set-SendConnector -Identity "Outbound to Office 365" -ProtocolLoggingLevel Verbose
Note: Protocol logging can take some time before it starts creating log files. You can jump-start this process by restarting the Microsoft Exchange Transport service. Keep in mind this will disrupt mail flow on that server while the service restarts. The default location for SMTP send logs in Exchange 2016 is %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpSend.
While we waited for logging to generate some entries we also confirmed that we could successfully make a connection from the problem server to Office 365. For this task, we confirmed that we could telnet over port 25 to Office 365 and send an email message. This confirmed two things. First that this server was not being blocked on outbound port 25. Second that this server could resolve and reach Office 365 servers.
Analyzing the SMTP logs
Once we re-tried a few of the messages in our queue we examined the SMTP logs. Like our telnet test, we could see Exchange was making the initial connection to Office 365. By line 14 we could see our server was transmitting its SSL certificate. However, this was quickly followed with the error TLS negotiation failed with error NoCredentials. This indicated an issue with our certificate. From face value, everything on the certificate looked correct.
Outbound to Office 365,<,"220 BL3FEO12EC057.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Thu, 28 Feb 2017 00:00:00 +0000", Outbound to Office 365,>,EHLO exchangeservergeek.com, Outbound to Office 365,<,250-BL3FEO12EC057.mail.protection.outlook.com Hello [207.46.150.129], Outbound to Office 365,<,250-SIZE 157286400, Outbound to Office 365,<,250-PIPELINING, Outbound to Office 365,<,250-DSN, Outbound to Office 365,<,250-ENHANCEDSTATUSCODES, Outbound to Office 365,<,250-STARTTLS, Outbound to Office 365,<,250-8BITMIME, Outbound to Office 365,<,250 BINARYMIME, Outbound to Office 365,>,STARTTLS, Outbound to Office 365,<,220 2.0.0 SMTP server ready, Outbound to Office 365,*,,Sending certificate Outbound to Office 365,*,"CN=webmail.exchangeservergeek.com, OU=Exchange Server Geek, O=SuperTekBoy, L=Cincinnati, S=Ohio, C=US",Certificate subject Outbound to Office 365,*,"CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US",Certificate issuer name Outbound to Office 365,*,07A7C02B0DF809CAB510F58A270A7DE9,Certificate serial number Outbound to Office 365,*,CE3A3D779940A6855B53E2F69EF2DA4BC374D3EE,Certificate thumbprint Outbound to Office 365,*,autodiscover.exchangeservergeek.com,Certificate alternate names Outbound to Office 365,*,,TLS negotiation failed with error NoCredentials
We then attempted to match the thumbprint of the certificate (highlighted in blue) against the working servers. It did not match. We then compared it against the certificate on the problem server. It did not match either. To compound the issue the thumbprint from the logs was not found in either the EAC or EMS. In fact, the thumbprint in Exchange Admin Center across all servers was the same. All servers also had that same certificate bound to SMTP. Yet somehow the thumbprint in the logs did not match.
Fixing TLS negotiation failed with error NoCredentials
What we found was that the problem server had two identical certificates with the same common name. One with the correct thumbprint and one with a thumbprint that matched the SMTP logs. We discovered this issue by loading up the Certificates MMC on the problem server.
To do this click Start and type MMC.
From the MMC console select the File menu followed by Add/Remove Snapin. Select Certificates and click Add. From the wizard select Computer account > Local computer > Finish. Click Ok.
Back on the console expand Certificates (Local Computer) > Personal > and select Certificates. Your duplicate is most likely here.
Determine which certificate is incorrect from the thumbprint. Right click on the duplicate and select Delete from the context menu. Click Yes to confirm.
As soon as we removed the incorrect duplicate certificate from the problem server (and retried the messages in the queue) mail started to flow. It’s uncertain why Office 365 does not attempt to match the thumbprint rather than just the common name. Hopefully, we will see this change in the future update to Office 365.
Have you run into this error? What was your solution? Drop a comment below or join the conversation on Twitter @SuperTekBoy.
Thank you, fixed an issue for us. There was a second cert with the same name which didn’t have a private key (probably botched renewal)
Wow, thank you! A different admin had tried to renew the cert yesterday and broke it. Removing that incorrect certificate did the trick!
This dug me out a hole quickly, thanks very much for your post Gareth, your MVP status is well deserved!
The same issue, but i have checked and no duplicate certificates. The validation from Exchange online is still failed:
450 4.4.317 Cannot connect to remote server [Message=UntrustedRoot]
I use a wildcard certificate.
Thank you so much, for the last 12 hours I’ve been banging my Head on the keyboard trying to resolve this.
Thank u so much
Thank you so much, for the last 30 hours I’ve been banging my Head on the keyboard trying to resolve this.
My problem was a bit different but the same solution.
After I configured my exchange 2016 with a Hybrid configuration, I started to get NDR from Gmail and alot of other domains saying:
“TLS Negotiation failed: FAILED_PRECONDITION: starttls error (0): protocol error”
What happened was that my Exchange Certificate for some reason got duplicated, after I deleted the certificate that was not in USE, mails started to flow again.
Thank you
Thank you so much for this.
in my case i imported the certificate the wrong way the first time around, and fixed it later.
but this made me get the issue you helped me resolve.
thank you for saving me alot of headaches.
Thanks, This dug me out a very frustraing hole.
This is one of the many reasons I bang on aabout people removing old certs when they expire!
Hi there,
Thanks for the response. The thumbprint in the logs does match the thumbprint of the wildcard certificate. But it’s not a new certificate, it was installed about a year ago and has been working successfully for our on-premise server.
I read something else that pointed to the intermediate certificate. I looked in certificates console, and I am pretty sure that the intermediate that came with our certificate is installed – but I re-imported it. And the root certificate. Restarted IIS but did not restart the server. I may do so this evening.
Something’s set up wrong, but I can’t put my finger on it.
Hello again,
Almost a month later and I am still working on this. Your comment about the intermediate certificate was right. SSL certificate checkers did not report any chain errors; however, when I emailed my certificate provider (Comodo, now Sectigo), they said I was missing an intermediate certificate. There should be two intermediates rather than one. I got that sorted out and reinstalled the certificate with the correct chain. When I view the certificate in IIS or in MMC\certificates, it shows the proper chain.
But, Exchange has not updated. When I go to OWA, and look at the certificate, it is missing the second intermediate.
Any ideas about how to force Exchange to update. Should I just remove the certificate from Exchange using EAC, and then re-add it? And which services need to be restarted?
I have a feeling I am very close to getting this straightened out.
Thanks, Lisa
Hey Lisa,
Have you confirmed the IIS and SMTP services are now using the fixed certificate?
I am not sure how many Exchange servers you have, but if you have more than one you will need to make sure the certificate and its chain is valid on each Exchange server. This can be done with a simple export/import (you don’t need to do a cert request for each one). Also, if you have a load balancer or reverse proxy you may also need to update the certificate, and its intermediary certs, there as well.
I know you originally mentioned an issue with SMTP, were you able to add this cert to the Office 365 send connector and confirm the thumbprint being used?
I have been using the same certificate throughout, so the thumbprint hasn’t changed. And I’m talking a single Exchange Server.
This last time I imported the certificate right in the Certificates console after manually removing it. I then assigned SMTP and IIS services in Exchange.
I think my next step will be to remove it and import it in Exchange and see how it goes. Seems odd to me that when I look at the certification path in IIS it shows the proper chain, but when I look at it as an end user in OWA it shows the chain with the missing intermediate.
Thanks for the good information.
I have a similar certificates issue that I have been unable to solve. I have a Exchange 2013 environment, with most mailboxes residing on our on-premise Exchange server. We are still directing mail flow through the on-premise server. I have an “outbound to Office 365” send connector in place, but Office 365 users cannot receive mail (they can send mail) from internal or external addresses. All mail to Office 365 mailboxes queues up in the xxx.mail.onmicrosoft.com queue. Turning on verbose logging, the error is “Outbound TLS authentication failed for auth level DomainValidation with error UntrustedRoot”.
I have studied the certificates on our end, and made sure root and intermediate certificates are installed. We have a wildcard certificate from Comodo (now Sectigo) that has been working properly for more than a year.
I have contact Microsoft technical support and they have been unable to help so far. If anyone has any ideas they’d be much appreciated!
Hey Lisa,
Have you confirmed the thumbprint being used in the Send Connector logs, to confirm it matches that of your new certificate?
The error sounds like it doesn’t like your certificate chain which would make me think the intermediate ceritificate has an issue.
Very helpful.
Wow! Perfect fix. Never would have thought. Weird!
Thanks!
Thank you So Much Gareth . Document really helped us to resolve the “Outbound to O365” send connector issue post we enable the verbose login and found that SMTP service is not bind to certificate associated with connector and server.
Thank you Gareth! I had this issue while using PHPmailer and SMTP, and after spendig several hours of research your hint saved me the day!
Thomas
Thanks for this! I’m sure this saved me several hours of troubleshooting.
In my case, the two certificates were an older revoked version and a new updated version with an additional name. Since the send connector itself picks based on the FQDN specified in the send connector, I’m guessing it just picked the first in the list.
I suspect that if the cert were expired instead of revoked, it would have ignored it an picked the correct one. But I’ve not tested.
Hey Bryon,
Unfortunately, with Forced TLS / EOP, it appears Exchange sends back any cert that matches the name EOP is requesting. With regard to expired, I am not sure. But I have definitely had the same experience as you with the revoked cert.
This means that staging a cert for Forced TLS can be a challenge. You essentially need to assign the cert to SMTP and remove the old certs immediately. Does not happen with Mutual TLS.
I faced a similar issue after replacement certificate that near to expire. And exactly at the time of expiration, my mail flow was stopped. After some research, I found the cause. Maybe this will be useful for readers of this blog. It’s not Office 365 issue, but the peculiarity of certificate binding method to SMTP connector object in Microsoft Exchange Server. When you enable “TLS required” end set X509 certificate to the connector for mutual authentication, the only way to bind the X509 certificate to the connector is the “TlsCertificateName” property. This property can contain only “SmtpX509Identifier” object class that has only three properties: CertificateIssuer, CertificateSubject and CertificateCommonName. Only two of this properties forms TlsCertificateName value in format “Certificate.Issuer
Certificate.Subject”. No information about certificate thumbprint stores here.Of course, you can derive this name from X509 certificate object in PowerShell where you identify this certificate by its thumbprint, but only two property will be used to bind the certificate to the connector. Issuer and Subject.
If you use the same certificate provider many years and your server changes his name not often, you receive certificates with precisely same Issuer name and Subject name whenever you recreate them.
And if you don’t delete old certificates from local machine store, this leads to issues as you described in your article, because server selects first suitable certificate (and new certificates is added at the end of the list)
If you are renewing an existing cert the TlsCertificateName parameter may still not help as it will likely pick the original cert. Changing this parameter will fix if you are changing the names on your cert.
Had the exact same problem, but only on one of the Exchange Mailbox servers. Turns out that particular mailbox server had an extra certificate with the same CN as the one I wanted to use and/or defined in the Exchange Hybrid Wizard. Once I deleted the other certificate, my mail queue no longer experienced the “retry” behavior.
Thanks for the write up!
Exact same problem – Thanks!
Well, as i can see you already set your MX to Office 365 i.e. exchangeservergeek-com.mail.protection.outlook.com so email flow did not hit your Hybrid server for delivery it hit EOP
This error is regarding communication issues between the on-premises Exchange servers and Exchange Online via the hybrid secure mail flow. In a hybrid scenario mail flow between on-prem and Exchange Online is considered an extension of the Exchange organization and therefore internal. MX records and EOP do not come into play here. Hope that helps.
this will definitely help someone in troubleshooting hybrid mail flow
After one entire day of research i found your post: you saved my day! thank you!