Exchange Solutions

Ran into a strange issue recently during an Exchange 2010 to 2016 migration. Internal mail sent from Exchange 2016 to Exchange 2010 was stuck in the mail queue. The queue viewer on Exchange 2016 reported the following error.

{LED=451 4.4.397 Error communicating with target host. -> 421 4.2.1 Unable to connect -> SocketTimedout: Socket error code 10060};{MSG=};{FQDN=<external.companyname.com>};{IP=<external IP>};

This is a fairly generic error and I have changed the FQDN and IP address in the example above. But the key here is that the Exchange 2016 server was trying to send all internal mail to the public IP of the Exchange 2010 server versus the internal IP.

For example, if a test user on Exchange 2016 tried to send an email to a test user on Exchange 2010, 2016 was routing the mail externally out of the firewall, only to try and hairpin back to one of the public-facing IPs.

This kind of hairpin attempted by Exchange was immediately blocked by the firewall which determined that internally sourced connections should not be trying to enter the public side of the firewall.

Ran into a strange issue recently where all the default address lists were missing in Exchange. Running a Get-AddressList in the Exchange Management Shell returned zero results. The address lists were also absent in the Exchange Admin Center as well.

When we attempted to recreate the missing address lists using the same name (for example, “All Users”) we received an error that the address list already existed.

 C:\> New-AddressList "All Users" -Included Recipients MailboxUsers
Active Directory operation failed on dc1.skaro.local. The object 'CN=All Users,CN=All Address Lists,CN=Address Lists Container,CN=SKARO,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=skaro,DC=local' already exists.
+CategoryInfo : Not specified (0:Int32) [New-AddressList], ADObjectAlreadyExistsException

If we attempted to modify this address list (or remove it) it reported it could not be found.

 C:\> Set-AddressList "All Users" -Included Recipients MailboxUsers
The operation couldn't be performed because object 'All Users' couldn't be found on 'dc1.skaro.local'.

Similarly, if we tried to create a brand new address list that we knew never existed in the environment previously this also failed.

 C:\> New-AddressList "Brand New List" -Included Recipients MailboxUsers
The operation couldn't be performed because object 'CN=Brand New List,CN=All Address Lists,CN=Address Lists Container,CN=SKARO,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=skaro,DC=local' couldn't be found on 'dc1.skaro.local'.

These address lists were also missing when we used ADSI Edit to examine the Address Lists Container.

I recently ran into an issue where the Microsoft Exchange FrontEnd Transport service refused to stay started. As a result, no external mail was being delivered to the Exchange server.

From the Services app, I could start the service, but it would stop within a couple of seconds.

When reviewing the Application logs in the Event Viewer I ran into a few separate errors with a source of MSExchangeFrontEndTransport. The first error was merely a symptom of a broken Front End Transport and not the root cause. We ignored this one and moved on.

Inbound direct trust authentication failed for certificate %1. The source IP address of the server that tried to authenticate to Microsoft Exchange is [%2]. Make sure EdgeSync is running properly.

The second error gave us the clue we needed. This error indicated something else was already listening to port 25.

The address is already in use. Binding: 0.0.0.0:25.

When installing an Exchange update you may run into the error; Service: MSExchangeHM failed to stop.

[ERROR] The following error was generated when "$error.Clear(); 
& $RoleBinPath\ServiceControl.ps1 -Operation:DisableServices -Roles
($RoleRoles.Replace('Role','').Split(',')) -SetupScriptsDirectory:$RoleBinPath;
& $RoleBinPath\ServiceControl.ps1 -Operation:Stop -Roles:
($RoleRoles.Replace('Role','').Split(',')) -IsDatacenter:([bool]$RoleIsDatacenter)" was run:
"Microsoft.Exchange.Configuration.Tasks.ServiceStopFailureException: 
 Service 'MSExchangeHM' failed to stop due to error:'Cannot stop MSExchangeHM service on computer '.'.'. 
---> System.InvalidOperationException:  Cannot stop MSExchangeHM service on computer  '.'. 
---> System.ComponentModel.Win32Exception:  The service has not been started   
    --- End of inner exception stack trace ---
    at System.ServiceProcess.ServiceController.Stop()

The MSExchangeHM service is part of the Managed Availability feature in Exchange. Managed Availability performs monitoring and self-healing to ensure Exchange stays up and running. This is a service you want to ensure stays running during normal operation, but in this particular scenario, we need to terminate the service so our cumulative update can complete.

The following servers in the Windows Failover Cluster are not in Active Directory: <server name>. This is usually the result of an incomplete membership change (add or remove) of the database availability group.

I ran into this error recently while trying to remove two Exchange 2010 members from a database availability group (DAG).

The error stated that a member of the DAG, a server named EXC3, did not exist in Active Directory. This was odd because queries to the Exchange 2010 management tools only returned two Exchange servers–EXC1 and EXC2.

We further confirmed that there was no computer account for EXC3 in Active Directory Users and Computers. We did, however, see remanents of EXC3 in ADSI Edit.

Talking with our customer we discovered that there had been a third Exchange server, named EXC3, that had crashed and was never recovered.

Fix–The following servers in the Windows Failover Cluster are not in Active Directory

To verify the status of all nodes in your database availability group, open PowerShell and import the Windows Failover Clustering cmdlets with Import-Module.

 C:\> Import-Module FailoverClusters

Next, run the Get-ClusterNode cmdlet. This will retrieve the status of all our nodes.

 C:\> Get-ClusterNode

Name                                                  State
----                                                  -----
EXC1                                                     Up
EXC2                                                     Up
EXC3                                                   Down 

In the example above, we can see EXC1 and EXC2 are operational, whereas EXC3 is offline.

Because EXC3 no longer exists (and the fact we plan to collapse the entire DAG anyway) we can forcibly evict the failed node. To do this issue the following command.

 C:\> Get-ClusterNode -Name "EXC3" | Remove-ClusterNode

Remove-ClusterNode
Are you sure you want to evict node EXC3
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): Y

You will be prompted to confirm. Press enter to accept the default action of “Yes”.

If we repeat the first Get-ClusterNode command we will only have the two operation cluster nodes remaining.

 C:\> Get-ClusterNode

Name                                                  State
----                                                  -----
EXC1                                                     Up
EXC2                                                     Up

With no more failed nodes we can remove the two operational nodes using either the Exchange 2010 management console or PowerShell.

Have you run into this error while add or remove members to a DAG? What did you do to fix it? Drop a comment below or join the conversation on Twitter

While applying a cumulative update I ran across the following error.

Warning:
An unexpected error has occurred and a Watson dump is being generated: Could not find a part of the path 'C:\Windows\Temp\ExchangeSetup\bin\EnterpriseServiceEndpointsConfig.xml'.

The Exchange setup log essentially repeated this same error, without any additional clues. The C:\Windows\Temp\ExchangeSetup\ folder is a staging area used by setup to unpack temporary files. These files are copied to this temporary location during the ‘Initializing Setup’ and ‘Copying Files’ steps.

When navigating to this folder it became apparent that not only was the file in question missing but the entire parent folder too.

When performing a cumulative update on Exchange 2013 I ran into an issue where the finalization step was hanging as 45%.

When I checked the Exchange Setup logs I saw the following repeated over and over.

Service  'MSExchangePOP3BE' failed to reach status 'Running'  on this server after waiting for '25000' milliseconds.
[WARNING] Service checkpoint has not progressed. Previous checkpoint='0'- Current checkpoint='0'.
Previous service status query time is '7/5/2018 1:35:37 AM'.
Current service status query time is '7/5/2018 1:36:02 AM'.
Will wait '25000' milliseconds for the service 'MSExchangePOP3BE' to reach status 'Running'.

In this instance, the Exchange setup was unable to start the POP3 backend service. Attempting to start the service manually through the Services MMC also failed. Upon checking the System Event Log the following error was reported.

The address is already in use. Binding: 0.0.0.0:9955.

This error indicates something else was already listening on port 9955, which was preventing the POP3 backend service from starting (and setup from completing).