When performing a cumulative update on Exchange 2013 I ran into an issue where the finalization step was hanging as 45%.
When I checked the Exchange Setup logs I saw the following repeated over and over.
Service 'MSExchangePOP3BE' failed to reach status 'Running' on this server after waiting for '25000' milliseconds. [WARNING] Service checkpoint has not progressed. Previous checkpoint='0'- Current checkpoint='0'. Previous service status query time is '7/5/2018 1:35:37 AM'. Current service status query time is '7/5/2018 1:36:02 AM'. Will wait '25000' milliseconds for the service 'MSExchangePOP3BE' to reach status 'Running'.
In this instance, the Exchange setup was unable to start the POP3 backend service. Attempting to start the service manually through the Services MMC also failed. Upon checking the System Event Log the following error was reported.
The address is already in use. Binding: 0.0.0.0:9955.
This error indicates something else was already listening on port 9955, which was preventing the POP3 backend service from starting (and setup from completing).
Finding the conflicting service
To determine the conflicting service open command prompt as an administrator and run the following command.
Netstat -ano | findstr :9955
Netstat shows all ports open to our server, what address has the port open, and the process ID (or “PID”) of the service or process that is using the port. We then pipe the output of Netstat into FindStr (“Find String”) which looks for any instances of “:9955” and returns it. In our case, Netstat returned the following.
TCP 10.0.0.1:9955 10.0.0.2:389 ESTABLISHED 5904
The important value here is the last. 5904 is the process ID of the service or process that is holding port 9955 open. We can discover what this process is by running the following from a command prompt.
Tasklist /fi "PID eq 5904" Image Name PID Session Name ========================= ======= =============== MSExchangeFrontEndTranspo 5904 Services
In this command, tasklist returns all processes running on the server and then filters that output for the process ID equal to 5904. From this output, we can see the Microsoft Exchange Frontend Transport has taken over port 9955.
Getting Exchange setup to complete
Now that we have identified the root cause of the service not starting we can remediate. The quickest way to resolve is to temporarily stop the Microsoft Exchange Frontend Transport service using the Services MMC. To do this, open the Services MMC, right-click on the service in question and select Stop from the context menu.
Once stopped right-click on the Microsoft Exchange POP3 Backend service and click Start.
The service should start successfully and setup will continue. Note that it may take a couple of minutes for setup to notice that the service has started. Once setup has completed, restart your Exchange server. After the reboot, verify all Exchange services, including the Microsoft Exchange Frontend Transport and the Microsoft Exchange POP3 Backend services, are started.
Have you seen this issue before? What did you do to fix it? Drop a comment below or join the conversation on Twitter @SuperTekBo