Random 500 errors on IIS when using client certificates / Boyan Mihaylov

Recently, I have been working on an application that provides some web services to a client. The communication between the client and our application runs through HTTPS. Furthermore, the client has to provide a client certificate whenever requesting our web services. The application is built on WCF and runs behind IIS 10 on Windows Server 2016. Everything worked fine until we looked at the response statistics our client had generated, where we could see we weren’t responding 100%. The statistics showed response drop to 90%-95% every other day.

Inspecting the logs

The first thing I did was to check our application log where I couldn’t find any exceptions logged. We log all (unhandled) exceptions in our application, which would normally lead to an error HTTP status code, so finding no such exception made me think that the request has not reached the application at all. The next step was to check the IIS logs and there I could find some 500 errors. The amount of these errors matches the response drop in the statistics we got earlier, so I was happy to be on the right track. I set up Failed Request Tracing on IIS and after a while I found following error logged:

ModuleName IIS Web Core
Notification 1
HttpStatus 500
HttpReason Internal Server Error
HttpSubStatus 0
ErrorCode 2147943395
ConfigExceptionInfo
Notification BEGIN_REQUEST ErrorCode

The I/O operation has been aborted because of either a thread exit or an application request. (0x800703e3)

To be honest, I had no idea what it meant, so I googled it. Unfortunately, there isn’t much about this error. Some forums pointed out it may be related to SSL and client certificates. This made me re-examine our application’s set-up.

Examining the set-up

IIS is configured with a HTTPS binding and under SSL Settings SSL connection with a client certificate are marked as required.

In the web.config the web services are also configured to accept only requests through HTTPS with a client certificate.

<system.serviceModel>
  <bindings>
    <basicHttpBinding>
      <binding name="MyServiceBinding" ...>
        <security mode="Transport">
          <transport clientCredentialType="Certificate" />
        </security>
      </binding>
    </basicHttpBinding>
  </bindings>
  ...
</system.serviceModel>

<system.serviceModel>

</security>

</binding>

</basicHttpBinding>

</bindings>

...

</system.serviceModel>

To me everything looked fine, until I found this blog post by Rick Barber, saying you need to do some more configurations on a lower level using netsh http command.

Configuring client certificate negotiation

To see all configured SSL bindings on your machine, you can run the following command

netsh http show sslcert

1	netsh http show sslcert

The result on the server looks like this (additional bindings are omitted for clarity)

Notice that, although we have selected in IIS that client certificate should be required, the negotiation stays as disabled. So what I did was to remove this binding and re-add it again with enabled client certificate negotiation.

netsh http delete sslcert ipport=0.0.0.0:443
netsh http add sslcert ipport=0.0.0.0:443 certhash=66226ad352d78302dc1c2a92d0db9c6a6d147998 appid={4dc3e181-e14b-4a21-b022-59fc669b0914} certstorename=My verifyclientcertrevocation=Enable VerifyRevocationWithCachedClientCertOnly=Disable UsageCheck=Enable clientcertnegotiation=Enable

netsh http delete sslcert ipport=0.0.0.0:443

netsh http add sslcert ipport=0.0.0.0:443 certhash=66226ad352d78302dc1c2a92d0db9c6a6d147998 appid={4dc3e181-e14b-4a21-b022-59fc669b0914} certstorename=My verifyclientcertrevocation=Enable VerifyRevocationWithCachedClientCertOnly=Disable UsageCheck=Enable clientcertnegotiation=Enable

Conclusion

After fixing the binding using the netsh http command, we haven’t experienced any more issues with the communication between the client and our web services. To be honest, I am still unsure why this has fixed the issue, as it wasn’t all-or-nothing for the client requests before, i.e. before it was still working, though only in 90% of the cases on average. If you have more information on the case, I would be more than happy to discuss in the comments below.