The Codist

We discovered this week an issue where network calls to certain third parties were failing around 45 seconds or so on our mobile website. The problem happened on a checkout page where after a long wait it showed an unknown error alert. The scary thing was that the checkout actually happened but the user never saw it.

At first we thought it was Safari since it appeared to be on an iPhone but the error didn't make any sense. Someone tried one of our mobile iPhone apps and got the same error. So we thought it might be some kind of iOS issue. Before asking Apple I thought I'd try to reproduce it on my own iPhone but it didn't happen. It also didn't happen on the iPhone simulator so it got really confusing. Eventually I realized that a wired network and a Verizon network had no issues. So we tried several more AT&T phones of various kinds running Android plus a Sprint iPhone and also tried the phones over Wi-Fi. Only the AT&T devices had issues and only on a cell network, either 3G or LTE.

The error turned out to be a 504 error, which is a gateway timeout. Apparently AT&T runs a proxy for all web calls and if there is no response in 45 seconds or so it will kill the connection and return a 504 to the client. Sprint apparently waits until 2 minutes and I couldn't get Verizon to timeout (though I didn't wait more than 2 minutes).

We have this issue as some of our checkouts call out to third parties that we have no control over and sometimes they simply take too long. One of our internal calls calls to services our group has no control over and can also take too long.

Fixing this is a pain in the butt, we will have to convert these two calls to some kind of polling system but still maintain backward compatibility. If we could find the right person at AT&T the hope was that there might be a header we can add to tell the proxy to hang in there a bit longer but so far no dice in finding them.

Apparently one can modify a phone's cellular settings (at least on Android) to avoid the proxy but clearly we can't throw up an error saying "please modify your phone to make this work". We also joked that if it was a known issue (AT&T cell connection and one of the third party checkouts) we'd put up an ad for Verizon!