In my two-plus years as a Technical Support Engineer at Imperva, I’ve handled a wide variety of customer cases. And I’ve had the satisfaction of helping resolve them quickly and successfully. But never before have I received a handwritten thank you note from an effusive customer.
Let me start at the beginning.
The customer first reached out to us with this request for help in January of this year. While I can’t reveal its name, I can say that the company is a leading European-based information services company who had long used Imperva Cloud Web Application Firewall (WAF, formerly Incapsula) to protect dozens of its mission-critical websites.
However, the company was having trouble with file uploads on one of its main websites, according to the customer’s senior enterprise architect submitting the ticket. Whenever an employee would try to upload large files to the site, the transmission would time out before completion. This, he said, was an extremely critical issue that needed to be treated as a top priority and resolved as soon as possible.
After going through some server logs, we noticed something that seemed like it could be the problem. Our cloud WAF acts as a reverse proxy, with our proxy server sitting in between the end user attempting to upload the file and the web server. When the user attempts to upload a large file, it is typically divided before being sent in chunks. Based on standard Internet protocols, also known as Request for Comments (RFC), the customer’s web server should have waited for the full HTTP request body to be received before returning an HTTP response confirming the request was successful. Because the web server responds early, that triggers an alert in our proxy server, which by default policy then terminates the connection.
I was able to spot this problem pretty quickly since I already had past experience with similar issues. And the mitigation was fairly straightforward — tweaking the configuration of our proxy servers so as not to close the existing connection with the web server whenever the malformed behavior is identified. Such a change does not loosen security or create any meaningful additional risk to their website.
However, it did take some time to convince the customer of this. They were right to be cautious: This website is key to their business, and any downtime would be disastrous.
So I took this up with our internal escalation team to confirm my analysis and get their insights, and then created a detailed explanation for the customer of the solution, the outcomes, and how their web security would remain unchanged. After reading that, they agreed.
Understanding the great impact this site has on their business, we decided to first make the changes to low-priority QA sites during off-hours when traffic rate is low, to ensure we are minimizing any possible risk. Once that proved successful, we made the same configuration changes to their production websites. Gladly, since then the upload failures haven’t re-occurred. And there’ve been no security incidents either, as we expected.
Figuring out the problem and the solution took only several days. Most of the additional time was spent researching and writing up the report, then getting the customer’s agreement, and then testing and deploying the solution. All told, we were able to resolve the case in under three weeks….and then I forgot about it. Until I received this in the mail:In this digital age, any analog communication is already a rarity. Such a great gesture and a pleasant surprise!
It’s a gratifying reminder to me of how important the work me and my colleagues do is, and how important Imperva’s security solutions are for our customers. And all of their customers, too.