Responsive, round-the-clock customer support is something we take very seriously at Imperva Incapsula. Our application delivery services are used to ensure the security and availability of websites and online applications. We understand that when a problem is encountered our enterprise customers need answers fast – even if it’s the middle of the night.
Our customer support department is built to do just that. We have over 30 highly trained support engineers on duty 24 hours a day, 365 days a year. They work in our state-of-the-art, global Network Operations Center (NOC), equipped with advanced monitoring tools that give them “eyes on the glass” for fast response to network events.
In this post, we’ll take you behind the scenes so you can see what it’s really like in the trenches with Incapsula’s Tier 1 support warriors. We’ll talk to a couple members of the team and let them describe how they deal with the day-to-day challenges of their job.
So strap on your helmets, check your gear and get ready for a taste of life on the front lines.
Our Mission: Making Sure Customers Always Get the Best Results from Incapsula
The main responsibility of our Customer Support team is to give our customers the fastest possible response to any and all technical, security and network issues. Tier 1 Support often starts with guidance for the initial Incapsula setup, and includes ongoing resolution of support tickets to maximize uptime for enterprise websites and online applications.
Customers can open support tickets by phone, email or via our online portal. Using our automated Zendesk system, tickets are prioritized based on each customer’s response time SLA.
If for some reason a support engineer needs to escalate a ticket, the first stop is with the Tier 1 Shift Lead. In many cases the Shift Leads are able to resolve the ticket by themselves. If not, the ticket is forwarded to Tier 2 support, just down the hall from the NOC. Tier 2 and Tier 3 support is provided by subject matter experts from the network, security or R&D teams, who have the necessary in-depth knowledge to handle more complex or uncommon issues. These experts are on call 24×7 to support escalation scenarios for customers anywhere on the globe at any hour.
Meet the People Who Make It All Happen
Most of the Tier 1 support engineers are between 20-32 years old and have a technical background. Their workday is comprised of three 8-hour shifts – each support engineer works at least 4 shifts per week, including nights and weekends.
We spoke to Don, who is a Tier 1 Shift Lead. When he’s not on duty in the NOC, he’s also a third-year medical student (talk about multi-tasking!). He told us that working shifts has given him the flexibility to mix work with a full study schedule – and still get a few hours of sleep.
What’s your favorite part of the job?
Don:The vibe in the NOC is awesome. There’s a constant buzz in the air, and it’s a dynamic and young atmosphere. I also enjoy speaking to customers from different countries and cultures around the world. We’re responsible for our customers’ security, and it’s a great feeling to help them solve problems that really matter to their business.
What’s the most memorable incident you’ve ever had with a customer?
Don: One incident I’ll never forget involved a large company that received a ransom email from DD4BC, a known DDoS attack group. The email informed them that they were under attack, and threatened to launch a second 400+ Gbps DDoS attack unless the company paid a 40 BTC (~$9,200) ransom within 24 hours.
Instead of paying the ransom, the company (now a customer) immediately contacted Incapsula to deploy DDoS Protection. It was our job to help them onboard Incapsula and set up the most appropriate thresholds for the DDoS attack. We guided them over the phone throughout the setup process, giving them step-by-step instructions. Within one hour, their site was protected by our service.
The follow-up attacks, which never exceeded 40Gbps, were easily mitigated by Incapsula, DD4BC never came back and – happily – no ransom was ever paid.
Can you give us an example of an escalation scenario?
Don: Sure. Last week one of our enterprise customers contacted us with a feature request for a specific customization of our notifications feature. We immediately escalated the request to our Tier 2 support team, who checked the feasibility of developing such a feature with the product team. As it turned out, the requested feature was already on our product roadmap and, as a result of the request, was moved up to the next release. Within two days, the customer received the good news and the ticket was closed.
What’s it like working the graveyard shift?
Don: Not as quiet as you might think. We’re in the European time zone, so by the time I start it’s usually pretty quiet on that front. But don’t forget, it’s only 1:00 PM then in New York and 10:00 AM in California. Towards the end of their day there can be a short lull, and then it’s “Good morning Australia.”
A Typical Day at the Office
“Typical” is probably not the right word to describe a work shift in the Incapsula NOC. According to Don, no two days are alike. An 8-hour shift can be consumed by one complex support ticket, or 30 tickets related to problems that affect multiple customers. During “quiet” periods and as needed, Don and his support engineers also write knowledge base articles and prepare tutorials for upload to the support portal.
As Shift Lead, Don arrives about half an hour before his shift starts to conduct shift handover. This is a documented process during which all noteworthy events that took place during the shift are reviewed together by the incoming and outgoing Shift Leads. To ensure that tickets do not fall between the cracks, support engineers make every effort to stay with their tickets until they are resolved. In the rare event of an unresolved ticket, the Shift Lead hands over all relevant information to the next support engineer.
Support Engineer Training
“One of the strengths of the Customer Support team is its in-depth technical and product knowledge,” said Ingrid, our Global Customer Support Manager. She’s in charge of the 24×7 support operation and the first one to receive a text in the middle of the night when a support engineer calls in sick.
How long does it take to get new Support engineers up to speed with the product?
Ingrid:It usually takes a few months before support engineers are certified to handle tickets independently. Most of the new hires come to Incapsula with good technical knowledge. Their training at Incapsula begins with an introductory 5-day course that includes simulations based on recorded conversations with customers. The next stage consists of three months of on-the-job training, where each new employee is paired with an experienced support engineer. This process culminates with a certification exam. We also provide ongoing training workshops once a quarter with the product team, as well as external workshops where the team can hone their inter-personal support skills.
What’s your favorite part of the job?
Ingrid: : To tell you the truth – everything. I love working with people. Every day there is something new to do and learn. I try to spend at least half a day every week in the NOC to really feel what my Tier 1 engineers are experiencing, get their perspective and see what processes can be improved.
What sort of feedback do you get from customers? Can you share examples?
Ingrid: For us, customer feedback is a tool for getting better. A great recent example is when a customer emailed us with a question about an error page he received with an error code that he wasn’t familiar with. He asked why we didn’t have a list of error codes and descriptions on our Zendesk-based support portal. We realized that such a list could be very helpful to all of our customers – we’ve prepared the page and plan to upload this information to the portal shortly.
In all cases, as soon as a ticket is resolved, we ask the customer to complete a short feedback form, rating our level of support. Sometimes, we are humbled by the responses we get for simply doing our job, like the one below (which was hard to forget):
Excellent service. Teresa was amazing at troubleshooting and determining the issue with my site. I appreciate the fact that she took ownership of the issue and would not pass it on to the next level without exhausting all of her trouble shooting techniques. She is a rock star.