Cisco needed toscale its virtual support engineer thatassists its technical supportteams around theworld.By leveraging its own Splunktechnology,Cisco wasabletoscale the AI assistant to support more than 1M casesandfree upengineers to concentrate on more complex cases, creating a 93+% customer satisfaction rating,and ensuringthe criticalsupport continues running in the face of any disruption. 

If you've ever opened a support case with Cisco, it's likely that the Technical Assistance Center (TAC) came to yourrescue. This around-the-clock, award-winning technical support team services online and over-the-phone support to all of Cisco's customers, partners, and distributors. In fact, it handles 1.5 million cases around the world every year.

Fast,accurate, and consistent support is critical toensuring the customersatisfaction that helpsusmaintain ourhigh standardsand grow our business. However, majorevents likecritical vulnerabilities or outagescancause spikes in the volume of casesthat slow response times and quickly swampour TAC teams, impacting customer satisfaction as a result.  In this blog,we'lldive intothe AI-powered support assistant that helps toease this issue, as well as how we usedour ownSplunk technologyto scale its caseloadand increase our digital resilience. 

Building an AI Assistant for Support

In 2022, ateam of elite TAC engineers with a passion for innovation setoutto builda solution that could accelerate issue resolution times byaugmenting an engineers' ability to detect and solve customer problems. With that, the Cisco AI Assistant for Supportwas created - it'smore than an AI bot and less than a human,designed towork alongsidethe human engineer. 

Fig. 1:All cases are analyzed and directed to the AI Assistant for Support or the human engineer based on which is most appropriate for resolution.

By directly plugging into the case routing system to analyze every case that comes in, the AI Assistant for Support evaluates which ones it can easily help solve, including license transactions and procedural problems, and responds directly to customers in their preferred language. 

With such great success, we set our eyes on even more support for our engineers and customers. While the AI Assistant for Support was originally conceived to help with the high-volume events that create a significant influx of cases, it quickly expanded to include more day-to-day customer issues,helping to reduce response times and mean time to resolution while consistently maintaining a 93+% customer satisfaction score. 

However, as the use of the AI Assistant grew, so did the complexity and volume of cases it handled. A solution that once handled 10-12 cases a day quickly ballooned into hundreds, outgrowing the methodology originally in place for monitoring workflows and sifting through log data. 

Initially, we created a methodology known as "breadcrumbs" that we tracked through a WebEx space. These "breadcrumbs," or actions taken by the AI Assistant for Support during a case from end to end, were dropped into the space so we could manually go back through the workflows to troubleshoot. When our assistant was only taking a small number cases a day, this was all we needed. 

The problem was it couldn't scale. As the assistant began taking on hundreds of cases a day, we outgrew the scale at which our "breadcrumbs" method was effective, and it was no longer feasible for us to manage as individuals. 

Identifying where, when, and why something went wrong had become a time-consuming challenge for the teams operating the assistant. We quickly realized we needed to: 

• Implement a new methodology that could scale with our operations 
• Find a solution that would provide traceability and ensure compliance

Scaling the AI Assistant for Support with Splunk 

We decided to build out a logging methodology using Splunk, where we could drop log messages into the platform and build a dashboard with case number as an index. Instead of manually sifting through our "breadcrumbs," we could instantaneously locate the cases and workflows we needed to trace the actions taken by the assistant. The troubleshooting that would have taken us hours with our original methodology could be accomplished in seconds with Splunk. 

The Splunk platform offers a robust and scalable solution for monitoring and logging that enables the capabilities required for more efficient data management and troubleshooting. Its ability to ingest large volumes of data at high rates was crucial for our operations. As an industry leader in case search indexing and data ingestion, Splunk could easily manage the increased data flow and operational demands that our previous methodology could not.   

Tangible benefits of Splunk

Splunk unlocked a level of resiliency for our AI Assistant for Support that positively impacted our engineers, customers, and business.

Fig. 2:The Splunk dashboard offers clear visibility into functions to ensure optimized performance and stability. 

With Splunk, we now have: 

• Scalability and efficiency:Splunk monitors the assistant's activities to ensure it's working correctly and provides the ability for TAC engineers to monitor and troubleshoot workflows, allowing the assistant to efficiently scale. The AI Assistant for Support has successfully worked on over one million cases to date. 
• Enhanced visibility:With dashboards that allow for quick access to case histories and workflow logs of our assistant, the TAC engineers overseeing the processes save time on case reviews to deliver faster than ever customer support. 
• Optimized processes with real-time metrics:The visibility into resource allocation allows us to optimize our business processes and workflows, as well as demonstrate the value of our solution with real-time metrics. 
• Proactive monitoring:Splunk ensures all APIs are fully functioning and monitors logs to alert us of potential issues that could impact our AI Assistant's ability to operate, allowing for quick remediation before customer experience is impacted. 
• Higher employee and customer satisfaction:Engineers are equipped to handle higher caseloads and efficiently reprioritize efforts, reducing burnout while optimizing customer experience. 
• Reduced complexity:The dashboards have a simple interface, making it much easier to train and onboard new employees. The ease of use also serves to improve the capabilities of the humans operating our AI Assistant by enhancing their accuracy and efficiency. 

By providing a scalable and traceable solution that helps us stay compliant, Splunk has enabled us to maintain our commitment to exceptional customer service through our AI Assistant for Support.

Additional Resources:

•  Experience Seamless Support: Unlocking Value with AI-enhanced TAC  
• Scaling the Cisco AI Assistant for Support with Splunk Case Study
• Cisco on Cisco 
• Driving IT efficiency and agility with proactive support from Cisco TAC 

PS:  Attending Cisco Live in San Diego this June? 

You'll have a special opportunityto talk live with Cisco IT experts to dive into these success stories and other deployments!Look for Cisco on Cisco in each of theshowcases and be sure tosearch Cisco on Cisco in the session catalogto add our sessions to your schedule!