case study
Delivering Quality for an AI-Driven Mobile Chatbot in Banking
Role
Test Consultant
Client Type
Banking
Introduction
Our client, a recognized innovation leader in the banking sector, set out to improve their customer experience by introducing an AI-driven, LLM-based mobile chatbot.
Primary Business Objectives
- Increase operational efficiency.
- Significantly reduce customer support costs.
- Seamlessly transfer web and call traffic to the mobile application.
- Drive higher engagement among the younger generation.
The Challenge
While the business goals were clear, the execution faced significant operational and
technical bottlenecks:
- Capacity Constraints: The application was being tested manually, which severely drained the limited capacity of the business team.
- Lack of AI-Specific Methodology: Traditional testing processes failed when applied to generative AI. There was no clear methodology to evaluate the probabilistic, non-deterministic answers generated by the LLM.
- The Mandate: Krone Consulting was brought in to lead the validation, engineer a robust testing methodology for probabilistic AI, and increase efficiency across the entire delivery pipeline.
The Solution
To address the unique challenges of generative AI, we shifted from manual validation to an AI-augmented testing framework.
Key Pillars of Implementation:
- LLM-Based Evaluation: We implemented an innovative “LLM-as-a-judge” methodology, where an independent LLM assessed the correctness, safety, and relevance of the chatbot’s answers.
- Automated Test Cycle Workflow: Deployed a streamlined workflow that automated result evaluation and provided real-time, transparent reporting for stakeholders.
- Data-Driven Governance: Established measurable acceptance criteria based on MAIN Next® principles, enabling management to make objective, data-backed release decisions.
The Results
By implementing this modern, AI-driven strategy, we eliminated the testing bottleneck and accelerated the product’s time-to-market.
Outcome: The product was successfully shipped, meeting all quality benchmarks and significantly reduced the operational burden on the client’s internal teams.