AI Recommendation Engine Testing - OTT

Case Study
AI Recommendation Engine Testing

Client

A large retail OTT Media house, embarked on a quality assurance engagement to ensure the accuracy and speed of an AI platform that recommended media choices for targeted end users. IGS was invited to test the AI driven recommendation engine.

Problem Statement

Test the AI platform for:

  • Enhancing recommendation accuracy and speed
  • AI Engine to handle complex scenarios
  • Testing Decision-Making Capability

Solution

1

Trained AI Engine with variety of situations, including content types, user preferences, and environmental conditions (synthetic data + real-world data)

2

Evaluated recommendation system's response time under various loads, latency testing to measure the time to generate recommendations

3

Testing how system performs under limited or no user history available, ability to recommend less popular or niche contents to users or adapt to sudden changes to end user's interests/preferences

4

Test recommendation system across different platforms (web, mobile, etc.) to ensure consistent performance and behavior

5

Developed test scripts to automate testing process for various scenarios identified earlier along with integrating automated tests into CI/CD pipeline to ensure tests run automatically whenever changes are made to the system

Outcome

The accuracy rate of media recommendation accuracy improved by 15%

Recommendation speed improved by 30%, reducing the average response time from 5 seconds to 3.5 seconds

Test execution time was reduced by 50%

The automated test suite achieved coverage of 90% of the AI engine's functionalities