MAYA REVIEW AGENT

Overview

Maya (formerly PayMaya) is one of the Philippines' leading digital banks — payments, transfers, savings, and crypto for millions of users. Despite strong product fundamentals, its app store ratings had been sliding, with a mixed-negative sentiment across 12,345 reviews between August 2025 and February 2026. Negative reviews contain highly specific, screen-level signal about what's actually broken. But at scale, no human team can read, tag, and rank 12,000+ reviews fast enough to act on them. This project asked a different question: what if the reviews could read themselves? Maya Review Agent is an AI pipeline that scrapes app store reviews, maps pain points to specific screens, benchmarks against the main competitor (GCash), and outputs a prioritized list of judgments — with real user quotes as evidence — in a single command.

My Role

Solo, two-week sprint. I owned the full loop — framing the research question, engineering the pipeline, and interpreting the output into ranked, defensible judgments about where Maya was losing users. Responsibilities: • Problem framing and data strategy • 8-step Python pipeline architecture • LLM prompt engineering for review classification • Dual-platform scraping (Google Play + iOS) for Maya and GCash • Fuzzy matching and version-level spike detection • Screen-level friction mapping • Case study synthesis

The Problem

Maya is losing users — and the reviews tell you exactly why. > "I've tried several times today, but I can't get in." > — Maya user, 1-star review > "My phone was stolen and the hacker was able to access this app and took my savings and made a loan in a snap of a finger." > — Maya user, 1-star review These aren't edge cases. Login failures alone carry a severity score of 5/5 with high frequency — impacting activation, retention, trust, and revenue simultaneously. The core challenge was surfacing signal at scale: a research team would need weeks to read and categorize 12,000+ reviews. By the time the findings landed, the next release would have shipped new problems on top of the old ones.

The Hypothesis

What if an AI agent could read every single review, map each complaint to a specific screen, and rank the findings by severity, frequency, and business impact — in minutes instead of weeks? Instead of manually tagging reviews in spreadsheets, the pipeline: 1. Scrapes reviews at scale (both Maya and competitor GCash) 2. Classifies sentiment and issue type using LLMs 3. Maps problems to specific screens and user journeys 4. Ranks issues by a composite severity × frequency × business-impact score 5. Surfaces evidence — real user quotes attached to every judgment

The 8-Step Pipeline

An 8-step Python pipeline powered by Gemini 2.0 Flash. Each step caches to JSON — interrupted runs resume automatically, and individual steps can be re-run without repeating the whole pipeline. 1. Scrape — 10,000+ Maya + 10,000+ GCash reviews from Google Play & iOS 2. Temporal Analyzer — month-over-month issue evolution (persistent vs. emerging vs. resolved) 3. Comprehensive Analyzer — deep categorization by issue type, journey, and severity 4. Funnel Analyzer — maps drop-off points to specific screens 5. Accessibility Analyzer — edge-case users (elderly, OFWs, budget devices, slow connectivity) 6. Competitor Analysis — side-by-side Maya vs. GCash benchmark 7. Screenshot Scraper — pulls current app screenshots for visual reference 8. Insights Generator — synthesizes everything into a ranked report with charts

Key Design Decisions

• Gemini 2.0 Flash over GPT-4 — faster and cheaper for high-volume text classification. 12K reviews processed in minutes, not hours. • Dual-platform scraping — Google Play + iOS gives a complete picture. iOS reviews often surface different issues (premium user frustrations, payment edge cases). • Competitor scraping — GCash reviews provide benchmark context. Users explicitly compare the two apps, and those comparisons are market intelligence, not just feedback. • Regex pre-filtering — Steps 4–5 use regex to tag signal-rich reviews before sending to the LLM, reducing API calls by ~80%. • Chunked analysis — Step 3 splits reviews into 200-review chunks for deeper attention, then synthesizes across chunks. • Checkpoint caching — every step writes to JSON. Re-runs skip completed steps automatically.

Ranked UX Opportunities

Top UX issues by priority score — login failures, security, and unauthorized transactions dominate

The agent identified and ranked 9 critical issues by a composite priority score (severity × frequency × business impact). 4 of the top 5 issues are severity 5/5 — and every one of them is trust-related. Maya doesn't have a feature problem. Maya has a trust crisis.

Funnel Drop-off Hotspots

Top drop-off hotspots colored by severity — the OTP Entry screen is both high-frequency and critical

Beyond ranking the issues, the agent mapped each pain point to the specific screen where it happens. This is the difference between "users have login issues" and "45 reviews specifically mention the Login > OTP Entry screen." Top drop-off screens: • Login > OTP Entry — 45 reviews, severity 5/5 • Cash In > Bank Integration — 30 reviews, severity 4/5 • KYC > Selfie Capture — 25 reviews, severity 4/5 • Crypto > Buy Token — 12 reviews, severity 3/5 • Send Money > Contact Selection — 8 reviews, severity 2/5

Version Spike Detection

Rating trend over 7 months — the dip starting October 2025 aligns with the v2.148.3 release

The temporal analyzer flagged a critical release: v2.148.3. • 45.2% one-star rate vs. 34.6% baseline (+10.6 percentage points) • 188 reviews flagged between November 2025 – February 2026 • Root causes: login regressions, payment failures, chatbot-only support replacing human agents This single release caused measurable trust erosion — the kind of finding that's invisible in aggregate dashboards but obvious once you align reviews against version numbers.

What the Agent Flagged to Fix First

The agent doesn't produce designs — it produces a defensible priority order. The top four actions it surfaced: 1. Rebuild login around biometrics — OTP entry is the single highest-evidence drop-off screen (45 reviews, severity 5/5). The reliable path should be the default, not the fallback. 2. Make cash-in status visible in real time — the complaint isn't that transactions take time; it's that users don't know what's happening while they wait. Every pending state needs a visible marker. 3. Put human support one tap away — the current Help Center optimizes for deflection. The review data shows users escalate anyway, just with a trust cost. Surface the escalation path instead of hiding it. 4. Ship a Security Center with a kill switch — the "phone was stolen" reviews are the most damaging kind. Freezing everything should be one screen, not a customer-service call. These aren't design outputs — they're the ranked conclusions the pipeline drew from the evidence. Turning them into screens is a separate project.

Competitive Position vs. GCash

Users actively switch to GCash citing: • Stronger security measures • Better data privacy practices • Faster transaction processing • Functioning customer support Maya's advantages: interface design preference, feature breadth, savings interest rates, crypto access. The takeaway is counterintuitive — Maya wins on product, loses on trust. The redesign priorities flow directly from that asymmetry: fix trust first, feature parity second.

What I Learned

On building AI agents for design research: 1. Reviews are underrated as a data source. 12,000 reviews contain more specific, screen-level UX feedback than most user interview programs. The signal-to-noise ratio on 1-star reviews is unreasonably high. 2. LLMs excel at categorization at scale. What would take a research team weeks — reading, tagging, categorizing 12K reviews — takes the agent minutes. Gemini 2.0 Flash was good enough at classification that human spot-checks rarely found disagreements. 3. Temporal analysis reveals what dashboards miss. Knowing an issue is "high severity" is useful. Knowing it's been high severity for 7 months and getting worse is actionable. 4. Competitor context changes the framing. When users say "I switched to GCash because…" — that's not just feedback, it's market intelligence. Scraping competitor reviews in parallel was one of the best decisions I made.

What I'd Build Next

• Monthly scheduled runs — track whether the redesign recommendations actually move the metrics • Tagalog/Filipino language support in the review scraper — a significant portion of 1-star reviews are in Filipino and currently get under-weighted by English-biased classification • Budget-device testing — pair the review analysis with device fingerprints to see whether issues cluster by hardware class • A live dashboard instead of static reports — the current output is a Markdown report. A dashboard would make monthly re-runs more useful to the design team

Reflection

This project reframed how I think about evidence at scale. App store reviews used to feel like noise — a few gems buried in thousands of complaints. The agent showed me the opposite: at 12,000+ reviews, the noise cancels and the signal is obvious. Login failures aren't a hunch anymore; they're Priority Score 18 with 45 screen-specific citations. The bigger shift is what LLM pipelines let one person do in a week. Reading and classifying 12K reviews used to require a research team. Now it requires a Python script, an API key, and the judgment to ask the right questions. The interesting work moves up the stack — from "can we process this data?" to "what should we conclude from it?"