Analytical report · 2026-10-11
August 2023 transaction success rate: where the problem is and what to fix first
Analysis of 32,122 transactions, July–August 2023 · E-Wallet
The success rate fell -4.7pp, mostly in Billing: Billing accounts for 64.5% of the drop (86.3% → 71.8%) in the same month Billing volume rose +50.0%. Failures rose on every platform, every payment method and every error type, so September's priority is the processing capacity of the Billing flow, starting with authentication, the error type behind 54.0% of the additional failures.
SituationThe e-wallet processed 14,662 transactions in July and 17,460 in August 2023 across Android, iOS and Web; the July success rate was 91.7%.
ComplicationIn August 2023 the success rate fell to 86.9% (-4.7pp), about 828 more failed transactions than if the July rate had held.
QuestionWhere should September's engineering capacity go to bring the success rate back to the July level?
91.7%
Transaction success rate 7
86.9%
Transaction success rate 8
-4.7pp
Change in Transaction success rate 7→8
828
Excess failed cases vs the 7 rate
+19.1%
% change in Transactions
+18.9%
% change in Total payment value (USD)
Key arguments
- Billing accounts for 64.5% of the wallet-wide drop: Billing's success rate fell -14.5pp while all other categories fell only -2.0pp, just as Billing transactions rose +50.0%.
- Every error type rose within Billing; authentication is the largest (192 → 544 cases, 54.0% of the additional failures), the other types rose 3.5x.
- Three platforms and four payment methods dropped by about the same amount (spreads of 3.2pp and 3.8pp), so this is neither an app bug nor the fault of one payment partner.
Where
The transaction success rate fell from 91.7% to 86.9% (-4.7pp) in August, about 828 excess failed transactions
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 04
Where
Billing accounts for 64.5% of the drop: its success rate fell from 86.3% to 71.8%, all other categories only -2.0pp
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 05
Where
Billing transactions rose from 2,280 to 3,420 (+50.0%), 40.7% of the wallet's total volume increase
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 06
What broke
Every error type in Billing rose; authentication errors are the largest, from 192 to 544 cases, 54.0% of the additional failures
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 07
What broke
Within Billing, Electricity (-18.1pp) and Internet (-17.7pp) fell more than Water (-11.2pp)
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 08
Ruled out
All three platforms in Billing fell by about the same amount, from -13.2pp to -16.4pp: not an app-specific bug
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 09
Ruled out
The four main payment methods in Billing all fell, from -11.9pp to -15.7pp: the fault is not one bank or card partner
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. · Exhibit 10
Hypothesis testing
5 hypotheses supported by the data, 1 rejected, 1 not testable with the current data
| ID | Hypothesis | ICE | Result | Evidence |
|---|---|---|---|---|
| H1 | The drop is concentrated in one category rather than spread evenly. | 630 | Supported | category = Billing: 86.3% → 71.8% (-14.5pp), 64.5% of the total change of -4.7pp; all other category values -2.0pp. |
| H2 | Within the most affected category, authentication errors (OTP/PIN) make up the majority of August failures. | 432 | Supported | Of 964 failed cases in the comparison period (category = Billing), error_category_l2 = AUTHENTICATION_ISSUES accounts for 56.4% (base period 61.5%, cases up 2.8x). As % of all rows: AUTHENTICATION_ISSUES 8.4% → 15.9%; all other types 5.3% → 12.3%. |
| H3 | Within the most affected category, every platform dropped about the same, so the fault is server-side rather than an app bug. | 420 | Supported | By platform (category = Billing): android -13.2pp, ios -16.4pp, web -13.8pp. Spread across groups 3.2pp vs a typical change of 13.8pp → the change is uniform. |
| H4 | Within the most affected category, every payment method dropped about the same, so it is not the fault of one bank or card partner. | 300 | Supported | By payment_method (category = Billing): credit_card -11.9pp, debit_card -15.3pp, linked_bank -14.9pp, wallet_balance -15.7pp. Spread across groups 3.8pp vs a typical change of 15.1pp → the change is uniform. |
| H5 | Transaction volume in the most affected category rose sharply in August. | 300 | Supported | category = Billing: 2,280 → 3,420 (+1,140), 40.7% of the total change of +2,798; all other category values +1,658. |
| H6 | Inside the most affected category, the drop is spread evenly across bill sub-categories. | 250 | Rejected | By sub_category (category = Billing): Apartment Fee -13.2pp, Consumer Finance -11.4pp, Electricity -18.1pp, Insurance -17.2pp, Internet -17.7pp, TV -12.5pp, Water -11.2pp. Spread across groups 6.9pp vs a typical change of 13.2pp → groups differ clearly. |
| H7 | Users drop off between authentication and payment confirmation and retry several times. | Not testable | Needs event_logs (funnel and path); the transaction file has no step-level data. |
Recommendations
September priorities: stabilise the Billing flow within 2 weeks, then test an alternative authentication method
| # | Action | Owner | Timeline | KPI | Type |
|---|---|---|---|---|---|
| 1 | Monitor the Billing success rate hourly and alert when it drops below the July level (86.3%) Catch overload hours early instead of waiting for the monthly report | Head of Payments | Within 1 week | Hourly Billing success rate | Quick win |
| 2 | Load-test the OTP gateway and the Billing backend at 3,420 transactions/month; raise the quota or add a backup SMS provider Restoring Billing to 86.3% at August volume would lift the whole wallet by about 2.8pp | Engineering Lead | Within 2 weeks | Billing failures per month (August: 964) | Quick win |
| 3 | Review the connections to the electricity and internet providers, the two bill types with the largest drops Close the 6.9pp gap between bill types | Partnerships Manager | Within 2 weeks | Billing success rate by bill type | Quick win |
| 4 | A/B test biometric authentication instead of OTP for bill payments (design from baseline, MDE, α 0.05, power 80%) Target: bring Billing authentication failures from 15.9% back to 8.4% as in July | Payments Product Manager | Design in September, run for 3–4 weeks | Authentication failure rate on Billing transactions | Big project |
| 5 | Add event logging to the bill-payment flow to build the funnel and paths: where users drop and whether they retry | Analytics Lead | Within 4 weeks | Billing funnel covering every step from bill selection to confirmation | Big project |
Limitations
What the current data cannot answer
- The transaction file has no event log, so step-by-step funnel and path analysis cannot be built (needs event_logs.csv).
- 734 failed transactions have no error category; error-type shares are computed on labelled cases.
- There is no experiment data, so the alternative authentication fix is a proposal that needs an A/B test.
- Only two periods (July and August) exist, so seasonality cannot be separated.
Next steps:
- Analyse event_logs.csv to build the funnel and post-authentication-failure paths (hypothesis H7 not yet testable).
- Match hourly infrastructure logs (OTP gateway, Billing backend) against transaction volume to test the overload hypothesis.
- Re-label the failed transactions that are missing an error category.
- Find out why categories outside Billing also fell -2.0pp.
Appendix
Method and source figures
Source: transactions_enhanced.csv (32,122 rows after filters); base period 7/2023, comparison period 8/2023. Issue tree structure: segmentation. ME: Each transaction belongs to exactly one category, one platform and one payment method; shift-share splits Δ into rate effect + mix effect that add up exactly to the total Δ. CE: The total Δ equals the sum of the contributions of every category (including Not Payment), so nothing sits outside the tree. The branch with no column (funnel steps) is recorded in boundaries.
Facts cited in this report (59)
| Fact id | Value | Description |
|---|---|---|
H1.Billing.base | 86.3% | Transaction success rate, category=Billing, base period |
H1.Billing.comp | 71.8% | Transaction success rate, category=Billing, comparison period |
H1.Billing.delta | -14.5pp | Change in Transaction success rate, category=Billing |
H1.Billing.rate_effect | -2.8pp | Part of the total change due to Billing performance (rate effect) |
H1.Billing.share_of_delta | 64.5% | % of the total change from Billing |
H1.mix_total | -0.9pp | Part of the change due to category mix (mix effect) |
H1.rate_total | -3.9pp | Part of the change due to performance within each category (rate effect) |
H1.rest.delta | -2.0pp | Change in Transaction success rate for all other category values |
H1.top_vs_rest | 7.1x | How many times the change in Billing exceeds that of the rest |
H1.total.base | 91.7% | Transaction success rate, base period (all rows) |
H1.total.comp | 86.9% | Transaction success rate, comparison period (all rows) |
H1.total.delta | -4.7pp | Change in Transaction success rate (all rows) |
H2.AUTHENTICATION_ISSUES.count_comp | 544 | Rows with error_category_l2=AUTHENTICATION_ISSUES, comparison period |
H2.AUTHENTICATION_ISSUES.multiplier | 2.8x | error_category_l2=AUTHENTICATION_ISSUES: comparison period as a multiple of base period |
H2.AUTHENTICATION_ISSUES.rate_base | 8.4% | error_category_l2=AUTHENTICATION_ISSUES as % of all rows (category = Billing), base period |
H2.AUTHENTICATION_ISSUES.rate_comp | 15.9% | error_category_l2=AUTHENTICATION_ISSUES as % of all rows (category = Billing), comparison period |
H2.AUTHENTICATION_ISSUES.share_base | 61.5% | Share of error_category_l2=AUTHENTICATION_ISSUES, base period |
H2.AUTHENTICATION_ISSUES.share_comp | 56.4% | Share of error_category_l2=AUTHENTICATION_ISSUES, comparison period |
H2.AUTHENTICATION_ISSUES.share_of_increase | 54.0% | % of the increase in cases from error_category_l2=AUTHENTICATION_ISSUES |
H2.others.multiplier | 3.5x | Cases in other error_category_l2 values: comparison period as a multiple of base |
H2.others.rate_base | 5.3% | Other error_category_l2 values as % of all rows, base period |
H2.others.rate_comp | 12.3% | Other error_category_l2 values as % of all rows, comparison period |
H2.rows.base | 312 | Outcome rows, base period (category = Billing) |
H2.rows.comp | 964 | Outcome rows, comparison period (category = Billing) |
H3.android.delta | -13.2pp | Change in Transaction success rate, platform=android |
H3.ios.delta | -16.4pp | Change in Transaction success rate, platform=ios |
H3.spread | 3.2pp | Max−min spread of the change across platform |
H3.typical_change | 13.8pp | Typical size of the change (median |Δ|) across platform |
H3.web.delta | -13.8pp | Change in Transaction success rate, platform=web |
H4.credit_card.delta | -11.9pp | Change in Transaction success rate, payment_method=credit_card |
H4.debit_card.delta | -15.3pp | Change in Transaction success rate, payment_method=debit_card |
H4.linked_bank.delta | -14.9pp | Change in Transaction success rate, payment_method=linked_bank |
H4.local_card.delta | -18.7pp | Change in Transaction success rate, payment_method=local_card |
H4.spread | 3.8pp | Max−min spread of the change across payment_method |
H4.typical_change | 15.1pp | Typical size of the change (median |Δ|) across payment_method |
H4.wallet_balance.delta | -15.7pp | Change in Transaction success rate, payment_method=wallet_balance |
H5.Billing.base | 2,280 | Transactions, category=Billing, base period |
H5.Billing.comp | 3,420 | Transactions, category=Billing, comparison period |
H5.Billing.delta | +1,140 | Change in Transactions, category=Billing |
H5.Billing.rel_change | +50.0% | % change in Transactions for Billing |
H5.Billing.share_of_delta | 40.7% | % of the total change from Billing |
H5.rest.delta | +1,658 | Change in Transactions for all other category values |
H5.total.base | 14.7K | Transactions, base period (all rows) |
H5.total.comp | 17.5K | Transactions, comparison period (all rows) |
H5.total.delta | +2,798 | Change in Transactions (all rows) |
H6.Apartment_Fee.delta | -13.2pp | Change in Transaction success rate, sub_category=Apartment Fee |
H6.Consumer_Finance.delta | -11.4pp | Change in Transaction success rate, sub_category=Consumer Finance |
H6.Electricity.delta | -18.1pp | Change in Transaction success rate, sub_category=Electricity |
H6.Insurance.delta | -17.2pp | Change in Transaction success rate, sub_category=Insurance |
H6.Internet.delta | -17.7pp | Change in Transaction success rate, sub_category=Internet |
H6.TV.delta | -12.5pp | Change in Transaction success rate, sub_category=TV |
H6.Water.delta | -11.2pp | Change in Transaction success rate, sub_category=Water |
H6.spread | 6.9pp | Max−min spread of the change across sub_category |
H6.typical_change | 13.2pp | Typical size of the change (median |Δ|) across sub_category |
kpi.base | 91.7% | Transaction success rate 7 |
kpi.comp | 86.9% | Transaction success rate 8 |
kpi.delta | -4.7pp | Change in Transaction success rate 7→8 |
negatives.comp | 2,281 | Failed cases in period 8 |
negatives.excess_comp | 828 | Excess failed cases vs the 7 rate |