Three two times numbers deceived me

​

​

March 2023

Vivek Pandey

Outline

• Programming and Thinking
• How numbers deceived me
• Simpson's paradox
• We saw this earlier
• AFS timeouts
• AFS = Anti Fraud Service
• Credit line delinquencies

Programming vs deducing

• We write programs
• Programs are often buggy
• We do testing and we correct it
• Computer is always right
• It always correctly executes your programs
• We also think, and make deductions from numbers
• Math is always right
• What if our thinking is buggy?
• What if our mental model is buggy?
• There is no testing tool available
• Need to be skeptical

Bugs in thinking

• Our mental model may not correspond to reality
• And so we can draw incorrect inferences from the numbers
• Need to be alert to correct mental model

Case Study 1: Simpson's paradox

• We discussed this in a previous tech talk

• Treatment A is better on small stones
• Treatment A is better on large stones
• But overall, treatment B is better

Case Study 2: AFS Timeouts

• First we need to understand relevant Simpl architecture

TS

Merchant Server

User clicks "Pay Using Simpl" option

charge call

AFS (Anti Fraud Service)

"AFS call"

100ms SLA

• On a charge call, TS calls AFS:
• If AFS returns "accept", TS commits the transaction to the ledger
• If AFS returns "deny", TS fails the charge call
• End user will need to pay by some other means
• When AFS denies the transaction, it often also "blocks" the user
• Next time, the user won't even see Simpl button
• This blocking is async, via celery...takes a few seconds to be operational
• AFS has SLA of 100ms
• If TS does not receive response in 100ms, it assumes "accept"
• TS can't wait for ever (end user is waiting)
• Most transactions are not fraudulent: so we can't have "deny" as default reply
• Will lead to big TPV loss if AFS has a problem

We thought AFS is performing well

• We kept track of AFS SLA
• AFS timeout was <1%
• Average AFS response time: ~10-20ms
• p99 AFS response time: ~80ms
• AFS was doing well
• So we though
• Till one day...

We had some frauds going on in Zomato...

• Fraud monitoring team confirmed that a group of phone numbers belonged to fraudsters
• Fraudsters had done some transactions on zomato
• There was a very similar pattern of activity on all phone numbers, and they were clear fraudsters
• We had some filters made specifically for those patterns
• Why did those filters not block the user?
• Following pattern was observed on all phone numbers

We had some frauds going on in Zomato...

• Fraud monitoring team confirmed that a group of phone numbers belonged to fraudsters
• Fraudsters had done some transactions on zomato
• There was a very similar pattern of activity on all phone numbers, and they were clear fraudsters
• We had some filters made specifically for those patterns
• Why did those filters not block the user?
• Following pattern was observed on all phone numbers

we are blocking the user so why are we not denying charge call?

Only explanation is timeout, but timeout is low

Findings: 1/2 - When does a timeout hurt?

• AFS timeout hurts only if response is "deny"
• If response is "accept", and AFS takes 110ms, it is ok
• TS will anyway assume a default "accept" response after 100ms timeout
• But if response is "deny", and you timeout then TS default of "accept" is wrong and you have failed to stop a fraud

Findings: 2/2 - Detailed stats

• AFS gets 200,000 charge calls in a day
• 195,000 "accept" responses
• 5,000 "deny" responses
• "Only" 2000 timeouts out of 200,000
• But all these timeouts were in "deny" responses
• So, out of 5000 frauds, you failed to stop 2000 of them
• True time out: 2000/5000 = 40%