Architecting a Modern Financial Institution SOUTHEAST BRAZIL REGION FROM SPACE
CREDIT CARD September 2014
GROWING QUICKLY IN A COMPLEX DOMAIN # of clients (M) 10.5 M 20 Credit Card 3,0 Unique applications Deploys per day 2.6 M 120 2,3 Customers Microservices 1,5 262 M 105 Purchases Engineers 0,8 198 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Countries
IMMUTABLE THEMES FROM OUR STACK LISP hosted on the JVM CLOJURE Functional (opinionated), immutable data structures DATOMIC Simple, concise, fast, concurrent Tight REPL feedback cycles KAFKA Gradual typing (schemas) CLOUD
IMMUTABLE THEMES FROM OUR STACK CLOJURE for your data Accumulate-only DATOMIC Reified ACID transactions, preserve what changed when KAFKA Query using data structures (Datalog) Cloud native with integrated caching and scalable reads CLOUD
IMMUTABLE THEMES FROM OUR STACK Immutable, persistent, partitioned log CLOJURE Logical decoupling between services Temporal decoupling, useful for DATOMIC asymmetric workloads Fault isolation and recovery (circuit breakers, dead letters) KAFKA Financial batch jobs expressed as a streams of messages CLOUD
IMMUTABLE THEMES FROM OUR STACK Infra as code (AWS) CLOJURE Immutable upon provisioning (Docker) Blue-Green deploys at service and DATOMIC company level Kubernetes for speed and scalability KAFKA CLOUD
Nubank HQ São Paulo, Brazil FUNCTIONAL BENEFITS HIRING POSITIVE SELF SELECTION 1-MONTH RAMP COMPLEXITY SMALL, PURE FUNCTIONS STRAIGHTFORWARD TO UNTANGLE CONSISTENCY COMPOSING A SMALL NUMBER OF IDIOMATIC LANGUAGE FEATURES
CREDIT CARD ARCHITECTURE Greenfield MVP Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Backoffice General Ledger Credit Limits (CRM) Collections Securitization Phone + Chat ETL
BANK ACCOUNT October 2017 13
CORE BANKING + CREDIT CARD ARCHITECTURE Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE
PURCHASE AUTHORIZATION Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE
PURCHASE AUTHORIZATION VALUE CHAIN ACQUIRER NETWORK MERCHANT CUSTOMER ISSUER
ISSUER AUTHORIZATION NETWORK ISSUER
ISSUER AUTHORIZATION MASTERCARD INTERFACE DEVICE Establish a connection 1 Receive authorization requests 2 AUTHORIZER
ISSUER AUTHORIZATION: ISO-8583 MASTERCARD INTERFACE DEVICE ISO-8583 Binary Message AUTHORIZER HARDWARE SECURITY MODULE
SCODEC BINARY PARSER FOR ISO-8583 object PANMappingFileD { import scala.language.reflectiveCalls val codec: Codec[SE33Subfield] = discriminated[SE33Subfield].by(intPadded(2)) .typecase(1, llvar(str).as[AccountNumberIndicator]) .typecase(2, llvar(intString(intPadded(2))).as[AccountNumber]) .typecase(3, llvar(yearMonth).as[ExpirationDate]) .typecase(4, llvar(str).as[ProductCode]) .typecase(5, llvar(intPadded(2)).as[TokenAssuranceLevel]) .typecase(6, llvar(intString(intPadded(2))).as[TokenRequestorID]) .typecase(7, llvar(intString(intPadded(2))).as[PANAccountRange]) }
ISSUER AUTHORIZATION: REQUIREMENTS BRAND INTERFACE DEVICE ISO-8583 Binary Message AUTHORIZER 1.Highly Available 2.Physical Infrastructure HARDWARE SECURITY MODULE (HSM)
AUTHORIZER SERVICE LAYOUT • Small set of highly available services ISO 8583 • Co-located with the MasterCard Finagle Client devices in the same datacenters router router Thrift • Isolated: transaction authorization hot path does not Finagle Server need communication with the authorizer authorizer authorizer authorizer cloud • Active-active disaster recovery crypto crypto fraud fraud (not shown) Proprietary protocol HSM HSM
KAFKA AS THE BRIDGE BETWEEN ENVIRONMENTS 100+ microservices kafka “neverland” “the real world” (nubank datacenter) (AWS VPC)
KAFKA-BASED LOG/SNAPSHOT Publish 1 Authorizer consumes 2 AWS Service 0 1 2 3 4 … Kafka Topic Partition Snapshot 3 service New consumes 5 authorizer started snapshotter Consumes from 7 snapshot offset Generates a 4 snapshot Reads offset 6 Snapshot 2
DRAMATIC IMPROVEMENTS IN RELIABILITY AND FRAUD % stand-in cutover cutover 80 % 1,4 % % fraud capture 60 % 1,05 % 40 % 0,7 % 20 % 0,35 % % fraud precision 0 % 0 % 2017-03 2017-04 2017-05 2017-06 2017-07 2017-08 2017-09 2016-07 2016-09 2016-11 2017-01 2017-03 2017-05 2017-07
DOUBLE ENTRY ACCOUNTING Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE
BUSINESS LOGIC DEPENDS ON DATA ACROSS MANY SERVICES Purchases Payments Chargebacks Interest Currencies Should we… authorize a purchase? Double Entry block a card? charge interest?
DOUBLE ENTRY: THE MODEL CREDIT ENTRY BOOK ACCOUNT $ DEBIT BOOK ACCOUNT = 𝚻 BALANCE $ The sum of all credits and debits for one book-account is its balance A customer’s balance sheet is a cumulative function of their entire history
DOUBLE ENTRY: THE RULEBOOK CREDIT ENTRY BOOK ACCOUNT $ DEBIT BOOK ACCOUNT CREDIT ENTRY 2 BOOK ACCOUNT NEW-PURCHASE $ NEW-PAYMENT DEBIT … BOOK ACCOUNT CREDIT ENTRY 3 BOOK ACCOUNT $ DEBIT BOOK ACCOUNT MOVEMENT
DOUBLE ENTRY: EXAMPLE MOVEMENT ( def unsettled-purchase [ { :entry/debit-account :book-account-type.asset/unsettled :entry/credit-account :book-account-type.liability/unsettled-counterparty :entry/amount #'transaction-amount :entry/post-date #'produced-date} { :entry/debit-account :book-account-type.liability/current-limit-counterparty :entry/credit-account :book-account-type.asset/current-limit :entry/amount #'transaction-amount :entry/post-date #’produced-date} ])
DOUBLE ENTRY: CHALLENGES ordering matters (i.e. movements are not commutative) late arriving events (e.g. a payment was made 3 days ago ) fixing invariants write throughput
DOUBLE ENTRY: GENERATIVE TESTING OF INVARIANT ( def loss-property ( prop/for-all [adjs ( gen/vector ( gen/one-of [gen-adjustment gen-payment gen-tx]) 1 10) initial-state ( gen/such-that ( comp not #{ :late :pre-loss } :state ) rbh/initial-state-gen) loss-event ( gen/tuple ( gen/no-shrink ( gen/elements #{ :pre-loss :credit-loss :id-fraud-loss :fraudster })) ( tg/make-generator LocalDateTime) ( tg/make-generator LocalDate))] ( check-properties adjs initial-state loss-event)))
DOUBLE ENTRY: CHALLENGES ordering actually matters (i.e. movements are not commutative) late arriving events (e.g. a payment was made 3 days ago ) fixing invariants write throughput
SHARDED, FAULT TOLERANT INFRASTRUCTURE Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE
SCALING BOTTLENECKS # of clients (M) Credit Card 3,0 1. database throughput limits required throttling writes 2,3 2. batch job latency impacting customer experience 1,5 0,8 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17
SCALING PLAN Need to partition the workload Customer data is spread across services Interactions between customers are minimal Safe to partition the user base
OPTION #1: PARTITION SERVICE DATABASES Database writes were the worst bottleneck Option: horizontally partition each database db shard s0 db shard s1 db shard s2 Change every service to route queries and writes to the appropriate shard backend service
OPTION #1: PROBLEMS Enormous effort to change every service Doesn’t address non-db bottlenecks Risks intermingling data infrastructure code with business logic
OPTION #2: SCALABILITY UNITS SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 3 SERVICE 3 SERVICE 3 . . . shard S0 shard S1 shard s2
OPTION #2: SCALABILITY UNITS + GLOBAL ROUTING global purchase deposit SERVICE 4 SERVICE 5 SERVICE 6 SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 3 SERVICE 3 SERVICE 3 shard S0 shard S1 shard s2
Recommend
More recommend
