BACKGROUND JOB PROCESSING DO'S AND DON'TS BACKGROUND JOB - - PowerPoint PPT Presentation

▶
background job processing
SMART_READER_LITE
LIVE PREVIEW

BACKGROUND JOB PROCESSING DO'S AND DON'TS BACKGROUND JOB - - PowerPoint PPT Presentation

Oct 19, 2023 455 likes •823 views

CHUCK REEVES @MANCHUCK BACKGROUND JOB PROCESSING DO'S AND DON'TS BACKGROUND JOB PROCESSING - DO'S AND DON'TS IMAGE PROCESSOR UPLOAD IMAGE PROCESS IMAGE ENTERS QUEUE ZendCon 2016 BACKGROUND JOB PROCESSING - DO'S AND DON'TS THE PROBLEM

slide-1
SLIDE 1

BACKGROUND JOB PROCESSING

CHUCK REEVES @MANCHUCK

DO'S AND DON'TS

slide-2
SLIDE 2

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

ZendCon 2016

UPLOAD IMAGE IMAGE PROCESSOR ENTERS QUEUE PROCESS IMAGE

slide-3
SLIDE 3 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

THE PROBLEM CHALLENGE

▸ Crawl Google Merchant Center ▸ Crawl Google Ad Words ▸ Crawl Google Analytics ▸ Match Data ▸ Analyze Data ▸ Make Decisions ▸ POST/PUT Google Ad Words ▸ Within 4 hours

slide-4
SLIDE 4

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

ZendCon 2016

TIME OF DAY CATALOGUE CRAWLER ENTERS QUEUE CRAWLS CATALOGUE BATCHES CATALOGUE BATCH PROCESSOR ENTERS QUEUE PROCESSES BATCH

slide-5
SLIDE 5

TYPE A QUOTE HERE.

Johnny Appleseed

slide-6
SLIDE 6

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

ZendCon 2016

REAL TIME IS BULLSHIT

slide-7
SLIDE 7 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

REALTIME (ACCORDING TO POSIX)

Ability of a system to provide a service within a bounded response time

slide-8
SLIDE 8 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

PROBLEMS CHALLENGES YOU WILL FACE

▸ Memory ▸ IO/Socket Blocks ▸ Network stack ▸ Timing of sequential jobs ▸ Queue

slide-9
SLIDE 9 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

QUEUES

▸ Push/Pull (poll) ▸ Priority ▸ Time To Live ▸ Delay ▸ Tubes

slide-10
SLIDE 10 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

TERMS

▸ Ready ▸ Reserved ▸ Delayed ▸ Buried ▸ Touch

slide-11
SLIDE 11 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB LIFE CYCLE

Time

RESERVED DELAY TTR

Posted Complete Start

slide-12
SLIDE 12 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB LIFE CYCLE GOTCHA

TTR

Time

DELAY RESERVED

Posted

TTR

Start

RESERVED

Re-Queued Start

slide-13
SLIDE 13

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

ZendCon 2016

CHECKPOINTS

slide-14
SLIDE 14

PUSH OR PULL?

slide-15
SLIDE 15 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

PUSH OR PULL?

▸ Hosted ▸ Queue Starts worker ▸ Scales with size of queue ▸ Limits number of workers ▸ Limited control over worker

environment

▸ Self-Hosted ▸ Process needed to check for new

queue

▸ Resources required for polling ▸ Additional delay ▸ Control of the Environment

PUSH PULL

slide-16
SLIDE 16 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

BEANSTALKD - PULL

▸ In Memory ▸ Lightweight text protocol ▸ Useful reporting ▸ Cannot scale

PROS CONS

slide-17
SLIDE 17 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

GEARMAN - PULL

▸ In Memory ▸ Distributed ▸ Workers can specify their capabilities ▸ Persistent with MySQL Plugin ▸ But Don't use it ▸ PHP Lib has Limited reporting ▸ Workers connecting to multiple job

servers can cause delays in jobs

▸ Job types can be confusing ▸ Simple Priorities

PROS CONS

slide-18
SLIDE 18 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

RESQUE - PULL

▸ Redis ▸ Easy Protocol ▸ Management GUI ▸ Not really a Job Queue ▸ Limited reporting ▸ Roll your own if you know Redis

PROS CONS

slide-19
SLIDE 19 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

RABBITMQ - PULL

▸ Easy Protocol ▸ Management GUI ▸ Persistent ▸ Not really a Job Queue ▸ Limited reporting ▸ Roll your own if you know Redis ▸ Memory error

PROS CONS

slide-20
SLIDE 20 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

WHAT YOU SHOULD DO - POLLING QUEUES

JOB PROCESS JOB PROCESS JOB PROCESS JOB PROCESS JOB PROCESS MINION MINION MINION OVERLORD (POLLS)

SERVER

slide-21
SLIDE 21 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

WHAT YOU SHOULD NOT DO - POLLING QUEUES

▸ Process every job with one process ▸ Have worker processes talk to the queue ▸ Use PCNTL (well...) ▸ PHP 7 Could be better ▸ But not on Windows

slide-22
SLIDE 22 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB

slide-23
SLIDE 23 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB

slide-24
SLIDE 24 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB AGGREGATE

slide-25
SLIDE 25 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB SERVICE

slide-26
SLIDE 26 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB HANDLER

slide-27
SLIDE 27 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB HANDLER

slide-28
SLIDE 28 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

JOB HANDLER

slide-29
SLIDE 29 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

AMAZON SQS/SNS/LAMBDA

▸ Scales Easily ▸ Easy Protocol (API) ▸ Management GUI ▸ Workers must be pre-defined ▸ Jobs can take between 10 seconds to 15 min to pop out ▸ Python, Java or NodeJS

slide-30
SLIDE 30 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

GOOGLE TASK QUEUE

▸ Scales Easily ▸ Queues can be programmatically created or through API ▸ Supports push and pull queues ▸ Schedule Tasks ▸ On Demand TTR

slide-31
SLIDE 31 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

FINIAL THOUGHTS

▸ Limit Connections to the Database ▸ Validate data going into a job ▸ Handel Errors ▸ Remove job on error ▸ Central Logging

slide-32
SLIDE 32 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

slide-33
SLIDE 33 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

slide-34
SLIDE 34 ZendCon 2016

BACKGROUND JOB PROCESSING - DO'S AND DON'TS

slide-35
SLIDE 35

THANKS

CHUCK REEVES @MANCHUCK