CHUCK REEVES @MANCHUCK BACKGROUND JOB PROCESSING DO'S AND DON'TS
BACKGROUND JOB PROCESSING - DO'S AND DON'TS IMAGE PROCESSOR UPLOAD IMAGE PROCESS IMAGE ENTERS QUEUE ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS THE PROBLEM CHALLENGE ▸ Crawl Google Merchant Center ▸ Crawl Google Ad Words ▸ Crawl Google Analytics ▸ Match Data ▸ Analyze Data ▸ Make Decisions ▸ POST/PUT Google Ad Words ▸ Within 4 hours ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS CATALOGUE CRAWLER TIME OF DAY CRAWLS CATALOGUE ENTERS QUEUE BATCH PROCESSOR PROCESSES BATCH BATCHES CATALOGUE ENTERS QUEUE ZendCon 2016
TYPE A QUOTE HERE. Johnny Appleseed
BACKGROUND JOB PROCESSING - DO'S AND DON'TS REAL TIME IS BULLSHIT ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS REALTIME (ACCORDING TO POSIX) Ability of a system to provide a service within a bounded response time ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS PROBLEMS CHALLENGES YOU WILL FACE ▸ Memory ▸ IO/Socket Blocks ▸ Network stack ▸ Timing of sequential jobs ▸ Queue ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS QUEUES ▸ Push/Pull (poll) ▸ Priority ▸ Time To Live ▸ Delay ▸ Tubes ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS TERMS ▸ Ready ▸ Reserved ▸ Delayed ▸ Buried ▸ Touch ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB LIFE CYCLE Posted TTR DELAY Time RESERVED Complete Start ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB LIFE CYCLE GOTCHA Posted Re-Queued TTR DELAY TTR Time RESERVED RESERVED Start Start ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS CHECKPOINTS ZendCon 2016
PUSH OR PULL?
BACKGROUND JOB PROCESSING - DO'S AND DON'TS PUSH OR PULL? PULL PUSH ▸ Hosted ▸ Self-Hosted ▸ Queue Starts worker ▸ Process needed to check for new queue ▸ Scales with size of queue ▸ Resources required for polling ▸ Limits number of workers ▸ Additional delay ▸ Limited control over worker environment ▸ Control of the Environment ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS BEANSTALKD - PULL CONS PROS ▸ In Memory ▸ Cannot scale ▸ Lightweight text protocol ▸ Useful reporting ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS GEARMAN - PULL CONS PROS ▸ In Memory ▸ PHP Lib has Limited reporting ▸ Distributed ▸ Workers connecting to multiple job servers can cause delays in jobs ▸ Workers can specify their capabilities ▸ Job types can be confusing ▸ Persistent with MySQL Plugin ▸ Simple Priorities ▸ But Don't use it ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS RESQUE - PULL CONS PROS ▸ Redis ▸ Not really a Job Queue ▸ Easy Protocol ▸ Limited reporting ▸ Management GUI ▸ Roll your own if you know Redis ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS RABBITMQ - PULL CONS PROS ▸ Easy Protocol ▸ Not really a Job Queue ▸ Management GUI ▸ Limited reporting ▸ Persistent ▸ Roll your own if you know Redis ▸ Memory error ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS WHAT YOU SHOULD DO - POLLING QUEUES JOB PROCESS SERVER MINION JOB PROCESS OVERLORD JOB PROCESS (POLLS) MINION JOB PROCESS MINION JOB PROCESS ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS WHAT YOU SHOULD NOT DO - POLLING QUEUES ▸ Process every job with one process ▸ Have worker processes talk to the queue ▸ Use PCNTL (well...) ▸ PHP 7 Could be better ▸ But not on Windows ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB AGGREGATE ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB SERVICE ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB HANDLER ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB HANDLER ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS JOB HANDLER ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS AMAZON SQS/SNS/LAMBDA ▸ Scales Easily ▸ Easy Protocol (API) ▸ Management GUI ▸ Workers must be pre-defined ▸ Jobs can take between 10 seconds to 15 min to pop out ▸ Python, Java or NodeJS ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS GOOGLE TASK QUEUE ▸ Scales Easily ▸ Queues can be programmatically created or through API ▸ Supports push and pull queues ▸ Schedule Tasks ▸ On Demand TTR ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS FINIAL THOUGHTS ▸ Limit Connections to the Database ▸ Validate data going into a job ▸ Handel Errors ▸ Remove job on error ▸ Central Logging ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS ZendCon 2016
BACKGROUND JOB PROCESSING - DO'S AND DON'TS ZendCon 2016
CHUCK REEVES @MANCHUCK THANKS
Recommend
More recommend
