evolving the architecture of guardian co uk
play

Evolving the architecture of guardian.co.uk Mat Wall Lead - PowerPoint PPT Presentation

Apr 24, 2023 •103 likes •931 views

Evolving the architecture of guardian.co.uk Mat Wall Lead software architect History 1821 - Manchester Guardian released 1936 - Scott Trust formed: No proprietor Unique in British newspapers to this day 1959 - The Guardian goes national

  1. Evolving the architecture of guardian.co.uk Mat Wall Lead software architect

  2. History

  3. 1821 - Manchester Guardian released

  4. 1936 - Scott Trust formed: No proprietor Unique in British newspapers to this day

  5. 1959 - The Guardian goes national

  6. 1959 - The Guardian goes national 2004 - “Berliner” redesign

  7. Digital History

  8. 1995 - web site launch Simple portal Experimental project

  9. 2006 - Europe’s largest online newspaper site. Reach of web far greater than national paper. 18M unique users, many international

  10. “The international audience for guardian.co.uk has brought a new goal within reach: for The Guardian to become the world’s leading liberal voice” GMG Scott Trust website

  11. “ The Guardian to become the world’s leading liberal voice” Outgrown our web platform New platform required

  12. 18 month build time 30M unique users 250M page impressions per month

  13. Beginning the R2 project What are we getting into? Intense 18 month agile build 4 development teams to manage >2M pages to migrate Lots of new functionality to develop

  14. R2 project approach Develop new system in parallel Zero downtime Migrate section by section to new system Architect system as we go along

  15. R1 Apache layer user requests community Custom apache module allows per-URL backend selection news Provides manageable migration R1 R2 ? business money sport migration environment science technology etc travel

  16. R1 Apache layer user requests community news R1 R2 ? business money sport migration environment science technology etc travel

  17. R1 Apache layer user requests community news R1 R2 business money migration sport environment science technology etc travel

  18. R1 Apache layer user requests community R1 R2 news migration business environment science money technology etc sport travel

  19. R1 Apache layer user requests R1 R2 business money sport community migration environment science news technology etc travel

  20. R1 Apache layer user requests news R1 R2 business money sport migration environment science technology etc community travel

  21. R1 Apache layer user requests community news R1 R2 business money sport environment science technology etc travel

  22. R1 Apache layer user requests community news R1 R2 business money sport environment science technology etc travel

  23. R2 architecture Start simple Impossible to predict final architecture Take an agile “Just in time” approach Learn from each release

  24. Travel site build Apache layer user requests Why Travel? community Only 14K articles to migrate Relatively low traffic news Manageable performance R1 R2 Test our information architecture business money migration sport environment science technology etc travel

  25. Application architecture Caucho resin Java 6 build Spring Controller (Spring MVC) Simple stateless app Velocity 1.5 EHCache Domain model Repositories Only needs to scale EHCache to14K articles Hibernate

  26. System architecture Apache Apache Apache R2 frontend Search R2 feeds Oracle R2 CMS

  27. Co-location Apache Apache Apache Apache R2 frontend R2 frontend R2 frontend R2 frontend standby CMS standby feeds standby Oracle MANCHESTER LONDON

  28. Co-location Apache Apache Apache Apache R2 frontend Search Search R2 frontend R2 frontend Search Search R2 frontend Unreliable database standby But: Only 14K articles. Cache fits in RAM! CMS standby feeds standby Oracle MANCHESTER LONDON

  29. Keyword Article Video Contributor Audio Series Tags Content Gallery Publication Cartoon Tone

  33. “Simple sites” Apache layer user requests What are “simple sites”? Sites with similar functionality to travel site community Content migration: 100K+ articles R1 R2 Front page of site news migration business environment science money technology etc sport travel

  34. “Simple sites” Apache layer user requests Performance tests indicate we should scale out application layer community 2 x app servers R1 R2 news migration business environment science money technology etc sport travel

  35. “Simple sites” Apache layer user requests Cache will longer fit in RAM: Site stability at risk We are in a WAN! ££££ to fix. community Site front page included in this release R1 R2 news I want to sleep at night migration business environment science money technology etc sport travel

  36. Emergency mode Apache Apache Apache Apache NFS Gracefully degrade in the event of an outage R2 frontend Handle clean releases Fall back to flat files for a short time R2 feeds Graceful (and cheap) Oracle R2 CMS

  37. Emergency mode Apache Apache Apache Apache NFS Store on NFS R2 frontend Get HTML Content available on site R2 feeds Oracle Poll queue R2 CMS Publish content

  38. Emergency mode Apache Apache Apache Apache NFS Store HTML on NFS disc Store on NFS R2 frontend Schedule refresh in queue: Get HTML Modified pages pressed in <2 minutes Unedited pages should be no more that 2 weeks old Content available on site R2 feeds When database down serve from NFS Graceful degredation in user experience Oracle Poll queue Fixed issue “Just in time” ie: before seen in production R2 CMS Publish content

  39. “Complex sites” Apache layer user requests What are “Complex Sites”? Sites with third party interactions. Complex feeds. More traffic. R1 R2 business 200K+ articles to migrate. money sport community migration environment science news technology etc travel

  40. “Complex sites” Apache layer user requests 200K + articles Performance tests indicate platform will be able to cope Some Oracle queries need optimising R1 R2 business No scale increase required on app server money sport community migration environment science news technology etc travel

  41. “Complex sites” Apache layer user requests R1 R2 business money sport community migration environment science news technology etc travel

  42. External information

  43. External information Web server Stop using database as integration point Simple change: REST integration with third party server side External Proxy net App server Use proxy server to ensure performance / stability system Third party control caching. Domain model. Used on our Sport site for football / cricket scores. Database

  44. External information Web server External Proxy net App server system Database

  48. R1 Apache layer user requests News site launch news The big one! R1 R2 Will end up with nearly 1M content pages! business money Much traffic sport migration environment science technology etc community travel

  49. R1 Apache layer user requests news R1 R2 business money sport migration environment science technology etc community travel

  50. R1 Scalability predictions Platform team formed. They predict problems with: related content tag pages Both will max out our database How radical will we have to be?

  51. R1 Related content 40% of Oracle load

  52. R1 Related content Difficult to decache

  53. R1 Related content High editorial value component

  54. R1 Related content Get it off the database!!

  55. R1 Solution Use Endeca search engine Index page ID > [tag IDs] Group tag IDs into buckets. Bucket size determined by content volume for tag.

  56. R1 Solution Page B1 B2 B3 B4 B5 B6 B7 B8 ID 123 34,575 632 45 645 124 15 551 389 125 45 4,676 34 Tags with most Tags with least content content

  57. R1 Solution When user requests page: Free text search for tag IDs. Search engine relevance ranks results. Tags with least content get higher relevance. Returns page IDs.

  58. R1 Problem 2: Tag queries

  59. R1 Tag queries Platform team predict problems Queries becoming more expensive as content volume increases. Not scalable.

  60. R1 Tag queries Team have 2 ideas: 1: Cache page fragments on disc. Use Apache SSI. 2: SQL Queries can be sufficiently optimised I am tempted to just pick option 1.

  61. R1 but....

  62. R1 Developers are better than architects

  63. R1 Developers have greater understanding of the real detail of the system innards Don’t dictate to developers Let them innovate Allow them to try both options. (Privately I bet on Option 1, cache page fragments on disc)

  64. R1 Actual solution Independent Oracle consultant (with beard) optimised problem queries in 1 day. Performance tests say we’re good to go for News. Not what I expected!

  65. R1 Beyond News Platform team predicting scalability horizon ahead Caches overflowing Database load increasing Can no longer add app tier servers to scale

  66. R1 Beyond News

  67. R1 Beyond News

  68. R1 Reduce database load by 50% Keep it there

  69. R1 JBOSS cache & memcached 6Gb distributed cache in development now Can scale app tier without killing database Akami reverse proxy to reduce frontend load Required much later than I thought!

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

UI Evolving Platform Evolving Architecture Evolving About Me Xianning ( Pronunciation

UI Evolving Platform Evolving Architecture Evolving About Me Xianning ( Pronunciation

UI Evolving Platform Evolving Architecture Evolving About Me Xianning ( Pronunciation Shining) Liu Full Stack Developer, Mobile SME, Lead Consultant @ThoughtWorks Tech Lead &amp; Solution Architect. Recently focusing on scale mobile

854 views • 49 slides
Once a Guardian Always a Guardian! Why St. Joseph? We believe that as Guardians we will work as

Once a Guardian Always a Guardian! Why St. Joseph? We believe that as Guardians we will work as

St. Joseph High School Once a Guardian Always a Guardian! Why St. Joseph? We believe that as Guardians we will work as a community to educate our students in a way that will allow them to go into the world and make a difference! We want to

762 views • 37 slides
Democratising Attention Data at guardian.co.uk Graham Tackley @tackers Director of

Democratising Attention Data at guardian.co.uk Graham Tackley @tackers Director of

Democratising Attention Data at guardian.co.uk Graham Tackley @tackers Director of Architecture Guardian News and Media Tuesday, 1 October 13 Democratising Attention Data at guardian.co.uk m o c . Graham Tackley @tackers

801 views • 49 slides
Get Ready for the New York State Paid Family Leave Law Presented By: Stephen DuFlo Guardian

Get Ready for the New York State Paid Family Leave Law Presented By: Stephen DuFlo Guardian

Get Ready for the New York State Paid Family Leave Law Presented By: Stephen DuFlo Guardian Life Insurance Company of America 5788 Widewaters Pkwy Dewitt NY, 13214 GUARDIAN and the GUARDIAN Logo are registered service marks of The Guardian

247 views • 14 slides
4/24/2014 Guardian Qualifications: A Journey Not a Destination Mindi R. Blanchard, M.Ed., CPG

4/24/2014 Guardian Qualifications: A Journey Not a Destination Mindi R. Blanchard, M.Ed., CPG

4/24/2014 Guardian Qualifications: A Journey Not a Destination Mindi R. Blanchard, M.Ed., CPG Bridge Builders, Ltd/ The Guardian Institute Sequim, Washington Eloisa Roses Ramos, LCSW, NCG Barry University/ Office of the Pubic Guardian

242 views • 8 slides
The Evolving Architecture of the Web Nick Sullivan Head of Cryptography CFSSL Universal SSL

The Evolving Architecture of the Web Nick Sullivan Head of Cryptography CFSSL Universal SSL

The Evolving Architecture of the Web Nick Sullivan Head of Cryptography CFSSL Universal SSL Keyless SSL Privacy Pass Geo Key Manager Recently Standards work TLS 1.3 C on peting Goals make browsing more performant private HTTP DNS

789 views • 77 slides
Why I chose mongodb for guardian.co.uk Mat Wall Lead Software Architect, guardian.co.uk It is

Why I chose mongodb for guardian.co.uk Mat Wall Lead Software Architect, guardian.co.uk It is

Why I chose mongodb for guardian.co.uk Mat Wall Lead Software Architect, guardian.co.uk It is not the strongest of the species that survives, nor the most intelligent. It is the one that is most adaptable to change. Early Period circa

818 views • 63 slides
Th The 2nd International Workshop on 2 d I t ti l W k h Mobility in the Evolving Internet

Th The 2nd International Workshop on 2 d I t ti l W k h Mobility in the Evolving Internet

Th The 2nd International Workshop on 2 d I t ti l W k h Mobility in the Evolving Internet Mobility in the Evolving Internet Architecture (MobiArch07) Xiaoming Fu, Katherine Guo, Sue Moon, Ryuji Wakikawa Jon Crowcroft Acknowlegements

83 views • 6 slides
NSW Public Guardian 24 July 2019 Genetic Alliance Australia Jessica Butz Outline of session

NSW Public Guardian 24 July 2019 Genetic Alliance Australia Jessica Butz Outline of session

NSW Public Guardian 24 July 2019 Genetic Alliance Australia Jessica Butz Outline of session Decision making The three separate agencies Who is the Public Guardian? Need for a guardian How a guardian is appointed in NSW Role and functions

567 views • 24 slides
Freedom to Speak Up Guardian Terri Vaselli National Guardian How much have your learnt? Freedom

Freedom to Speak Up Guardian Terri Vaselli National Guardian How much have your learnt? Freedom

Freedom to Speak Up Guardian Terri Vaselli National Guardian How much have your learnt? Freedom to Speak Up Q1 If you were a patient, which hospital was no barrel of laughs? Mid Staffs Email, intranet, app, in Q2 Name 3 mediums for

487 views • 16 slides
Fifi: An Architecture to Realize Self-evolving of Java Program Ming-Yang Hou Xi-Yang Liu He-Hui

Fifi: An Architecture to Realize Self-evolving of Java Program Ming-Yang Hou Xi-Yang Liu He-Hui

Fifi: An Architecture to Realize Self-evolving of Java Program Ming-Yang Hou Xi-Yang Liu He-Hui Liu ehmy@263.net xyliu@xidian.edu.cn hhliu@mail.xidian.edu.cn Software Engineering Institute Xidian University, China ICSE 2006 Workshop on

675 views • 18 slides
Algorithms and Architecture for Managing Evolving ETL Workflows Judith Awiti Universit Libre

Algorithms and Architecture for Managing Evolving ETL Workflows Judith Awiti Universit Libre

Algorithms and Architecture for Managing Evolving ETL Workflows Judith Awiti Universit Libre de Bruxelles, Belgium Esteban Zimnyi (Home Supervisor) : Universit Libre de Bruxelles Robert Wrembel (Host Supervisor) : Pozna University of

526 views • 34 slides
people's confidentiality is protected: the Caldicott Guardian's perspective 1 Introduction

people's confidentiality is protected: the Caldicott Guardian's perspective 1 Introduction

Making sure data from wearable medical devices is used properly and people's confidentiality is protected: the Caldicott Guardian's perspective 1 Introduction Tim Kendall United Kingdom Caldicott Guardian Council (UKCGC) The

404 views • 9 slides
National Guardian Freedom to Speak Up Dr Henrietta Hughes National Guardian What are Freedom to

National Guardian Freedom to Speak Up Dr Henrietta Hughes National Guardian What are Freedom to

SPF 2016 Oct 2.1 National Guardian Freedom to Speak Up Dr Henrietta Hughes National Guardian What are Freedom to Speak up Guardians? for the NHS Social Partnership Forum 18 October 2016 1 National Guardian Dr Henrietta Hughes Freedom to

290 views • 9 slides
Introduction The volume of information is increasing as evolving modern science

Introduction The volume of information is increasing as evolving modern science

HAMA : An Efficient Matrix Computation with the MapReduce Framework IEEE CLOUDCOM 2010 Workshop, Indianapolis, USA Sangwon Seo , Computer Architecture Lab, KAIST, 1 Dec. 2010 Introduction The volume of information is increasing as evolving

1k views • 21 slides
??? Adult Guardianship in Sweden Each municipality has to organize either a Chief Guardian

??? Adult Guardianship in Sweden Each municipality has to organize either a Chief Guardian

4/25/2014 ADULT GUARDIANSHIP IN SWEDEN Dr theol. Christer Fjordevik Adult Guardianship in Sweden 1957 the Break Point Municipel Chief Guardian Authority Guardian Chamber a Politician Court of Law ??? Adult Guardianship in Sweden

176 views • 4 slides
Socialising Health Burden Through Different Network Topologies: A Simulation Study Adrian PEACOCK

Socialising Health Burden Through Different Network Topologies: A Simulation Study Adrian PEACOCK

Socialising Health Burden Through Different Network Topologies: A Simulation Study Adrian PEACOCK a , Anthony CHEUNG b , Peter KIM b and Simon K. POON a,1 a The University of Sydney, School of Information Technologies, Australia b The University of

306 views • 7 slides
Jameson Rollins LIGO/Virgo Joint Run Planning Committee 12/19/2013 Overview Guardian is the

Jameson Rollins LIGO/Virgo Joint Run Planning Committee 12/19/2013 Overview Guardian is the

aLIGO Guardian Synopsis and Status Update Jameson Rollins LIGO/Virgo Joint Run Planning Committee 12/19/2013 Overview Guardian is the aLIGO automation system. It will manage the state of the interferometers by coordinating state transitions

342 views • 6 slides
RETURN TO SCHOOL UPDATE Special Board Meeting July 8, 2020 Dr. Vivian Ekchian, Superintendent

RETURN TO SCHOOL UPDATE Special Board Meeting July 8, 2020 Dr. Vivian Ekchian, Superintendent

RETURN TO SCHOOL UPDATE Special Board Meeting July 8, 2020 Dr. Vivian Ekchian, Superintendent Purpose How GUSD Has Been Gathering Stakeholder Input Instructional Models Preliminary Findings from Stakeholder Surveys o Parent/Guardian

358 views • 13 slides
Social Distancing Technology f r o m Your speakers Dave Asheim Andrew Harte Emily White

Social Distancing Technology f r o m Your speakers Dave Asheim Andrew Harte Emily White

Social Distancing Technology f r o m Your speakers Dave Asheim Andrew Harte Emily White Founder &amp; CEO Sales Representative Project Manager Engage by Cell for Advanced Industrial Marketing for Engage by Cell Social distancing is key

621 views • 24 slides
CS 188: Artificial Intelligence Introduction Instructors: Anca Dragan, Sergey Levine University

CS 188: Artificial Intelligence Introduction Instructors: Anca Dragan, Sergey Levine University

CS 188: Artificial Intelligence Introduction Instructors: Anca Dragan, Sergey Levine University of California, Berkeley (slides adapted from Dan Klein, Pieter Abbeel) Today o What is artificial intelligence? o Where did it come from? o What can

829 views • 54 slides
Welcome Web Application Security MIT Security Camp August 21, 2003. Boston, MA. Chris Lambert

Welcome Web Application Security MIT Security Camp August 21, 2003. Boston, MA. Chris Lambert

Next: Who am I? Welcome Web Application Security MIT Security Camp August 21, 2003. Boston, MA. Chris Lambert &lt;chris@ccs.neu.edu&gt; http://www.clambert.org/talks/ 1 Previous: Welcome Next: Why this talk? Who am I? A bit about me

747 views • 26 slides
Sergeant at Arms Training Session www.toastmasters.org Sergeant at Arms TRAINING DISTRICT 37

Sergeant at Arms Training Session www.toastmasters.org Sergeant at Arms TRAINING DISTRICT 37

Sergeant at Arms Training Session www.toastmasters.org Sergeant at Arms TRAINING DISTRICT 37 TLI EAST RALEIGH, NC JANUARY 17, 2015 Sergeant at Arms (SAA) Sergeant at Arms Role Sergeant at Arms Responsibilities Sergeant at Arms

874 views • 21 slides
Speaker Profile Speaker Profile Information Technology Hesham Tamraz Technology

Speaker Profile Speaker Profile Information Technology Hesham Tamraz Technology

Speaker Profile Speaker Profile Information Technology Hesham Tamraz Technology consultant specializing in non-profits. for Non-Profits Former Information Technology (IT) Director at a national association for 8 years (1991 1999)

70 views • 4 slides

More recommend