1 Decoupled & Uprooted Case Study, Government of Flanders - - PowerPoint PPT Presentation

1

Case Study, Government of Flanders

Tomas Fülöpp (Vacilando)

Decoupled & Uprooted

2

Decoupled & Uprooted

Builder Track: Case Study

Building a Challenging Web Platform at the Flemish Government

Tomas Fülöpp (Vacilando)

3

Your Speaker What to Expect

4

Tomas Fülöpp

a.k.a. Vacilando

ICT architect at the Flanders Information Agency Working for the Government of Flanders in Belgium since 2012 Drupal aficionado since 2006 „Vacilando“ on drupal.org Born in Czechoslovakia, Belgian by choice, European at heart https://www.drupal.org/u/vacilando https://www.linkedin.com/in/vacilando/

5

The Next ½ Hour

Your speaker Government of Flanders 8 years of Drupal 7 Time to Reinvent Context Building Blocks Development Takeaways Thanks Pages Questions?

6

Government of Flanders Belgium, Europe

7

Belgium

Europe

That speck on the map Federal constitutional monarchy divided into three highly autonomous regions:

• Flanders,
• Wallonia,
• and Brussels.

Famous for:

• Beer
• Chocolate
• French fries
• Dries Buytaert

8

Flanders

Belgium

Tiny by any standards

• 6.6 million inhabitants
• 13.5 thousand km²

Or to put it in U.S. context:

• about the size of Puerto Rico and
• with the population of Indiana

9

Flemish Public Administration

a.k.a. Government of Flanders

… or Flemish Government Prepares and executes policies of the Flemish Region and the Flemish Community. Main organs

• Flemish Parliament
• Flemish Government

Main public portal https://www.vlaanderen.be Responsibility of the Flanders Information Agency That’s what we’re talking about here!

10

8 Years of Drupal 7 www.vlaanderen.be

11

Early Adopter

• f Drupal 7

Drupal 7 was released in 2011 The Flemish Government embraced it as CMS of choice still in the same year Main citizen portal www.vlaanderen.be in D7 was a great success after the previous CMS (FatWire).

• Modern, comfortable and fast editing
• Separate interfaces thanks to Domain Access
• Plenty of data source integrations
• Lots of custom code but reasonably clean
• Several redesigns over the years
• Inspirational for great many websites around the Government (also as open source)
• Some 80k pages, permanent editing team keeping the information current and accurate
• 2 million sessions per month (3.5 million page views)

12

Vlaanderen.be in 2011

13

Vlaanderen.be in 2018

14

8 Years of Drupal 7

Time for a Next Step

Usual Dru rupal 7 pain ins, especially: Rigid; one large monoli lith

• Editing environment + front-end + data synchronized from external sources
• Clumsy and risky deployment (manual configuration -> Features); limited speed of

development / time to production Diffic ifficult lt performance sc scalin ling

• Non-cached performance dismal = particularly frustrating for the Editorial Team (front

blazing fast thanks to Varnish and smart invalidation of all related pages) Simply old ld

• After 8 long years time to retire

15

Time to Reinvent From Portal to a Web Platform

16

From Portal to a Web Platform

www.vlaanderen.be

Much more th than yet another re redesi sign. Transition from a simple portal to a Web Pla latf tform

• able to maintain and publish information from any and all Government

departments (API-First, No Wrong Door)

• for citizens and professionals.

17

Requirements

• Scalabil

ility: front-end, editi ting envi

• vironment. Be able to grow to an information platform for

any and all government entities. Accommodate many diverse Editorial Team.

• Fle

lexib ible content model

• el. Editors must be able to use a wide variety of content structures.
• Main

intain inabili

• ility. Microservices over monoliths. Continuous integration and deployment.
• Accessib

ibili

• lity. Everybody needs to be able to get the information they need.
• Per
• erformance. Best practically attainable.
• Se
• Security. Uncompromising.
• Reuse. Avoid duplication of content, code, systems. Use the open source alternative

wherever possible – starting with Drupal.

• In

Inte

• tegrate. Respect existing integrations.
• Sustainabil

ilit

• ity. Emanate stability, reliability, trustworthiness to the citizens. Not a short-term

product or campaign site.

• St
• Standards. Commitment to industry as well as organizational standards.

18

Fundamental Questions

• 1. Is Drupal - still - the best CMS?
• After 8 years Drupal perceived as “old technology”.
• Appetite for something radically different, for the cutting edge.
• Drupal 8 seemed nice, but not convincing (for non-Drupalers!)
• 2. Go decoupled (API-first)? Or is it perhaps just a hype?
• 3. Can we decouple (uproot) the content as well? Can we use Drupal for

editing but not store the data in its database?

19

Fundamental Decisions

Beginning of 2018

• 1. CMS: We continue to trust Drupal (8)
• Review of competitors (Adobe Experience Manager, Contentful, GraphCMS, Prismic , WP, etc.)
• Drupal still the most feature-complete and mature
• Lots of expertise and enthusiasm in-house
• 2. Front-end: Go decoupled, using data from Drupal API
• Added complexity, risk of hype but attraction of modularity / freedom; more than one front-end
• Huge enthusiasm from the front-end team
• 3. Remote / uprooted data sources
• We concluded we could not detach all content; it would make little if any sense to use Drupal

(decided with contribution of Wim Leers)

• We can however use remote sources of data and manipulate them just as if they were stored in
• Drupal. Not well documented but D8 can do this!
• Use an asset library (DAM) instead of Drupal for storing images, videos, PDFs, etc.
• 4. Set up and move to a new, scalable hosting system.

20

Context Integrate & Reuse

21

Existing Integrations

Context

Even a green field (project) is part of a wider ecosystem. A.k.a. “brown field”.

• 1. Dependent on a number of data sources
• Organizations
• Publications
• Press releases
• Vacancies
• Assets (DAM)
• ...
• 2. Data source for several existing applications
• Call Center Knowledge Base
• Catalogue of Government Products
• Reporting
• …

22

Library of Web Components

Lib ibra rary of

• f Web

eb Componen ents (specific for the Flemish Government)

• CSS

SS & JS JS snippets (and also VueJS)

• Conform to the corporate id

iden entit ity of the Government

• Mobile

ile-fir first

• Tested for acces

essi sibili ility

• Bes

est design pra ractic ices Ava vaila ilable le fo for use se by other Flemish Government entities Documentatio ion at https://overheid.vlaanderen.be/webuniversum/v3/

23

Global Header & Footer Widgets

Not Web Components as such but “wid idget ets” Highly configurable JS JS applications Conform to the corporate id iden entit ity of the Government Easy sy in inte tegration (embed script or SSI) Gateway to to a growing se secured area where each citizen can find personalized and private information related to the Government (family information, degrees obtained, status of subsidy requests, etc.)

24

Global Header & Footer

25

Global Header & Footer

26

Building Blocks CMS, Front, Content API

27

• I. Heart of Gold

Drupal 8 as the CMS

Conten ent st stru ructure: heavy use of Paragraphs

• Much more flexible than just using plain content types
• Alternatives: Bricks (underdeveloped), or within WYSIWYG

In Inte ternal se searc rch: Search API with Solr backend

• Alternative: Elasticsearch, but the integration seemed to still lag behind

API: GraphQL (https://www.drupal.org/project/graphql). Could as well have been REST or JSON. POST requests, yet cached by Varnish → very little load on Drupal. Site te API: Decoupled Router (https://www.drupal.org/project/decoupled_router) provides an endpoint that will help you resolve path aliases and redirects for entity related routes. In Inte ternal l cachin ing: Redis

• Proven slower than DB caching on HA infrastructure (network latency). Switched off to

increase performance!

28

Remote / Uprooted Data Sources

External Entities

Module External Enti titi ties es allows you to seamlessly integrate datasets from external databases: https://www.drupal.org/project/external_entities While the content lives external, Drupal sees those remote entiti ties as as internal. This makes it possible to alter the field displays, add references, comments, path aliases, share buttons, and more. New ew, significantly rew eworked branch 8.x-2.x by Raf @rp7, with contributions from Wim Leers. See https://www.drupal.org/project/external_entities/issues/2995140 Featured in a presentation at at Drupal Europe in in Darmstadt in September 2018, check https://www.drupaleurope.org/session/how-cope-external-entities

29

Drupal Contributions

By Our Team

External enti titi ties es — improved and refactored branch 8.x-2.x by Raf @rp7, with contributions from Wim Leers

• https://www.drupal.org/project/external_entities

Entit tity Usa sage In Inte tegrit ity — new module by Maciej @gugalamaciek

• https://www.drupal.org/project/entity_usage_integrity
• Protects integrity of references based on entity usage, preventing

inaccessible references Is Issu suu Vie iewer er – new module by Tamar @tamarpe

• https://www.drupal.org/project/issuu_viewer

30

Drupal Contributions

By Our Team

Patches to a host of d.o. projects:

• https://www.drupal.org/project/entity_browser
• https://www.drupal.org/project/bynder
• https://www.drupal.org/project/entity_usage
• https://www.drupal.org/project/decoupled_router
• https://www.drupal.org/project/openid_connect
• https://www.drupal.org/project/anonymous_redirect
• https://www.drupal.org/project/unique_field_ajax
• https://www.drupal.org/project/field_group
• https://www.drupal.org/project/inline_entity_form
• https://www.drupal.org/project/purge
• ...

Full list of contributions:

• https://www.drupal.org/government-of-flanders

31

• II. Decoupled Front-end

VueJS & NuxtJS

Opinions divided between ReactJS and VueJS VueJS for client side selected mainly because of

• Further reuse that technology choices within the Agency
• The Front-end Team existing experience and huge motivation

NuxtJS for server-side rendering (analogous to NextJS for React) Vuex for state management (analogous to Redux for React) CDN (Akamai) with smart invalidation. Not just HTML but also JSON data. Largest, but just one of the front-ends possible thanks to the API.

32

Front-end Caching

Cachin ing at t th the e ed edge: Akamai

• World class CDN, many more edge locations than competitors
• Smart invalidation = custom implementation
• Cache tags possible although rather new

WAF (preventing attacks such as DOS, XSS, SQL-injection, ...) – on the edge L7 lo load bala lancer – on the edge Fast stly ly was a viable alternative, mainly because it’s in essence a Varnish at the edge, but has less locations and support options.

33

Performance Testing

Using CDN

34

Why Do We Need WAF

Watching For Suspect Requests

35

• III. Remote / Uprooted Data Sources

New building block we call Content API

• Aggre

regatio ion and equalization of data coming from the individual remote data source APIs

• Cachin

ing (slow or unstable APIs) in MongoDB (vs memory – not permanent, or Elasticsearch)

• Sy

Synchroniz izatio ion scripts (common base but tailored improvements). Particular complic licatio ions with data ta that need to be en enri riched by Drupal.

• Filt

ilter ering, , so sort rtin ing (not available in all APIs)

• Mic

icrose servic ices, separation of concerns, modularity. Small blocks, each with clear purpose.

• For the Content API is the dec

ecouple led fro front-end ju just st on

• ne of
• f th

the e data ta con

• nsumers. Front-end,

Drupal, and other integrators talk to one single interface

• For the Content API is Dru

rupal l ju just st another data ta so source

• Output using Gra

raphQL instead of REST or JSON Mes essa saging queu eue using RabbitMQ

36

Open-source data query language for APIs. Ideal for heterogenous sources. Here‘s a simple query (to Content API, not Drupal) that provides:

• Publications (an external source) with matching

title

• Total number of items in the result list
• Specific items
• Publication date item will be formatted as needed

Use of GraphQL

37

Example GraphQL Query

Results

38

Digital Asset Management (DAM)

Im Images, s, vid video eos, s, and als lso file files s liv live e in in th the e DAM. Use Drupal files only for generated stuff. One e of f th the Content API so sources. Chosen for Byn ynder

• Existing modules not suited for our use case (no need to sync to media entities) → custom

module le

• In

Inadequate support for im image e der eriv ivativ ives (limited number, batch-created, hardcoded specs) Clo loudin inary chosen as a solution for the fro front en end

• Significant comple

lexit ity due e to to sy synchroniz izatio ion from Bynder to Cloudinary (technical debt)

39

Building Blocks

40

Hosting Infrastructure

Proudly using Amazee. e.io io host sting

• Very sc

scala lable le, based on managed Kubernetes es and OpenShift (EKS soon)

• Able to ru

run virtually any y applic lication thanks to containerization (far more than just Drupal hosting). Able to mix containers with cloud services (eg we use AuroraDB instead of MariaDB containers)

• Can be run lo

locally lly, on pre remis ise or r in in th the e clo loud (we use AWS)

• Own choice of infrastructure, you choose the specs
• Deployment based on GitO

itOps along the whole development path, from local through testing to production

• Excel

elle lent su support, invaluable for deep Dru rupal-specific hosting issues

• Enormous flexibility; rather a part

rtnership ip than just a hosting service

41

Development Happens April 2018 – March 2019

42

Eagle has Landed

March 21, 2019

Two majo jor ev even ents s on one e day 1. Spring is here! (in the Northern Hemisphere only, sorry) 2. Launch of the Web Platform at www.vlaanderen.be @ 21:55 CET

45

Eagle has Landed

www.vlaanderen.be

46

For the Future

What We Did Not Do

Dro rop Byn ynder and choose Cloudinary or an even better DAM with full capabilities Add a non-public lic se sectio ion to the same website (for now the whole site is publicly accessible) Pre revie iew and vi visual l edit itin ing (NB: in the decoupled front-end) Robust ro role les and per ermis issi sions management system Wri ritin ting to to (some) re remote so sources through the Content API Improve Content API (the aggregator) so that it becomes a fu full ll-fle ledged data ta source Implement a powerfu ful cust stom se searc rch across the diverse sources (now Google CSE) Write many more te test sts (unit testing, end-to end scripts). Remove te technic ical l debt built up as the deadline approached.

47

Takeaways Well Done | Bleeding Edge Can Hurt

48

Takeaways

Decoupled Drupal

Adva vantages es

• Your Drupal team does not nee

eed to to worr rry about fro front-end

• API

I out t of f th the e box or using mature modules

• Fast, and the API can easily be cached for even better performance

Dis isadvantages

• So

Some of f basi sic Dru rupal functionality and contrib modules won‘t work with decoupled front simply by definition

• Dru

rupal l als lso use sed fo for r ro routin ing (Site API); so it’s more than simply a data source. However, there seem to be no good routing solutions for this in NuxtJS (perhaps generally in front-end JS apps), so it may be the best solution (Egg of Columbus). Watc tch out when using Red edis is or r Mem emcache on HA in infra

• frastructure. Measure if it actually speeds

up the system. Use Dru rupal l core re and qualit lity contrib to the fullest. Deviate only if necessary as it will cost you resources.

49

Takeaways

Decoupled Front

Adva vantages es

• Separately maintainable, lighter deployments
• Easily replaceable with every new hype – or business requirement
• Your front-end team will love you forever (freedom!)
• No worries about finding good front-enders that are familiar with Drupal theming

Dis isadvantages

• A LOT of added complexity to maintain
• The most natural things, like viewing a saved page, cannot be taken for granted any more

Don‘t decouple unless you are sure that the advantages dwarf the disadvantages for your use case. Watch out what data manipulation is in the front end, and which stays in the aggregator. Invest not just in a good CDN but also WAF. We’ve been seeing attacks from the very first day.

50

Takeaways

Remote / Uprooted Data Sources

Adva vantages es

• Perf

rfect re reuse, no mirroring of external data in Drupal

• Easily doable thanks to Exte

ternal l Entit titie ies 2.x .x Dis isadvantages (over native Drupal)

• Every API

I is is diffe ifferent and we had to creatively patc tch th the e hole les (in sync scripts)

• Selective cache

e in inva vali lidation must be cust stom built

• More

e comple lex: roles and permissions, internal search, sitemap generation, multilingual setup, ...

• Rea

eadin ing still relatively simple as s oppose sed to to wri ritin ting Bew eware e of f Byn ynder er as DAM. Cloudinary may be a better solution when it comes to front-end. Complic licatio ions in cases where external content is not used directly but as a part of Drupal entities (enric ichment).

51

Takeaways

Final Reminders

Remember the ecosystem. No gre reen en fie field ld exists in isolation. Follow architecture ideals but don‘t become a purist (common sense, prioritizing). Res espect yo your te team‘s expertise and motivation. In Inve vest st hea eavily ily in in re reuse: code, data, architecture, processes, teams. The open source community is your extended team. St Stay agile

• ile. The waterfall is poisoned.

“Ble leed eding ed edge e can hurt yo you” (credit: Dirk Stevens!) Dare to dream. Dare to fail. Dare re.

52

Thanks Pages Questions Welcome

53

Thanks to Our Wonderful Team

+ all those I forgot!

54

Thanks for Your Attention

Questions?

Questions, comments, or suggestions welcome! Happy to bring you in contact with the tech leads for deeper discussions. Contact me at Tomas.Fulopp@kb.vlaanderen.be https://www.drupal.org/u/vacilando https://www.linkedin.com/in/vacilando/ Visit https://www.vlaanderen.be/ And visit Flanders, Belgium!

55

Join us for contribution opportunities

Friday, April 12, 2019

9:00-18:00 Room: 602

Mentored Contribution First Time Contributor Workshop General Contribution

#DrupalContributions

9:00-12:00 Room: 606 9:00-18:00 Room: 6A

56

What did you think?

Locate this session at the DrupalCon Seattle website:

http://seattle2019.drupal.org/schedule

Take the Survey!

https://www.surveymonkey.com/r/DrupalConSeattle

57