yes the size matters
play

Yes! The size matters! PostgreSQL implementation on a large - PowerPoint PPT Presentation

Sep 09, 2022 •579 likes •1.65k views

Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Yes! The size matters! PostgreSQL implementation on a large operational database,. . . Thomas BOUSSEKEY (@ThomasBoussekey) DECATHLON, retail IT

  1. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Size considerations Volume of Records Physical E-commerce POSLOG Length 12 kB 8 kB POSLOG per Tx 1 7 Days opened per year 325 365 Tx per year (M) 683.96 6.71 Poslog per year (M) 683.96 47.01 Tx = Transaction Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 28 / 95

  2. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Database size estimation for EUROPE only Transactions (M) DB Size (GB) Yearly growth (%) End 2017 80 350 2300 End 2018 480 2.000 570 End 2019 1.000 5.000 250 End 2020 1.600 8.000 160 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 29 / 95

  3. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Industrialisation Implementing PostgreSQL puppet Brick Derivated from puppet forge Modified for multiple instance purposes. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 30 / 95

  4. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps PUPPET PGSQL Brick structure classes: dkt_postgresql: instances: cash: instance_port: 5432 pg_conf: [ config_options @ postgresql.conf ] pg_hba: [ hba_entries @ pg_hba.conf ] databases: [ database_config ] Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 31 / 95

  5. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 32 / 95

  6. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 33 / 95

  7. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 34 / 95

  8. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Risk management Spread data in multiple tables to limit risks, locking . . . Possible solutions: Design the application to store data for each pos in a different table. Partitionning manually Using extensions pg_partman pg_pathman Both extensions were easy to install and configure Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 35 / 95

  9. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Decision When we had to take a decision, pg_partman was chosen: Extension maturity (IN 2016 Q2): pg_partman was already “mature” pg_pathman was in version 0.4 had many stars on GITHUB “almost” compliant with our cloud DB managed service Project was up-to-date (regarding Pg releases) Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 36 / 95

  10. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Breaking news PostgreSQL10 Partitioning is now fully integrated into PostgreSQL 10! Native partitioning feature Link to PostgreSQL WIKI Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 37 / 95

  11. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Partitioning Build a partition key for point of sales 3 columns to define a Point of sale Type of TIER TIER number TIER sub-number Merge these 3 columns into a single column Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 38 / 95

  12. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Partitioning Build a partition key for point of sales 3 columns to define a Point of sale Type of TIER TIER number TIER sub-number Merge these 3 columns into a single column Huge interest for database statistics Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 38 / 95

  13. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Breaking news PostgreSQL 10 Cross-column Statistics feature is now AVAILABLE Link to PostgreSQL WIKI Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 39 / 95

  14. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Type of value for the partitioning key INTEGER format preserved, for: search efficiency ID partitioning VARCHAR might be: more difficult to handle less performant for searches and joining Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 40 / 95

  15. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Range (Exact-ID) partitionning Need to size partitioning range to 10 (minimum set by pg_partman extension) Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 41 / 95

  16. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Range (Exact-ID) partitionning Need to size partitioning range to 10 (minimum set by pg_partman extension) NEED Get a ‘human readable identifier‘ Create a ‘bigint‘, grouping the 3 fields and a final "0" Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 41 / 95

  17. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Range (Exact-ID) partitionning Need to size partitioning range to 10 (minimum set by pg_partman extension) NEED Get a ‘human readable identifier‘ Create a ‘bigint‘, grouping the 3 fields and a final "0" Example: 7 00118 00118 0 for: Type of tier: 7 , 2 digits (7, 48, 50) Tier number: 00118 , 5 possible digits Sub-tier number: 00118 , 5 possible digits Plus , the trailing 0 , for range partitioning Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 41 / 95

  18. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Build a partition key for daily retention Create 2 partitioned tables named transaction_live and poslog_live to store records and ease time-window searches. These partitions have a 30 day retention time. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 42 / 95

  19. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps All partitions Table Name partitioning Data purge specifity transaction BU NEVER none transaction_live day > 2 weeks none poslog BU NEVER XML poslog poslog_live day > 1 week none BU = Business Unit Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 43 / 95

  20. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps SQL commands Integer range partitioning SELECT partman.create_parent( p_parent_table := 'posdata.transaction', p_control := 'id_business_unit', p_type := 'id', p_interval := '10', p_premake := 1); Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 44 / 95

  21. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Timestamp daily partitioning SELECT partman.create_parent( p_parent_table := 'posdata.transaction_live', p_control := 'creation_date', p_type := 'time', p_interval := 'daily'); Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 45 / 95

  22. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Additionnal tooling For adding: point of sales new partitions for new days Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 46 / 95

  23. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 47 / 95

  24. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 48 / 95

  25. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 49 / 95

  26. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Updating configuration for pg_partman instances: cash: pg_conf: [ .../... ] max_locks_per_transaction: 512 max_pred_locks_per_transaction: 256 shared_preload_libraries: [ ... ] ,pg_partman_bgw databases: posdata: database_name: posdata extensions: pg_partman: ensure: present schema: partman [ .../... ] Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 50 / 95 [ .../... ]

  27. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Create partitions for new business units: On modification on the parameter table business_unit Fire the trigger business_unit_fct If the partitions doesn’t exist, they are created Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 51 / 95

  28. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Version1 GO PROD Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 52 / 95

  29. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 53 / 95

  30. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 54 / 95

  31. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 55 / 95

  32. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Why do we encounter IOWait , 6 months after live on production ? IOWAIT strong increase discoverd on Friday JUNE 23rd morning on Production . . Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 56 / 95 WHAT MAY HAPPEN THIS WEEK-END ???

  33. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Searching a ROOT-cause! Here are your 3 new friends: pg_buffercache pg_stat_statements EXPLAIN Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 57 / 95

  34. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Working with PG_BUFFERCACHE SELECT c.relname as "Name", count(*) as "buffers", round(count(*)*8.0/1024,2) AS Buffer_MB, round(count(*)*100.0/( SELECT count(*) FROM pg_buffercache),2) as "%" FROM pg_buffercache b INNER JOIN pg_class c ON b.relfilenode = pg_relation_filenode(c.oid) WHERE b.reldatabase IN (0, ( SELECT oid FROM pg_database WHERE datname = (current_database()))) GROUP BY c.relname ORDER BY 2 DESC LIMIT 50; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 58 / 95

  35. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Clue #1 Small amount of heap partitioned POSLOG were found into the BUFFERS Can be improved by index creation Which query needs to be indexed? Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 59 / 95

  36. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Working with PG_STAT_STATEMENTS Having a good sample Do not investigate on N-1 version queries Avoid DBA & Dev queries A small tip On Friday afternoon, Run a SELECT pg_stat_statements_reset(); On Monday morning, “Snapshot” pg_stat_statements table Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 60 / 95

  37. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Query used to focus on IOWait SELECT left (pss.query, 120), pss.calls, pss.total_time, pss.mean_time, pss.stddev_time, pss.shared_blks_hit, pss.shared_blks_dirtied from pg_stat_statements pss where pss.dbid = ( SELECT oid from pg_database where datname = current_database()) order by pss.total_time desc limit 80; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 61 / 95

  38. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps CULPRIT identified: SELECT COALESCE (max(p.version), 0) FROM posdata.poslog_p700346003460 p WHERE p.id_business_unit= 700346003460 AND p.id_transaction= 1234; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 62 / 95

  39. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Execution plan EXPLAIN ( ANALYZE , BUFFERS) SELECT COALESCE (max(p.version), 0 WHERE p.id_business_unit= 700346003460 AND p.id_transaction= Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 63 / 95

  40. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Execution plan EXPLAIN ( ANALYZE , BUFFERS) SELECT COALESCE (max(p.version), 0 WHERE p.id_business_unit= 700346003460 AND p.id_transaction= Aggregate (cost=8797.93..8797.94 rows=1 width=4) (actual time=67.086..67.087 Buffers: shared hit=**5504** -> **Seq Scan** on poslog_p700346003460 p (cost=0.00..8797.92 Filter: ((id_business_unit = '700346003460'::bigint) AND (id_transaction Rows Removed by Filter: 219595 Buffers: shared hit=5504 Planning time: 0.321 ms Execution time: **67.162 ms** Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 63 / 95

  41. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps How to improve? How to index it efficiently? B-Tree index BRIN index Introduced in PostgreSQL 9.5 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 64 / 95

  42. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps BRIN index Definition BRIN stands for Block Range Index, default range is 1MB. BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table. Store MIN and MAX values for the related block. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 65 / 95

  43. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Test it before execution plan, once the BRIN index created: Aggregate (cost=16.03..16.04 rows=1 width=4) (actual time=0.116..0.1(10 Buffers: shared hit=**20** -> **Bitmap Heap Scan** on poslog_p700346003460 p (cost=12.01..16.03 Recheck Cond: ((id_business_unit = '700346003460'::bigint) AND Buffers: shared hit=20 -> **Bitmap Index Scan** on poslog_p700346003460_brin_idx01 Index Cond: ((id_business_unit = '700346003460'::bigint) AND Buffers: shared hit=20 Planning time: 0.406 ms Execution time: **0.198 ms** Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 66 / 95

  44. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Compare the 2 kinds of index Size (kB) Blocks read Heap 736.000 5.504 B-Tree INDEX 8.738 6 BRIN INDEX 48 20 BRIN index needs less IO on INSERT. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 67 / 95

  45. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Industrialize index creation CREATE OR REPLACE FUNCTION add_brin_index_on_poslog_tables() DECLARE live_part_table RECORD; BEGIN RAISE NOTICE 'Creating BRIN indexes on poslog partitions FOR live_part_table IN SELECT tab.relname as "tab_relname" RAISE NOTICE 'Updating constraint for table % ...', EXECUTE 'CREATE INDEX IF NOT EXISTS ' || quote_ident(live_part_table.tab_relname) --- CREATE INDEX CONCURRENTLY IF NOT EXISTS poslog_p700346003460_brin_idx01 END LOOP ; RAISE NOTICE 'Done creating BRIN indexes on poslog partitions RETURN 0; END ; $$ LANGUAGE plpgsql; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 68 / 95

  46. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Other thing, I’ve learned at this point PL/pgsql LANGUAGE doesn’t allow: VACUUM inside a transaction block. CREATE INDEX CONCURRENTLY inside a transaction block. COMMIT / ROLLBACK inside a transaction block. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 69 / 95

  47. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Oooopps!!! new IOWait peak several weeks after !!! RTFM! Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 70 / 95

  48. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Oooopps!!! new IOWait peak several weeks after !!! RTFM! BRIN index maintenance: Tuples on a new heap page are not summarized into the BRIN index automatically Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 70 / 95

  49. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Oooopps!!! new IOWait peak several weeks after !!! RTFM! BRIN index maintenance: Tuples on a new heap page are not summarized into the BRIN index automatically 2 possible ways to update the BRIN index: Using procedure: brin_summarize_new_values(regclass) Perform a VACUUM on the table. a a . https://www.postgresql.org/docs/9.5/static/brin-intro.html Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 70 / 95

  50. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 71 / 95

  51. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Solution CREATE OR REPLACE FUNCTION refresh_brin_index_on_poslog_tables() DECLARE live_part_table RECORD; BEGIN FOR live_part_table IN SELECT tab.relname as "tab_relname" RAISE NOTICE 'Refreshing BRIN index for table % ...', quote_ident(live_part_table.tab_relname); EXECUTE 'SELECT brin_summarize_new_values(''' || quote_ident(live_part_table.ind_relname) END LOOP ; RAISE NOTICE 'Done refreshing BRIN indexes on poslog partitions RETURN 0; END ; $$ LANGUAGE plpgsql; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 72 / 95

  52. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Emergency DDL update Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 73 / 95

  53. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Possible solutions 1 Create a new table for ReceiptNumber 2 Add a new column for ReceiptNumber into Poslog table Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 74 / 95

  54. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Impact analysis Tested on integration, on ~300k rows Size impact (MB) + Column + Table Delta (%) Poslog HEAP 13.75 0 Poslog INDEX 0 0 Receipt HEAP 0 31.79 Receipt INDEX 0 7.69 Total size 13.75 39.48 +287 Insert 75k rows (IO) 60,526 62,124 +3 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 75 / 95

  55. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Time impact (ms) + Column + Table Delta (%) Creation time 40 114162 Time for insertion 99236 0 Vacuum time 30981 0 Total 130257 114162 +14/-14 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 76 / 95

  56. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Improve XML storage Try on JSONB, 7% space reduced Offers full text search improvement Quick look over XMLTABLE 2 feature on PostgreSQL 10 2. https://blog.2ndquadrant.com/xmltable-intro/ Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 77 / 95

  57. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 78 / 95

  58. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Cloud Image from [Chris Potter](www.seywut.com/Chris). Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 79 / 95

  59. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Technology State of the art 2016Q1 2017Q3 Location onPremise Cloud OS RHEL 7 CentOS 7 Pg version 9.5 10 OR Postgres_XL Config mgmt Puppet V4 Foreman Terraform + foreman SQL scripts Flyway Flyway Deployment JENKINS JENKINS Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 80 / 95

  60. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps New hosting solution Leave onPremise datacenters to go to the cloud. Improve industrialization Start contenerization Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 81 / 95

  61. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Terraform Infrastructure as code Ability to perform operations on multiple hosting solutions cloud computing onPremise datacenters Interact with FOREMAN & PUPPET Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 82 / 95

  62. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Postgres_XL Key drivers Critical size estimated to 1 database for 200 databases. Needing 7 different databases for production (regarding DB size) Adding complex platform management to spread and search data accross the platforms Scalable ALTER NODE data_5 WITH ( TYPE = 'datanode'); We can define the sharding structure for EACH table ALTER TABLE disttab ADD NODE (data_5); Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 83 / 95

  63. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 84 / 95

  64. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Features Fully ACID Open Source Cluster-wide Consistency Multi-tenant Security PostgreSQL-based Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 85 / 95

  65. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Pros No new engine Only a few system tables & columns difference Table location strategy DISTRIBUTED: for large table, align distribution startegy for joins REPLICATED: for parameter tables SELECT xc_node_id, count(*) FROM transaction GROUP BY xc_node_id; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 86 / 95

  66. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Pros No new engine Only a few system tables & columns difference Table location strategy DISTRIBUTED: for large table, align distribution startegy for joins REPLICATED: for parameter tables SELECT xc_node_id, count(*) FROM transaction GROUP BY xc_node_id; Cons Prevent cross datanodes queries Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 86 / 95

  67. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Poslog storage Stored in raw mode No specific indexing on its content Only field to contain personal information Can be stored into a cloud file storage solution with encryption, to comply with GDPR regulation Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 87 / 95

  68. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Other new functionnalities / tools Improved monitoring TICK ELK Use containers for JAVA application New flow solution Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 88 / 95

  69. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thanks for your attention! Any question? Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 89 / 95

  70. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Created with opensource tools: markdown pandoc laTEX beamer and PostgreSQL Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 90 / 95

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Step Size Matters in Deep Learning Kamil Nar Shankar Sastry Neural Information Processing

Step Size Matters in Deep Learning Kamil Nar Shankar Sastry Neural Information Processing

Step Size Matters in Deep Learning Kamil Nar Shankar Sastry Neural Information Processing Systems December 4, 2018 Nar & Sastry Step Size Matters 1 Gradient Descent: Effect of Step Size Example min x R ( x 2 + 1)( x 1) 2 ( x

753 views • 18 slides
Size Matters How Loading is Losing You Players W3C Workshop on Web Games - 27-28 June 2019;

Size Matters How Loading is Losing You Players W3C Workshop on Web Games - 27-28 June 2019;

Size Matters How Loading is Losing You Players W3C Workshop on Web Games - 27-28 June 2019; Redmond, WA, USA Kasper Mol Game Tech Engineer @ Poki Poki - Web Gaming Platform 30M 200+ 28 Languages Monthly Active Game Developers Users SIZE

624 views • 14 slides
I . Preliminaries: practical matters I . Preliminaries: practical matters A. Office

I . Preliminaries: practical matters I . Preliminaries: practical matters A. Office

Krol Lear Knig Lear Krol Lear King Lear King Lear King Lear W. Shakespeare Shakespeare Shakespeare Wieslaw Walkuski Jerzy Czerniawski Rafal Olbinski theater; 1992; size B1 theater; 1981; size A1 theater; 2000; size B1 Polish Poster

677 views • 10 slides
Qtopia for KDE developers 07 | 07 That Size-and-matters-thingy Remember that PC you had 5 years

Qtopia for KDE developers 07 | 07 That Size-and-matters-thingy Remember that PC you had 5 years

Qtopia for KDE developers 07 | 07 That Size-and-matters-thingy Remember that PC you had 5 years ago? That hardware is in your phone now 2 Screens :) 3 So what is it? Qtopia contains... Qtopia Core a.k.a Qt/embedded - same Qt API as

325 views • 16 slides
Hardware Operators for Pairing-Based Cryptography Part I: Because size matters Jean-Luc

Hardware Operators for Pairing-Based Cryptography Part I: Because size matters Jean-Luc

Hardware Operators for Pairing-Based Cryptography Part I: Because size matters Jean-Luc Beuchat Laboratory of Cryptography and Informantion Security University of Tsukuba, Japan jeanluc.beuchat@gmail.com Joint work with: enaire, LIP,

1.73k views • 139 slides
2016 UPDATE ON CLASS SIZE, OVERCROWDING, AND CAPITAL PLAN: P .S. 39, DISTRICT 15 AND CITYWIDE

2016 UPDATE ON CLASS SIZE, OVERCROWDING, AND CAPITAL PLAN: P .S. 39, DISTRICT 15 AND CITYWIDE

2016 UPDATE ON CLASS SIZE, OVERCROWDING, AND CAPITAL PLAN: P .S. 39, DISTRICT 15 AND CITYWIDE Leonie Haimson for P.S. 39K PTA Class Size Matters March 31, 2016 Class Size reduction: a proven reform Lowering class size is only of only four

432 views • 24 slides
Size Matters, If You Control Your Junk Clifford S. Asness AQR Capital Management LLC Andrea

Size Matters, If You Control Your Junk Clifford S. Asness AQR Capital Management LLC Andrea

Size Matters, If You Control Your Junk Clifford S. Asness AQR Capital Management LLC Andrea Frazzini AQR Capital Management LLC, NYU Ronen Israel AQR Capital Management LLC, NYU Tobias Moskowitz AQR Capital Management, University of Chicago,

441 views • 31 slides
URBAN FRIENDLY BIOMASS @ SCALE Size Matters Scott Layne, CEM Nexterra System Corp.

URBAN FRIENDLY BIOMASS @ SCALE Size Matters Scott Layne, CEM Nexterra System Corp.

Advanced Energy Conference October 2011 URBAN FRIENDLY BIOMASS @ SCALE Size Matters Scott Layne, CEM Nexterra System Corp. slayne@nexterra.ca US Renewables Demand The EIA Annual Energy Outlook for 2010 projects that the demand

157 views • 15 slides
When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really

When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really

When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters Yogesh K. Ramadass, AnanthaGroup Microsystems Technology Laboratory Outline Outline Outline Outline

507 views • 36 slides
Lab 2 discussion Last Time Debugging Its a science use experiments to refine

Lab 2 discussion Last Time Debugging Its a science use experiments to refine

size= 64 correct= 0 size= 73 correct= 0 Shuying Liang size= 107 correct= 1 size= 107 correct= 0 size= 108 correct= 0 size= 108 correct= 1 Please do not handin a .doc file, a .zip file, a .tar file, size= 111 correct= 1 size=

430 views • 7 slides
The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions! joint work with

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions! joint work with

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions! joint work with Yachong Yang and Weijie Su Hua Wang The Wharton School, University of Pennsylvania June 2, 2020 Hua Wang (Wharton) The Price of Competition June

1.06k views • 66 slides
Measuring What Matters Quality, Impact and Measuring Social Value Philip Angier, Angier Griffin

Measuring What Matters Quality, Impact and Measuring Social Value Philip Angier, Angier Griffin

Measuring What Matters Quality, Impact and Measuring Social Value Philip Angier, Angier Griffin Measuring what matters Introductions: Name Organisation Where youre from first part of postcode Size approx annual

583 views • 27 slides
Titus 1:5-9 It matters who you are as a leader and it matters what you do as a leader. God is

Titus 1:5-9 It matters who you are as a leader and it matters what you do as a leader. God is

Leadership Matters (Verb) I Titus 1:5-9 It matters who you are as a leader and it matters what you do as a leader. God is primarily interested in who you are and then what you do (Daniel Akin). The Pastoral Crucible I.

600 views • 23 slides
Lumber Size Lumber Size Control Control Studies Studies Lumber Size Control Lumber Size

Lumber Size Lumber Size Control Control Studies Studies Lumber Size Control Lumber Size

Lumber Size Lumber Size Control Control Studies Studies Lumber Size Control Lumber Size Control Troubleshoot machine center performance Locate & identify problems Stabilize & improve sawing accuracy Reduce

686 views • 16 slides
S et the Bar Low. Be a WINNER every time. Public Power Matters Public Power Matters Innovation

S et the Bar Low. Be a WINNER every time. Public Power Matters Public Power Matters Innovation

S et the Bar Low. Be a WINNER every time. Public Power Matters Public Power Matters Innovation Workforce Advocacy Y our S tory Rates Matter Matters Matters Matters Matters Its the Rates, S tupid! Innovation Matters We must

280 views • 26 slides
FUNCTION MERGING by SEQUENCE ALIGNMENT DOES SIZE MATTER? DOES SIZE MATTER? S E Y DOES SIZE

FUNCTION MERGING by SEQUENCE ALIGNMENT DOES SIZE MATTER? DOES SIZE MATTER? S E Y DOES SIZE

Rodrigo Rocha http://rcor.me University of Edinburgh FUNCTION MERGING by SEQUENCE ALIGNMENT DOES SIZE MATTER? DOES SIZE MATTER? S E Y DOES SIZE MATTER? S E Y ... EMBEDDED SYSTEMS SAVINGS IN CODE Smaller Memories 1 0 1 1 0 1

1.89k views • 177 slides
0 Tips for Viewing this Webinar The questions box and buttons are on the right side of the

0 Tips for Viewing this Webinar The questions box and buttons are on the right side of the

0 Tips for Viewing this Webinar The questions box and buttons are on the right side of the webinar window. This box can collapse so that you can better view the presentation. To unhide the box, click the arrows on the top left corner

596 views • 59 slides
Online SNAP Basics NGAF Technical Assistance Center 6/29/20 The Nutrition Incentive Hub (NTAE)

Online SNAP Basics NGAF Technical Assistance Center 6/29/20 The Nutrition Incentive Hub (NTAE)

Online SNAP Basics NGAF Technical Assistance Center 6/29/20 The Nutrition Incentive Hub (NTAE) is supported by Gus Schumacher Nutrition Incentive Grant Program grant no. 2019-70030-30415/project accession no. 1020863 from the USDA National

480 views • 21 slides
A Deep Learning Approach to Automatic Call Routing Rajiv Shah Director of Solution Architect and

A Deep Learning Approach to Automatic Call Routing Rajiv Shah Director of Solution Architect and

A Deep Learning Approach to Automatic Call Routing Rajiv Shah Director of Solution Architect and Professional Services 2019 About GigaSpaces 300+ Direct customers We deliver the fastest big data 50+ / 500+ analytics processing platform to

731 views • 48 slides
E-commerce issues ROME 24 | 28 SEPTEMBER 2018 Mary Beth Garneau (Statisitics Canada) Erika

E-commerce issues ROME 24 | 28 SEPTEMBER 2018 Mary Beth Garneau (Statisitics Canada) Erika

E-commerce issues ROME 24 | 28 SEPTEMBER 2018 Mary Beth Garneau (Statisitics Canada) Erika Barrera (Central Bank of Chile) John Murphy (US Census Bureau) Ramon Bravo (INEGI Mexico) Cristina Cecconi, (Istat, Italy) Overview ROME 24 | 28

593 views • 10 slides
Partially-Ranked Choice Models for Data-Driven Assortment Optimization Sanjay Dominik Jena

Partially-Ranked Choice Models for Data-Driven Assortment Optimization Sanjay Dominik Jena

Partially-Ranked Choice Models for Data-Driven Assortment Optimization Partially-Ranked Choice Models for Data-Driven Assortment Optimization Sanjay Dominik Jena Andrea Lodi Hugo Palmer Canada Excellence Research Chair, andrea.lodi@polymtl.ca

874 views • 51 slides
Introduction Client Side Web Application Development George Corser, PhD Once upon a time...

Introduction Client Side Web Application Development George Corser, PhD Once upon a time...

Introduction Client Side Web Application Development George Corser, PhD Once upon a time... People wanted access to documents ... remotely from anywhere (so they invented network communications) ... without depending on a specific network

634 views • 41 slides
Teaching Players: Tutorial and opening mission design for Company of Heroes Aldric Sun Neil

Teaching Players: Tutorial and opening mission design for Company of Heroes Aldric Sun Neil

Teaching Players: Tutorial and opening mission design for Company of Heroes Aldric Sun Neil Jones-Rodway What were going to talk about Breaking the game down into teachable concepts and methods Designing and implementing tutorials to

378 views • 35 slides
Guiding Visual Search Tasks Using Gaze-Contingent Auditory Feedback Authors: Viktor Losing

Guiding Visual Search Tasks Using Gaze-Contingent Auditory Feedback Authors: Viktor Losing

Cognitive Interaction Technology Center of Excellence Guiding Visual Search Tasks Using Gaze-Contingent Auditory Feedback Authors: Viktor Losing Lukas Rottkamp Michael Zeunert Thies Pfeiffer CITEC Building at Bielefeld 1

580 views • 34 slides

More recommend