Uncoupling Michael T. Nygard @mtnygard About the Speaker Blog: - - PowerPoint PPT Presentation

Uncoupling

Michael T. Nygard @mtnygard

About the Speaker

Developer Architect Operations

Blog: http://www.michaelnygard.com/ Twitter: http://twitter.com/mtnygard

About the Speaker

• Stewards of Clojure & ClojureScript
• Creators of Datomic
• Open source contributions
• Pedestal
• Vase
• Onto
• Simulant

Tensegrity

No-tegrity

Coupling

Scharfenberg Coupler

Watt’s Linkage

Chebyshev Linkage

Standard Model Particle Interactions

Coupling

Coupling

• Determines degrees of freedom.

Coupling

• Determines degrees of freedom.
• Enables some movements.

Coupling

• Determines degrees of freedom.
• Enables some movements.
• Inhibits other movements.

Coupling

• Determines degrees of freedom.
• Enables some movements.
• Inhibits other movements.
• Connects effects.

Everything is coupled to everything

• Ambient temperature
• Ambient atmosphere
• Acoustic vibrations
• Electromagnetic field
• Gravity
• Higgs field

We cannot attend to all things all the time

Kinds of Coupling

Type Effect Operational Consumer cannot run without the provider Development Changes in producer and consumer must be coordinated Semantic Change together because of shared concepts Functional Change together because of shared responsibility Incidental Change together for no good reason. (E.g., breaking API changes.)

Analyzing Coupling I

E-mail system [Software System] Microsoft Exchange

Sends a notification that a report is ready to

Email Component [Component: C#] Sends emails

Analyzing Coupling I

E-mail system [Software System] Microsoft Exchange

Sends a notification that a report is ready to

Email Component [Component: C#] Sends emails

Operational: Strong. SMTP is synchronous, connection-oriented, conversational

Analyzing Coupling I

E-mail system [Software System] Microsoft Exchange

Sends a notification that a report is ready to

Email Component [Component: C#] Sends emails

Operational: Strong. SMTP is synchronous, connection-oriented, conversational Development: Weak. SMTP is well-defined standard with history of interoperability

Analyzing Coupling I

E-mail system [Software System] Microsoft Exchange

Sends a notification that a report is ready to

Email Component [Component: C#] Sends emails

Operational: Strong. SMTP is synchronous, connection-oriented, conversational Development: Weak. SMTP is well-defined standard with history of interoperability Semantic: Very strong. SMTP defines entities, attributes, and allowed values.

Analyzing Coupling I

E-mail system [Software System] Microsoft Exchange

Sends a notification that a report is ready to

Email Component [Component: C#] Sends emails

Operational: Strong. SMTP is synchronous, connection-oriented, conversational Development: Weak. SMTP is well-defined standard with history of interoperability Semantic: Very strong. SMTP defines entities, attributes, and allowed values. Functional: Very weak. Sender and MTA both use network connections.

Analyzing Coupling II-A

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

SQL connection to RDBMS

Analyzing Coupling II-A

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

SQL connection to RDBMS Operational: Very strong. Dependent on availability of server. Must be aware of topology and failover strategy

Analyzing Coupling II-A

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

SQL connection to RDBMS Operational: Very strong. Dependent on availability of server. Must be aware of topology and failover strategy Development: Very strong. Dependent on schema, server version, protocol version.

Analyzing Coupling II-A

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

SQL connection to RDBMS Operational: Very strong. Dependent on availability of server. Must be aware of topology and failover strategy Development: Very strong. Dependent on schema, server version, protocol version. Semantic: Very strong. Tables, columns, and joins must be known to both parties.

Analyzing Coupling II-A

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

Operational: Very strong. Dependent on availability of server. Must be aware of topology and failover strategy Development: Very strong. Dependent on schema, server version, protocol version. Semantic: Very strong. Tables, columns, and joins must be known to both parties. Functional: Weak. Functions of data maintenance don’t overlap with retrieval into objects. SQL connection to RDBMS

Suppose we use REST instead of SQL?

Analyzing Coupling II-B

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

HTTPS request to REST API

Analyzing Coupling II-B

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

Operational: Strong, but less than before. Dependent on availability of server. HTTPS request to REST API

Analyzing Coupling II-B

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

Operational: Strong, but less than before. Dependent on availability of server. Development: Strong, but less. Insulated from data format changes. Open encoding can further reduce coupling HTTPS request to REST API

Analyzing Coupling II-B

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

Operational: Strong, but less than before. Dependent on availability of server. Development: Strong, but less. Insulated from data format changes. Open encoding can further reduce coupling Semantic: Still very strong. REST resources and C# entities must align. Concepts will still map 1:1 HTTPS request to REST API

Analyzing Coupling II-B

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with

Gets counterparty data from

Reference Data Importer [Component: C#] Imports data from the reference data system

Operational: Strong, but less than before. Dependent on availability of server. Development: Strong, but less. Insulated from data format changes. Open encoding can further reduce coupling Semantic: Still very strong. REST resources and C# entities must align. Concepts will still map 1:1 Functional: Still weak. Different languages, techniques, design patterns apply. HTTPS request to REST API

What if we invert the flow of control?

Analyzing Coupling II-C

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with Reference Data Receiver [Component: C#] Accepts and caches data from the reference data system

Broadcasts

Message Broker [Software System] Pub/sub hub, bub

Broadcasts

Analyzing Coupling II-C

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with Reference Data Receiver [Component: C#] Accepts and caches data from the reference data system

Broadcasts

Message Broker [Software System] Pub/sub hub, bub

Broadcasts

Operational: Very weak. Receiver can run with stale data when either broker or upstream are broken.

Analyzing Coupling II-C

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with Reference Data Receiver [Component: C#] Accepts and caches data from the reference data system

Broadcasts

Message Broker [Software System] Pub/sub hub, bub

Broadcasts

Operational: Very weak. Receiver can run with stale data when either broker or upstream are broken. Development: Weak. Insulated from schema changes.

Analyzing Coupling II-C

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with Reference Data Receiver [Component: C#] Accepts and caches data from the reference data system

Broadcasts

Message Broker [Software System] Pub/sub hub, bub

Broadcasts

Operational: Very weak. Receiver can run with stale data when either broker or upstream are broken. Development: Weak. Insulated from schema changes. Semantic: Strong, but not as strong. Broker allows for remapping concepts.

Analyzing Coupling II-C

Reference Data System [Software System] Manages reference data for all counterparties the bank interacts with Reference Data Receiver [Component: C#] Accepts and caches data from the reference data system

Broadcasts

Message Broker [Software System] Pub/sub hub, bub

Broadcasts

Operational: Very weak. Receiver can run with stale data when either broker or upstream are broken. Development: Weak. Insulated from schema changes. Semantic: Strong, but not as strong. Broker allows for remapping concepts. Functional: Moderate. All components must share the same messaging tech.

Coupling was present in all examples – It is necessary and inescapable

Chains of Coupling

Chains of Coupling

Retek IBM PIM

Informatica

Everybody Else Dotcom Catalog

3rd Party Data Vendors Reviews, Ratings, Imagery Dotcom Promotions

Online Store Faceted Search SKU SKU SKU SKU SKU SKU SKU SKU

Chains of Coupling

Retek IBM PIM

Informatica

Everybody Else Dotcom Catalog

3rd Party Data Vendors Reviews, Ratings, Imagery Dotcom Promotions

Online Store Faceted Search SKU SKU SKU SKU SKU SKU SKU SKU Price Point

Chains of Coupling

Retek IBM PIM

Informatica

Everybody Else Dotcom Catalog

3rd Party Data Vendors Reviews, Ratings, Imagery Dotcom Promotions

Online Store Faceted Search SKU SKU SKU SKU SKU SKU SKU SKU Price Point Price Point Price Point Price Point Price Point Price Point Price Point

Chains of Coupling – Semantic Polymer

Retek IBM PIM

Informatica

Everybody Else Dotcom Catalog

3rd Party Data Vendors Reviews, Ratings, Imagery Dotcom Promotions

Online Store Faceted Search SKU SKU SKU SKU SKU SKU SKU SKU Price Point Price Point Price Point Price Point Price Point Price Point Price Point

Long Arrows

Each “interface” was really a chain

1. Extract tables to files 2. Push files across network 3. Load tables into “LZ” 4. Process into “cold” DB 5. Swap hot & cold DBs (hours later) 1. Send message to queue 2. Take message from queue, unwrap, inspect, and dispatch to 1-of-N other queues. 3. Drain queue to file 4. Batch job wakes up 2 times a day, does FTP to remote end 5. Another batch job pulls a reconciliation file, drops file into file system 6. Parser reads the file, shreds it into messages, puts them on another queue

Architecture Qualities in Long Chains

Losses accumulate:

• Latency strictly worse than the slowest link in the chain.
• Availability strictly worse than the least available link.
• Throughput strictly worse than the throughput of the worst bottleneck
• Security strictly worse than the security of the weakest link

Digging Out

Information Hiding

“On the Criteria To Be Used in Decomposing Systems into Modules", David Parnas, CACM, 1972

A KWIC Example

• Input

Software comprises an endless supply of structures.

• Output

an endless supply of structures. Software comprises comprises an endless supply of structures. Software endless supply of structures. Software comprises an

• f structures. Software comprises an endless supply

Software comprises an endless supply of structures.

• structures. Software comprises an endless supply of

supply of structures. Software comprises an endless

Modularization 1

1. Input Read EBCDIC characters, store them in core. 6-bit characters packed 4 per word. EOL is a special character. 2. Circular shifter Prepare index; pair of addr of first char of shift, original index of line in input array 3. Alphabetizer Take arrays from 1 & 2, produce new array of pairs like in 2, but in alphabetical order. 4. Output Using arrays from 1 & 3, format output 5. Control Allocate memory, call operations in1 - 4, report errors.

Consider the Effect of Changes

For each change case listed here, how many modules have to be changed?

• Read and print ASCII instead of EBCDIC.
• Stop using packed characters, store one character per word.
• Write index for circular shifts to offline storage instead of core to support larger

input documents.

Modularization 2

1. Line Storage Offers functional interface: SETCH, GETCH, GETW, DELW, DELLINE 2. Input Reads EBCDIC chars, calls line storage to put them into lines. 3. Circular Shifter Offers same interface as line storage. Makes it appear to have all shifts of all lines. 4. Alphabetizer Offers sort function INIT, and access function iTH that gets a line. 5. Output Repeatedly call iTH on alphabetizer, printing the line. 6. Control Similar to first approach, call each module in sequence.

Consider the Effect of Changes

For each change case listed here, how many modules have to be changed?

• Read and print ASCII instead of EBCDIC.
• Stop using packed characters, store one character per word.
• Write index for circular shifts to offline storage instead of core to support larger

input documents.

Why is the second one better?

• It hides decisions inside modules.
• Functional interfaces provide an abstract representation of the underlying data.
• Information hiding

Line Line Line Line Line Ordered Ordered Storage Input Circular Shifter Alphabetizer Output

"Needs an interface" "Offers an interface"

Why did the second modularization survive change better?

• Very few data types
• Small number of well

defined interfaces

• Highly composable

Line Line Line Line Line Ordered Storage Input Circular Shifter Output Ordered Alphabetizer Controller run run Creates all components

Composing Interfaces

Line Line Line Line Line Ordered Storage Input Circular Shifter Output Ordered Alphabetizer Controller run run Creates all components

Composing Interfaces

Consider each connector with respect to Operational, Development, Semantic, and Functional coupling

Relating Composability

Inversely proportional to # of interfaces Inversely proportional to # of data types

Uncoupling

Orthogonality in Software

• Separation of concerns
• High cohesion within a module or component
• Low coupling between modules or components
• Little overlap in functionality between modules
• Information hiding / decision hiding

Batch Process

Batch Process

Risk calculator produces a data structure that the report generator must consume. Example from c4model.com

Batch Process

Risk calculator produces a data structure that the report generator must consume. Data importers probably have similar implementation needs Example from c4model.com

Batch Process

Risk calculator produces a data structure that the report generator must consume. Data importers probably have similar implementation needs Report checker doesn’t appear to connect with the file system that holds the reports. FS location is latent coupling that will be a nasty surprise later. Example from c4model.com

Batch Process

Risk calculator produces a data structure that the report generator must consume. Data importers probably have similar implementation needs Report checker doesn’t appear to connect with the file system that holds the reports. FS location is latent coupling that will be a nasty surprise later. Orchestrator might end need to do lots of data transformation to bridge interfaces. Example from c4model.com

Batch Process

Risk calculator produces a data structure that the report generator must consume. Data importers probably have similar implementation needs Report checker doesn’t appear to connect with the file system that holds the reports. FS location is latent coupling that will be a nasty surprise later. Orchestrator might end need to do lots of data transformation to bridge interfaces. Example from c4model.com

Let’s explore options to uncouple these

Problem: Risk calculator produces a data structure that the report generator must consume. Solutions depend on architectural style Here we’re in a Windows service so we might use a shared library to define the interface.

Orchestrator [Component: C#] Orchestrates the risk calculation process Risk Calculator [Component: C#] Does math Report Generator [Component: C# and Microsoft.Office.Interop.Excel] Generates an Excel compatible risk report

Generates the risk report using Calculates risk using

Problem: Redundant implementation details This would be a good place to use a shared library for common implementation.

Trade Data Importer [Component: C#] Imports data from the trade data system Reference Data Importer [Component: C#] Imports data from the reference data system Orchestrator [Component: C#] Orchestrates the risk calculation process

Imports data using Imports data using

File System [Container: Network File Share] Stores risk reports Report Distributor [Component: C#] Publishes the report for the web application

Publishes risk reports to

Report Checker [Component: C#] Checks that the report has been generated by 9 a.m. Singapore time Alerter [Component: C# with SNMP library] Sends SNMP alerts

Sends alerts using

Scheduler [Component: Quartz.net] Starts the risk calculation process

Starts Publishes the risk report using

Problem: Latent coupling about filesystem layout. Solution: A module to hide the decision about filesystem layout from both the Report Distributor and the Report Checker

Just Used Libraries in 3 Ways

Just Used Libraries in 3 Ways

• Define meeting point (interface)

Just Used Libraries in 3 Ways

• Define meeting point (interface)
• Extract repeated functionality

Just Used Libraries in 3 Ways

• Define meeting point (interface)
• Extract repeated functionality
• Isolate shared concept/decision

Find solutions by rotating your perspective

• When looking at components, think about modules
• When looking at modules, think about components
• When looking at data, think about code
• When looking at code, think about data

Use all your tools

1. Module structure – layout of your code and libraries 2. Component structure – interactions between runtime components 3. Abstraction – Emphasize similar interfaces & data formats

Coupling

• Determines degrees of freedom.
• Enables some movements.
• Inhibits other movements.
• Connects effects.

Kinds of Coupling

• Operational
• Development
• Semantic
• Functional

Summary

• Hide decisions
• Choose degrees of freedom that matter
• Avoid semantic polymers
• Use static and dynamic structures
• Find more instances of fewer, more general interfaces
• Prefer explicit to tacit

Uncoupling

Michael T. Nygard @mtnygard