Tables Zero-dimensional data with position R.W. Oldford University - - PowerPoint PPT Presentation
Tables Zero-dimensional data with position R.W. Oldford University of Waterloo Magnitude Visual representations - labels Recall from last day, the ordering of the elementary tasks from most accurate to least accurate: 1. Position along a
1. Position along a common scale 2. Position on identical but nonaligned scales 3. Lengths (N.B. line segments were all oriented horizontally or vertically, though nonaligned) 4. Angle Slope (not close to 0, π/2, or π radians) 5. Area 6. Volume 7. Colour density, colour saturation 8. Colour hue
left centre right decimal aligned rounded
unordered ascending descending
unordered ascending descending
Multiplication Long division
Algorism: “technique of performing basic arithmetic by writing numbers in place value form and applying a set
One of the earliest written number systems was that of cuneiform, used by the Babylonians (circa 2500 BCE). A wedge-shape (cuneus in Latin) tipped reed is pressed into wet clay and a single stroke indicates a single unit, the first number 1. (No zero.) A simple visual representation with a standardized layout of the symbols. 1 2 3 4 5 6 7 8 9 Needs a compression for larger numbers. 10 20 30 40 50 Ten is like two hands pressed together. Note standardized layout. Ends at 50.
10 11 12 · · · 19 Which is nice up until 59. After that positional representation is used! E.g. 70 =
◮ few characters to learn. ◮ Other bases, e.g. 12, might be even better. Babylonians (cuneiform) uses base 60.
◮ character encodes value ◮ position encodes magnitude (increasing right to left)
◮ align in columns ◮ sequences separate in groups of 3 Allows them to be used dynamically as visual aids to reasoning.
Visually executed algorithms rely on position (e.g. multiplication, long division, . . . )
Natural grouping by position
Several pass sorting (by first digit, then second, . . . )
(a) Balance sheet (b) Pythagorean triplets
Background: One of the most pressing environmental problems facing large areas of North America is acidic precipitation. All of southern Ontario receives a steady bombardment of acids, acid forming gases and associated pollutants. The acids come down with rain, snow, fog, and small particles in the air. Man’s activities are responsible for the large majority of these acids. Smelters and coal-fired electric generating stations both in Canada and the United States spew millions of tonnes of sulphur dioxide into the atmosphere
sunlight, oxygen, ozone, water and other gases to form sulphuric and nitric acid – strong, corrosive acids. In unpolluted areas, rain and snow are naturally slightly acidic since carbon dioxide, which is a natural component of the atmosphere, dissolves in water to form weak carbonic acid. Water quality of lakes has developed in response to weathering processes induced by this weak acid. Rocks and minerals react with carbonic acid to form bicarbonate, which is found in natural waters everywhere. The complex biological communities in lakes, streams and forests have adapted and evolved in equilibrium with these natural conditions and processes. However, acid rain has seriously disturbed this equilibrium. There are over 250,000 lakes in Ontario. Thousands have been affected by acid rain, many of them in the Muskoka-Haliburton region, where there is a substantial cottage and tourist industry.
Ontario Government Publication: “Acid Sensitivity Survey of Lakes in Ontario – 1989”.
Rows ordered alphabetically by “County
Lots of redundant information.
Difficult to see patterns, if any. Ontario Government Publication: “Acid Sensitivity Survey of Lakes in Ontario – 1989”.
Arrange by region, remove uninteresting redundancy.
Arrange by acidity.
Arrange by acidity, remove further redundancy, annotate.
TABLE 1.1 Data in Four Areas and Eight Three-Month Periods in 1969-1970. 13-15 16-18 19-21 22-24 25-27 28-30 31-33 34-36 A 97.62 92.24 100.90 90.39 95.69 94.44 91.13 97.81 B 48.29 42.31 49.98 39.09 46.38 49.74 41.74 37.39 C 75.23 75.16 100.11 74.23 74.23 76.97 71.66 76.47 D 49.69 57.21 80.19 51.09 52.88 49.41 59.32 52.56
Goal is to see some patterns in the data.
Develop a summary description (“model”) for the pattern.
Assess the agreement of the pattern with the data. Source: A.S.C. Ehrenberg (1975) Data Reduction: Analysing and Interpreting Statistical Data.
13-15 16-18 19-21 22-24 25-27 28-30 31-33 34-36 A 97.62 92.24 100.90 90.39 95.69 94.44 91.13 97.81 B 48.29 42.31 49.98 39.09 46.38 49.74 41.74 37.39 C 75.23 75.16 100.11 74.23 74.23 76.97 71.66 76.47 D 49.69 57.21 80.19 51.09 52.88 49.41 59.32 52.56 TABLE 1.1 Data in Four Areas and Eight Three-Month Periods in 1969-1970.
Quarters (1969) Quarters (1970) Area 1 2 3 4 1 2 3 4 North 97.62 92.24 100.90 90.39 95.69 94.44 91.13 97.81 South 48.29 42.31 49.98 39.09 46.38 49.74 41.74 37.39 East 75.23 75.16 100.11 74.23 74.23 76.97 71.66 76.47 West 49.69 57.21 80.19 51.09 52.88 49.41 59.32 52.56
Quarters (1969) Area 1 2 3 4 North 97.62 92.24 100.90 90.39 South 48.29 42.31 49.98 39.09 East 75.23 75.16 100.11 74.23 West 49.69 57.21 80.19 51.09 ⇒ Quarters (1969) Area 1 2 3 4 North 98 92 101 90 South 48 42 50 39 East 75 75 100 74 West 50 57 80 51 It is a common mistake to present too many digits. Number of significant digits is typically 1, 2 or 3. It may require a change in units of measurement being displayed (e.g. 100, 000s of dollars rather than 1000s). Patterns?
Because we are looking for patterns in the table, reducing to “significant” digits may not always mean the usual “scientifically significant digits”. For example, had the table been, say Quarters (1969) Area 1 2 3 4 North 12345097.62 12345092.24 12345100.90 12345090.39 South 12345048.29 12345042.31 12345049.98 12345039.09 East 12345075.23 12345075.16 12345100.11 12345074.23 West 12345049.69 12345057.21 12345080.19 12345051.09 We still would have wanted the table below: Quarters (1969) Area 1 2 3 4 North 98 92 101 90 South 48 42 50 39 East 75 75 100 74 West 50 57 80 51 Because the first five significant digits are iden- tical any pattern in the table will be in the re- maining digits. Simply subtract 1,234,500 from every entry and report a location 1,234,500 to accompany the table.
Quarters (1969) Area 1 2 3 4 Total North 98 92 101 90 381 South 48 42 50 39 179 East 75 75 100 74 324 West 50 57 80 51 238 Total 271 266 331 254 1122 Patterns? Sums are on a different scale (magnitude) than the points within the table.
Quarters (1969) Area 1 2 3 4 Ave. North 98 92 101 90 95 South 48 42 50 39 45 East 75 75 100 74 81 West 50 57 80 51 60 Average 68 67 83 64 70 Patterns? Quarterly averages don’t differ much from overall average. Area averages differ more. Area values also do not differ much from the Areas averaged across quarters.
Quarters (1969) Area 1 2 3 4 Ave. North 98 92 101 90 95 South 48 42 50 39 45 East 75 75 100 74 81 West 50 57 80 51 60 Average 68 67 83 64 70 Area 1969 North South East West Ave. Q1 98 48 75 50 68 Q2 92 42 75 57 67 Q3 101 50 100 80 83 Q4 90 39 74 51 64 Average 95 45 81 60 70 Note how much easier it is to scan down the North column to see the little variability in the leading digit. Exceptions now stand out more easily – the 100 in the East column and the 80 in the West. Similarly Q3 overall. More widely spaced rows would diminish this visual advantage.
Area 1969 South West East North Ave. Q1 48 50 75 98 68 Q2 42 57 75 92 67 Q3 50 80 100 101 83 Q4 39 51 74 90 64 Average 45 60 81 95 70 Columns ordered from smallest to largest. Much simpler to see the same decreasing pattern occurs in all quarters. Could rearrange rows, except: not much variation across quarters & the quarters have a meaningful time order. For rearranged rows, place largest at top. It’s easier to look at subtractive difference between rows.
Area 1969 South West East North Ave. Q1 48 50 75 98 68 Q2 42 57 75 92 67 Q3 50 80 100 101 83 Q4 39 51 74 90 64 Average 45 53∗ 75∗ 95 67∗
∗ Excluding Q3 in West and East.
Exceptions coloured brown. Could also indicate with parentheses, i.e. 100 or (100).
Compare where we started, with where we ended Original 1969 data 13-15 16-18 19-21 22-24 A 97.62 92.24 100.90 90.39 B 48.29 42.31 49.98 39.09 C 75.23 75.16 100.11 74.23 D 49.69 57.21 80.19 51.09 Area 1969 South West East North Ave. Q1 48 50 75 98 68 Q2 42 57 75 92 67 Q3 50 80 100 101 83 Q4 39 51 74 90 64 Average 45 53∗ 75∗ 95 67∗
∗ Excluding Q3 in West and East.
Four areas: North 95, East 75, West 53, and South 45. There were two exceptionally high values in Q3 from the East and West regions, differing from their averages by about 25 units. Area 1969 South West East North Ave. Q1 48 50 75 98 68 Q2 42 57 75 92 67 Q3 50 80 100 101 83 Q4 39 51 74 90 64 Average 45 53∗ 75∗ 95 67∗
∗ Excluding Q3 in West and East.
1. Base rules ◮ Reduce number of digits. (N.B. could require finding a common location) Mental arithmetic is more difficult with more than two significant (varying) digits. ◮ Figures to be compared should be close together. ◮ Use memorable self-explanatory symbols and labels. ◮ Separate different types of items/groups with white space or gridlines. 2. Calculations ◮ Avoid introducing new variables or scales (e.g. totals) whenever possible. ◮ Use averages (or medians) to help focus the eye over the array. ◮ Note dramatically exceptional values and exclude them from pattern summary calculations. 3. If possible swap rows and columns, reorder rows and/or columns: ◮ Numbers that vary the least should appear in columns. Both regularities and exceptions are easier to spot. ◮ Rearrange rows to have large numbers appear above small numbers. Differences should easier to detect following subtraction rules. ◮ Rearrange columns so that averages are strictly decreasing (or increasing) from left to right. Easier then to detect departures from this pattern within the table.
Area 1969 South West East North Ave. Q1 3
3 1 Q2
4
Q3 5 27 25 6 6∗ Q4
Average 0∗ 0∗ 0∗
∗ Excluding Q3 in east and West.
Model has South 45, West 53, East 75, and North 95. Comments? Column averages must be zero, they are from the model. Note rounding effects.
Model has North 95, East 75, West 53, South 45 Deviations are: Area 1969 South West East North Ave. Q1 3
3 1 Q2
4
Q3 5 27 25 6 6∗ Q4
Average 0∗ 0∗ 0∗
∗ Excluding Q3 in West and East.
Summarize size of deviations: Use average absolute deviation. Area 1969 South West East North Ave.
4 3∗ 0∗ 4 3
∗ Excluding Q3 in West and East.
1. Base rules ◮ Reduce number of digits. Mental arithmetic is more difficult with more than two significant (varying) digits. ◮ Figures to be compared should be close together. ◮ Use memorable self-explanatory symbols and labels. ◮ Separate different types of items/groups with white space or gridlines. 2. Calculations ◮ Avoid introducing new variables or scales (e.g. totals) whenever possible. ◮ Use averages (or medians) to help focus the eye over the array. ◮ Note dramatically exceptional values and exclude them from pattern summary calculations. 3. If possible swap rows and columns, reorder rows and/or columns: ◮ Numbers that vary the least should appear in columns. Both regularities and exceptions are easier to spot. ◮ Rearrange rows to have large numbers appear above small numbers. Differences should easier to detect following subtraction rules. ◮ Rearrange columns so that averages are strictly decreasing (or increasing) from left to right. Easier then to detect departures from this pattern within the table. 4. Summarize irregular aspects of the data statistically, e.g. by average deviations from appropriate averages.
We started with two years data, 1969 and 1970, but only built a model based on the first of these. Can we apply what we learned about 1969 directly to 1970? If the model works for both years, we have in some sense validated it. The possibility of an underlying structure, described by our model, would seem to be connected with reality (cf. pictorial form) Applying the model amounts to building a final table for 1970 according to the same table organization. For 1970 this yields Area 1970 South West East North Ave. Q1 46 53 74 96 67 Q2 50 49 77 94 68 Q3 42 59 72 91 66 Q4 37 53 76 98 66 Average 44 54 75 95 67 which tells essentially the same story.
Area 1969 South West East North Ave. Q1 48 50 75 98 68 Q2 42 57 75 92 67 Q3 50 80 100 101 83 Q4 39 51 74 90 64 Average 45 53∗ 75∗ 95 67∗
∗ Excluding Q3 in West and East.
Area 1970 South West East North Ave. Q1 46 53 74 96 67 Q2 50 49 77 94 68 Q3 42 59 72 91 66 Q4 37 53 76 98 66 Average 44 54 75 95 67 Consistent results across both years.
Area 1969 South West East North Ave. Q1 3
3 1 Q2
4
Q3 5 27 25 6 6∗ Q4
Average 0∗ 0∗ 0∗
∗ Excluding Q3 in West and East.
Area 1970 South West East North Ave. Q1 2
1 Q2 6
2
Q3
5
Q4
1 1 3 Average No regular patterns, average absolute deviation about the same (≈ 3)
Averaging the deviations over the two years: Average of Area 1969 & ’70 South West East North Ave. Q1 2
2 1 Q2 2 1
Q3 2 3∗ −2∗ 2 1 Q4
Average
∗ Excluding Q3 in 1969.
Still no regular patterns, average absolute deviation about 2. (Note that 3/ √ 2 ≈ 2.12.)
Consistent model for averages (over both years): Area South West East North Ave. 1969 45 53∗ 75∗ 95 67∗ 1970 44 54 75 95 67
Consistently patternless, irregular deviations (over both years). ◮ Average deviation of zero. (Force of calculation.) ◮ Average absolute deviation the same over each year (about 3). ◮ Consistently patternless deviations over quarters and areas for each year and over both years. It is this consistently irregular pattern of deviations that indicates the ability to generalize the consistent averages of areas to other, unseen, years. Analysis of our tabular model suggests that it can be used to predict.