designing for discovery in the era of data intensive

DESIGNING FOR DISCOVERY IN THE ERA OF DATA-INTENSIVE ASTRONOMY Sarah - PowerPoint PPT Presentation

DESIGNING FOR DISCOVERY IN THE ERA OF DATA-INTENSIVE ASTRONOMY Sarah Hegarty with A/Prof Christopher Fluke, Dr Aidan Hotan (CSIRO), & Dr Amr Hassan (Monash) Melbourne University | August 29th, 2018 Making Discoveries in Astronomy


  1. The Discovery Workflow Telescopes + Data Reduction Data Analysis (aka “thinking”) Compare with models, other = data, and publish Contextualise, and debate with colleagues Adapted from Norris (2010) Sarah Hegarty | Melbourne University | August 29th, 2018

  2. The Discovery Workflow Telescopes Data + Information Data Reduction Data Analysis Knowledge (aka “thinking”) Compare with models, other = Understanding data, and publish Contextualise, and debate with Wisdom colleagues Adapted from Norris (2010) Sarah Hegarty | Melbourne University | August 29th, 2018

  3. The Discovery Workflow Telescopes Data + Information Data Reduction Data Analysis New Knowledge (aka “thinking”) Compare with models, other = New Understanding data, and publish Contextualise, and debate with New Wisdom colleagues Adapted from Norris (2010) Sarah Hegarty | Melbourne University | August 29th, 2018

  4. What Do We Know About Making Discoveries in Astronomy? Sarah Hegarty | Melbourne University | August 29th, 2018

  5. What Do We Know About Making Discoveries in Astronomy? Discoveries strongly follow technological developments that ● open up new parameter space Sarah Hegarty | Melbourne University | August 29th, 2018

  6. What Do We Know About Making Discoveries in Astronomy? Discoveries strongly follow technological developments that ● open up new parameter space Many of the most exciting discoveries are serendipitous ● Sarah Hegarty | Melbourne University | August 29th, 2018

  7. What Do We Know About Making Discoveries in Astronomy? Discoveries strongly follow technological developments that ● open up new parameter space Many of the most exciting discoveries are serendipitous ● Visual inspection of the data can be invaluable ● Sarah Hegarty | Melbourne University | August 29th, 2018

  8. What Do We Know About Making Discoveries in Astronomy? Discoveries strongly follow technological developments that ● open up new parameter space Many of the most exciting discoveries are serendipitous ● Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the ● instrument, are crucial in recognising something new Sarah Hegarty | Melbourne University | August 29th, 2018

  9. What Do We Know About Making Discoveries in Astronomy? Discoveries strongly follow technological developments that ● open up new parameter space Many of the most exciting discoveries are serendipitous ● Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the ● instrument, are crucial in recognising something new → Developing this expertise through training is key Sarah Hegarty | Melbourne University | August 29th, 2018

  10. What Do We Know About Making Discoveries in Astronomy? Discoveries strongly follow technological developments that ● open up new parameter space Many of the most exciting discoveries are serendipitous ● Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the ● instrument, are crucial in recognising something new → Developing this expertise through training is key ● Discoveries are the end result of effective workflows Sarah Hegarty | Melbourne University | August 29th, 2018

  11. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  12. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  13. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  14. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  15. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  16. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  17. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  18. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  19. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  20. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  21. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  22. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  23. What Do We Know About Making Discoveries in Astronomy? Planning Discoveries strongly follow technological developments that ● Technological open up new parameter space Development Training Many of the most exciting discoveries are serendipitous ● + Visual inspection of the data can be invaluable ● ~20Tb/night Individual expertise, and familiarity with the data and the Expertise ● instrument, are crucial in recognising something new → Developing this expertise through training is key = ● Discoveries are the end result of effective workflows Visualisation Serendipity capabilities Sarah Hegarty | Melbourne University | August 29th, 2018

  24. How can we capitalise on the discovery potential of data-intensive astronomy? → Understand how we make discoveries Sarah Hegarty | Melbourne University | August 29th, 2018

  25. How can we capitalise on the discovery potential of data-intensive astronomy? → Understand how we make discoveries → Use this understanding to “design in” discovery when we build data-intensive workflows Sarah Hegarty | Melbourne University | August 29th, 2018

  26. Designing Effective Discovery Workflows Automated pipelines and machine-learning approaches are essential for data-intensive astronomy but We must integrate a role for the human astronomer alongside automated methods to maintain discovery mechanisms that we know to be important Manual Inspection 60% 40% 0% 100% 80% 20% 0% 20% 60% 40% 80% 100% Automated Inspection Fine-tune to maximise discovery Adapted from Fluke et al. (2016) Sarah Hegarty | Melbourne University | August 29th, 2018

  27. Responding to the Data-Intensive Discovery Challenge Director: A/Prof Christopher Fluke Sarah Hegarty | Melbourne University | August 29th, 2018

  28. Responding to the Data-Intensive Discovery Challenge Director: A/Prof Christopher Fluke Sarah Hegarty | Melbourne University | August 29th, 2018

  29. Responding to the Data-Intensive Discovery Challenge Sarah Hegarty | Melbourne University | August 29th, 2018

  30. Responding to the Data-Intensive Discovery Challenge Designing Out Data Artefacts: Better Beamforming for ASKAP Sarah Hegarty | Melbourne University | August 29th, 2018

  31. Responding to the Data-Intensive Discovery Challenge Designing Out Data Artefacts: Building eResearch Workflows: Better Beamforming for ASKAP Theoretical Astrophysical Observatory and: Deeper, Wider, Faster Sarah Hegarty | Melbourne University | August 29th, 2018

  32. Responding to the Data-Intensive Discovery Challenge Understanding the Astronomer’s Role: Designing Out Data Artefacts: Building eResearch Workflows: Better Beamforming for ASKAP PerSieve Theoretical Astrophysical Observatory and: Deeper, Wider, Faster Sarah Hegarty | Melbourne University | August 29th, 2018

  33. A Case Study: Deeper, Wider, Faster A detection and follow-up program for fast transients (Cooke+, in prep.) Sarah Hegarty | Melbourne University | August 29th, 2018

  34. A Case Study: Deeper, Wider, Faster A detection and follow-up program for fast transients (Cooke+, in prep.) Nugent, 2015 Sarah Hegarty | Melbourne University | August 29th, 2018

  35. A Case Study: Deeper, Wider, Faster A detection and follow-up program for fast transients (Cooke+, in prep.) Targets transients on timescales from ❏ hours down to seconds Nugent, 2015 Sarah Hegarty | Melbourne University | August 29th, 2018

  36. A Case Study: Deeper, Wider, Faster A detection and follow-up program for fast transients (Cooke+, in prep.) Targets transients on timescales from ❏ hours down to seconds Aims to achieve real-time, ❏ multiwavelength observations, and rapid multiwavelength follow up Nugent, 2015 Sarah Hegarty | Melbourne University | August 29th, 2018

  37. A Case Study: Deeper, Wider, Faster A detection and follow-up program for fast transients (Cooke+, in prep.) Targets transients on timescales from ❏ hours down to seconds Aims to achieve real-time, ❏ multiwavelength observations, and rapid multiwavelength follow up Courtesy J. Cooke Sarah Hegarty | Melbourne University | August 29th, 2018

  38. A Case Study: Deeper, Wider, Faster A detection and follow-up program for fast transients (Cooke+, in prep.) Targets transients on timescales from ❏ hours down to seconds Aims to achieve real-time, ❏ multiwavelength observations, and rapid multiwavelength follow up Courtesy J. Cooke Sarah Hegarty | Melbourne University | August 29th, 2018

  39. A Case Study: Deeper, Wider, Faster Andreoni & Cooke, 2018 Sarah Hegarty | Melbourne University | August 29th, 2018

  40. A Case Study: Deeper, Wider, Faster Figure: Meade+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  41. A Case Study: Deeper, Wider, Faster 3 square degree FOV ~60 CCD images / 40 seconds 2048 x 4096 pixels each Figure: Meade+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  42. A Case Study: Deeper, Wider, Faster 3 square degree FOV ~60 CCD images / 40 seconds JPEG2000 data compression 2048 x 4096 pixels each (Vohl+, 2017) Figure: Meade+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  43. A Case Study: Deeper, Wider, Faster 3 square degree FOV ~60 CCD images / 40 seconds JPEG2000 data compression ‘Mary’ data reduction 2048 x 4096 pixels each (Vohl+, 2017) pipeline (Andreoni+, 2017) Figure: Meade+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  44. A Case Study: Deeper, Wider, Faster Andreoni+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  45. A Case Study: Deeper, Wider, Faster Visual inspection by 3 square degree FOV volunteer astronomers ~60 CCD images / 40 seconds JPEG2000 data compression ‘Mary’ data reduction 2048 x 4096 pixels each (Vohl+, 2017) pipeline (Andreoni+, 2017) Figure: Meade+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  46. A Case Study: Deeper, Wider, Faster Meade+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  47. A Case Study: Deeper, Wider, Faster Photos courtesy B. Meade Sarah Hegarty | Melbourne University | August 29th, 2018

  48. A Case Study: Deeper, Wider, Faster Integrating the visualisation, analysis and assessment work of volunteer astronomers as part of the DWF workflow would allow us to: Photos courtesy B. Meade Sarah Hegarty | Melbourne University | August 29th, 2018

  49. A Case Study: Deeper, Wider, Faster Integrating the visualisation, analysis and assessment work of volunteer astronomers as part of the DWF workflow would allow us to: Continue capitalising on the expertise and crucial discovery skills of these astronomers ❏ Photos courtesy B. Meade Sarah Hegarty | Melbourne University | August 29th, 2018

  50. A Case Study: Deeper, Wider, Faster Integrating the visualisation, analysis and assessment work of volunteer astronomers as part of the DWF workflow would allow us to: Continue capitalising on the expertise and crucial discovery skills of these astronomers ❏ Simplify and streamline the discovery workflow, and remove margin for error ❏ Photos courtesy B. Meade Sarah Hegarty | Melbourne University | August 29th, 2018

  51. A Case Study: Deeper, Wider, Faster Integrating the visualisation, analysis and assessment work of volunteer astronomers as part of the DWF workflow would allow us to: Continue capitalising on the expertise and crucial discovery skills of these astronomers ❏ Simplify and streamline the discovery workflow, and remove margin for error ❏ Better understand the discovery process itself ❏ Photos courtesy B. Meade Sarah Hegarty | Melbourne University | August 29th, 2018

  52. A Case Study: Deeper, Wider, Faster Andreoni+, 2017 Sarah Hegarty | Melbourne University | August 29th, 2018

  53. PerSieve Sarah Hegarty | Melbourne University | August 29th, 2018

  54. PerSieve An application for interactive visualisation and assessment - in real time, in the browser ❏ Sarah Hegarty | Melbourne University | August 29th, 2018

  55. PerSieve An application for interactive visualisation and assessment - in real time, in the browser ❏ Integrates visualisation and the human astronomer into DWF’s automated pipeline ❏ Sarah Hegarty | Melbourne University | August 29th, 2018

  56. PerSieve An application for interactive visualisation and assessment - in real time, in the browser ❏ Integrates visualisation and the human astronomer into DWF’s automated pipeline ❏ Sarah Hegarty | Melbourne University | August 29th, 2018

  57. February 2018 DWF Observing Campaign During a four-night, Subaru-led DWF observing campaign, PerSieve was used successfully as ❏ the primary visualisation and analysis tool Sarah Hegarty | Melbourne University | August 29th, 2018

  58. February 2018 DWF Observing Campaign During a four-night, Subaru-led DWF observing campaign, PerSieve was used successfully as ❏ the primary visualisation and analysis tool Over 30 astronomers participated on-site ❏ Sarah Hegarty | Melbourne University | August 29th, 2018

Recommend


More recommend