Telecommuting 11 5 time zones Telecommuting 11.5 time zones Tilottama - - PowerPoint PPT Presentation

Telecommuting 11 5 time zones Telecommuting 11.5 time zones

Tilottama Ghosh and Kimberly Baugh

(CIRES, University of Colorado, Boulder, USA)

Asia Pacific Advanced Network 32nd Meeting

( , y , , )

Asia‐Pacific Advanced Network – 32nd Meeting New Delhi, India 26th August, 2011

Brief employment history and Migration decision

• Started as an intern with the Earth Observation Group (EOG) at the

National Geophysical Data Center (NGDC), National Oceanic and Atmospheric Administration (NOAA) in June 2007 Atmospheric Administration (NOAA), in June, 2007.

• Continued working as a part‐time employee while pursuing a PhD

degree in Geography from the University of Denver degree in Geography from the University of Denver

• Became a doctorate in June of 2010
• Decided to get married in December, 2010 and migrate back to New

Delhi, India.

• Became a full‐time employee at NGDC from August of 2010 and

expressed the desire to continue working part‐time with the Earth Observation Group even after returning to India Observation Group even after returning to India

• Permission was granted, paper work was done, and then all the

arrangements had to be made so that I could work smoothly even while g y being located 11.5 time zones away!

11.5 time zones away – Mountain time zone and Indian Standard time

Time difference increases to 12 and a half hours when the daylight saving time ends in November in the U.S.

Arrangements made while in Boulder, Colorado

A good laptop ‐

• A good laptop which would be able

to handle the download and processing of huge datasets processing of huge datasets

• HP Pavilion dv4 notebook PC
• CPU processing speed and Random

A M (RAM) Access Memory (RAM) were important considerations

• The notebook has Intel Core i3 64

bit processor, and an installed memory of 4 GB

Arrangements made while in Boulder, Colorado

I t lli P TTY Installing PuTTY ‐

• PuTTY is a client program for the SSH, Telnet and Rlogin network

protocols

• These protocols are used to run a remote session on a computer,

These protocols are used to run a remote session on a computer,

• ver a network
• How it works – you run PuTTY on a Windows machine and tell it to
• How it works – you run PuTTY on a Windows machine, and tell it to

connect to a Linux machine. PuTTY opens a window. Anything you type into that window is sent straight to the Linux machine, and yp g , everything the Linux machine sends back is displayed in the window

• PuTTY was installed on my computer so that I could remotely access

PuTTY was installed on my computer so that I could remotely access NOAA’s Linux machines through NGDC’s portal. Through NGDC’S portal I can connect to Process 1 and Process 2, two of the machines which has all the data that I access and download for processing

PuTTY

Arrangements made while in Boulder, Colorado

Installing Secure FX‐

• Secure FX is a secure file transfer application with a visual interface
• Secure FX is used to exchange files between NGDC’s machine and

my computer

Secure FX

Arrangements made while in Boulder, Colorado

• For viewing, processing , and analyzing data ArcGIS and ENVI were

Installing ArcGIS and ENVI ‐

g, p g , y g required

• ENVI is an image processing software

package produced by the ITT Visual Information Solutions

• Premier software solution used for

processing and analyzing geospatial processing and analyzing geospatial imagery

• Combines spectral image processing

and image analysis technology with a user‐friendly interface

• Interactive Data Language (IDL) is the

Interactive Data Language (IDL) is the scientific programming language associated with ENVI that lets users transform numbers into dynamic and transform numbers into dynamic and meaningful visual representations

ArcGIS

• ArcGIS is a Geographic

Information System (GIS) tool produced by the Environmental produced by the Environmental Systems Research Institute (ESRI)

• It is a powerful tool used for
• It is a powerful tool used for

spatial analysis

Arrangements that had to be made in Delhi

Setting up broadband internet connections at home in New Setting up broadband internet connections at home in New Delhi, India

Airtel Broadband Service

• Airtel broadband connection

provided by Bharti Airtel

Airtel Broadband Service

provided by Bharti Airtel

• Based on Digital Subscriber Loop

(DSL) technology

• Got the required telephone and

modem to go along with it Th thl l t k

• The monthly plan we took

provides an internet speed of 512 kbps

• The plan also has restrictions on

usage, after 8GB of data download the speed goes down p g to 256 kbps

Airtel speed test

Arrangements that had to be made in Delhi

Other internet connection – mobile broadband service provided by the Tata Teleservices Limited

• TATA Photon Plus is a small USB data card

which when plugged into the USB port of my which when plugged into the USB port of my laptop would connect me to high speed broadband internet

• Internet speed up to 3.1 Mbps through TATA

network

• The monthly plan we took allows free usage up

to 5GB, after which the speed goes down to 256 kbps and 50 paisa is charged per extra 256 kbps and .50 paisa is charged per extra MB of data download

Why two internet connections?

• Fluctuation in the consistency of internet service through the

day, especially of Airtel

• Power outages cause interruption in file transfer and then
• Power outages cause interruption in file transfer, and then

Photon is used instead of Airtel

• Tata photon enables quicker file transfer when needed

O i hil I d th i th t f d t

• Once in a while I do go over the cap in the amount of data

download, and having two internet connections helps in balancing it out to some extent balancing it out to some extent

Data Processing

• Processing monthly composites
• Processing rolling annual stable lights products
• Processing fixed gain products
• Socio‐economic analyses using the nighttime light images

Data Processing – processing monthly composites

W k fl

NGDC’s portal

Personal computer NOAA’s computer (Process 1 )

Work flow

Access the OLS nighttime sub‐ p (Process 1 )

• Corresponding flag data orbits
• Data are flagged on a pixel by pixel basis.
• Data included only if flag data bits are set

as ‐ g

• rbits for each month

(approx. 400 files) as Daytime: off Nighttime marginal: off Zero lunar illuminance: on Clouds present: off Cluster jobs submitted to create masked grids of only ‘low moon’ nightfiles

• uds p ese
• Re‐project
• Run additional flags that need to be done

in 30 arc‐second space (clouds)

• Crop the sub‐orbits to mid‐swath to
• Approx. 300 corresponding

‘low moon’ nightfiles each month

Secure FX

p generate the masked grids month NOAA’s computer (through Process 1 to

6‐20 mins/nightfile,

• approx. 3 GB of zipped

data

Personal computer f li i (through Process 1 to NGDC’s portal) for line‐screening

Data Processing – processing monthly composite

Flag bands

For each nighttime suborbit, a companion flag band is

VIS FLAG

This entire suborbit was flagged as

Flag bands

p g generated with bit‐codes designating:

• daytime (solar elevation > ‐6)

was flagged as having zero lunar illuminance. daytime (solar elevation > 6)

• nighttime marginal

(‐15 < solar elevation < ‐6)

• zero lunar illuminance

Red: daytime Green: nighttime

• zero lunar illuminance

(< 0.0005 lux) marginal Black: This area is considered high

Solar elevation angles are computed

g quality nighttime data and will be processed further.

Solar elevation angles are computed based on lat, lon, and time of each OLS pixel. Lunar illuminance is a function of

processed further.

Lunar illuminance is a function of lunar phase, azimuth, and elevation, which are also based

• n lat, lon, and time of each OLS

Source: Baugh, et al. (2010). Development

• f a 2009 stable lights data using

DMSP‐OLS data. Proceedings of the

Nighttime portion of orbit F16200901281215

• n lat, lon, and time of each OLS

pixel.

Asia Pacific Advanced Network. Hanoi, Vietnam.

Data Processing – processing monthly composite

Line screening

Red: daytime G i h i

VIS FLAG

• Suborbits containing high

Line‐screening

Green: nighttime marginal Yellow: discarded by g g quality nighttime data are screened by an analyst for aurora and abrupt gain linescreening process Blue: edge‐of‐scan changes.

• Analyst chooses a start and

g data Black: This area is considered high end line of data to include for compositing. D t t d f th considered high quality nighttime data and will be processed further

• Data at edges of swath are

discarded due to increased noise and poorer geolocation (scan angle > 40 91) processed further. (scan angle > 40.91) .

Source: Baugh, et al. (2010). Development

• f a 2009 stable lights data using

DMSP‐OLS data. Proceedings of the

Nighttime portion of orbit F16200901281215

Asia Pacific Advanced Network. Hanoi, Vietnam.

Data Processing – processing monthly composite

Re projecting

OLS vis, tir and corresponding flag bands are gridded to 30 arc‐second grids, constrained to latitudes 65S‐75N.

Re‐projecting

For clarity, only mid‐swath, line‐screened data are shown.

Green: nighttime Green: nighttime marginal VIS TIR FLAG

Nighttime portion of orbit F16200901281215

Source: Baugh, et al. (2010). Development of a 2009 stable lights data using DMSP‐OLS data. Proceedings of the Asia Pacific Advanced Network. Hanoi, Vietnam.

Data Processing – processing monthly composite

Cloud mask

• A cloud mask is generated by comparing the re‐projected OLS thermal band to a

surface temperature grid provided by National Center for Environmental Prediction (NCEP) ( )

• Difference images are made as Diff = Surface Temp ‐ TIR.
• Due to the increased variability in land temperature values, land and ocean regions

d t l are processed separately.

• Thresholds are computed from the difference images in latitudinal tiles as mean +

N*stdev. Values greater than this threshold are flagged as clouds g gg

TIR NCEP Surface Temp. Diff (Land) Diff (Sea) Cloud Mask

Personal

Data Processing – processing monthly composite

Work flow

Personal computer NOAA’s computer (through NGDC’s linescreen text file

Secure FX

Work flow

Header files of the masked id d t d portal to Process 1 ) linescreen text file grids are updated ‘Linescreened’ and ‘non‐ l d’ h b linescreened’ nighttime sub‐

• rbits submitted for

averaging Tiles of suite of products of which the most important ones are –

• avg_vis – Average visible band data

values

Secure FX : 1 ‐ 1.5 hrs per file, approx 180 MB

Converted to geotiffs

• cf_cvg – Number of cloud free
• bservations used
• Histograms of input visible band

data for each grid cell Average visible band data converted to NOAA’s computer (through Process 1 to

• approx. 180 MB
• f zipped data

Made available to customers g bytes NGDC’s portal) Personal computer for viewing

Example of monthly composites

Monthly line screened composite of the world F18 20101101 20101130 Monthly line‐screened composite of the world – F18_20101101_20101130 The most recent monthly line‐screened composite of the world – F18_20110701_20110731

Data Processing – rolling annual stable lights

Example Example

• The average visible band

includes lights from fires, fishing boats, and other light sources

• To create stable lights

products the transient light p g sources are removed

The most recent rolling stable lights composite of India – F18_20100701_20110630

Data Processing – rolling annual stable lights

NOAA’s computer Merging of monthly

Work flow

NOAA’s computer (through NGDC’s portal to Process 1) Merging of monthly composites, say from July of 2010 to June of 2011

Work flow

Outlier removal: composite histograms analyzed for bright

• utliers, which are removed

To get samples of background values k d Background removal: separate areas in the outlier removed average that contain lights from those background areas where no lights are present. markers drawn over light‐free areas Personal Is it ok?

Secure FX,

computer for viewing Areas with values greater than the maximum light‐free values are tallied as “greater than background”. Stable lights mask is created from areas considered gtb 40% of the time NOAA’s computer (through Process 1 to NGDC’s portal) Shift the avg_vis, cf_cvg , and mask to the LandScan population grid

Secure FX 100 MB 105 MB of zipped data, 1.5 hrs

from areas considered gtb 40% of the time NOAA’s computer (through Process 1 to NGDC’s Personal computer Stable lights mask applied on raw avg_vis image to create the final stable lights product

Secure FX 100 MB data, 1.5 hrs NO

(through Process 1 to NGDC’s portal) Markers re‐drawn Creating geotiffs of raw avg_vis, cf_cvg, stable lights Made available to customers

NO

Data Processing – rolling annual stable lights

Outlier removal process example Outlier removal process example

Orissa coast, India – ‘raw’ average vis Orissa coast, India – outliers removed from raw average vis

Data Processing – rolling annual stable lights

Background removal process example

Light‐free areas chosen by an analyst in red Resulting Stable Lights mask Stable lights

Data Processing – fixed gain composites

• DMSP sensor is typically operated at high gain setting for the detection of

moonlit clouds S d i b i h f b b f i bi i i

• Saturated in bright cores of urban centers because of six bit quantization

and limited dynamic range

• Every month a limited set of observations are obtained at low lunar

Every month a limited set of observations are obtained at low lunar illuminations where the detector is set significantly lower than its typical

• perational settings (sometimes by a factor of 100)

h d b i ll d b lli 16 fi d i

• These data are at present being collected by satellite F16, at fixed gains
• f 15, 35, and 50 decibels
• The fixed gain nightfiles have to be line‐screened separately for aurora

The fixed gain nightfiles have to be line screened separately for aurora and fixed gain

• File transferring, line‐screening, and processing are same as the nightfiles

b f h l d ( f h f l d

• rbits of the operational data (Approx. 3 GB of nightfiles, and 220 MB

combines averages are transferred)

• Merging the stable lights and fixed gain product creates the unsaturated

Merging the stable lights and fixed gain product creates the unsaturated, superior radiance‐calibrated images

Stable lights vs. Radiance‐calibrated image of Delhi ‐ 2004

Challenges

• Time difference between Boulder, Colorado (Mountain time), and

New Delhi, India (Indian Standard Time, IST) – 11.5 hours during h i d f D li h S i Ti (f M h N b ) the period of Daylight Savings Time (from March to November), and at a difference of 12.5 hours when the daylight saving time ends (December to February) ends (December to February)

• Inter‐transference of files between NOAA’s computer and my

l i i i i I l h f laptop is sometimes very time‐consuming. I always have to transfer a file from NOAA’s system to mine before I can view it

• Miss the personal interaction with colleagues, the round‐table

conferences, don’t always come to know what others in the group ki are working on

Thank You!! Questions? Thank You!! Questions?

EOG website: http://www ngdc noaa gov/dmsp/dmsp html http://www.ngdc.noaa.gov/dmsp/dmsp.html Emails: h l d h l d Chris Elvidge: chris.elvidge@noaa.gov Kimberly Baugh: kim.baugh@noaa.gov be y aug baug @ oaa go Sharolyn Anderson: h l d @ sharolyn.anderson@noaa.gov Tilottama Ghosh: tilottama.ghosh@noaa.gov g g