INFORMATION TECHNOLOGY TRACK Russell Brown Chapter 13 Standing - - PowerPoint PPT Presentation

information technology
SMART_READER_LITE
LIVE PREVIEW

INFORMATION TECHNOLOGY TRACK Russell Brown Chapter 13 Standing - - PowerPoint PPT Presentation

NACTT STAFF SYMPOSIUM SERIES INFORMATION TECHNOLOGY TRACK Russell Brown Chapter 13 Standing Trustee - Phoenix, AZ Allan Reininger System Manager - San Antonio, TX Chapter 13 Standing Trustee - Mary Viegelahn Carl Brooks System Manager -


slide-1
SLIDE 1

STAFF SYMPOSIUM - IT TRACK

NACTT STAFF SYMPOSIUM SERIES INFORMATION TECHNOLOGY TRACK

Russell Brown Chapter 13 Standing Trustee - Phoenix, AZ Allan Reininger System Manager - San Antonio, TX Chapter 13 Standing Trustee - Mary Viegelahn Carl Brooks System Manager - Detroit, MI Chapter 13 Standing Trustee - Tammy Terry Tom O’Hern Program Manager, ICF International, Baltimore, MD STACS - Standing Trustee Alliance for Computer Security

4/14/2015

SESSION 5 - DISASTER RECOVERY 1

slide-2
SLIDE 2

STAFF SYMPOSIUM IT TRACK

IT Track Outline

Day 1

Session 1 (9:00 - 10:30) - HOW TO KEEP YOUR TRUSTEE HAPPY FROM AN IT PERSPECTIVE

Session 2 (10:45 - 12:15) - A RIVERWALK THROUGH YOUR NETWORK Lunch (12:15-1:30)

Session 3 (1:30 – 3:00) - DESKTOP & SERVER MANAGEMENT

Session 4 (3:15 – 4:45) - THE CLOUD – WHO REALLY UNDERSTANDS IT? DAY 2

Session 5 (8:30 - 10:00) - DISASTER RECOVERY – YOUR WORST FEAR COMES TRUE

Session 6 (10:15 - 11:45) - BUERRITO BOWL - MIXED HOT TOPICS List of Reference Material

4/14/2015

SESSION 5 - DISASTER RECOVERY 2

slide-3
SLIDE 3

STAFF SYMPOSIUM - IT TRACK

Disaster Recovery by Example Backups and the Cloud Server Virtualization

Sess ssion Focal Po Points ts

4/14/2015

SESSION 5 - DISASTER RECOVERY 3

slide-4
SLIDE 4

STAFF SYMPOSIUM IT TRACK

CryptoLocker: Office A

 Staff member w/o administrator rights opens an infected email

attachment containing what appears to be a valid Invoice

 Other staff members report errors accessing encrypted files to the system

manager

 System manager see antivirus alerts on server and calls STACS  STACS helps identify infected computer which is removed from network

(time <60 minutes)

 Source of infection identified, no additional infections confirmed  Office reinstalls infected computer and restores prior night’s backup of

encrypted files (< 48 hours)

4/14/2015

SESSION 5 - DISASTER RECOVERY 4

slide-5
SLIDE 5

STAFF SYMPOSIUM IT TRACK

CryptoLocker: Office B

 Staff Attorney with administrator rights opens an infected email

attachment containing what appears to be a valid Invoice

 Other staff members report errors accessing encrypted files to the system

manager

 System manager attempts to clean infection and restore encrypted files  Infected computer not identified and continues to encrypt shared files

from server

 Undiscovered secondary infection of SPAM bot malware occurs  Trustee has system manager contact STACS (1 week after infection

discovered)

4/14/2015

SESSION 5 - DISASTER RECOVERY 5

slide-6
SLIDE 6

STAFF SYMPOSIUM IT TRACK

CryptoLocker: Office B (continued)

 STACS goes onsite (<12 hours), helps identify infected

computer and removes from network

 STACS helps correct missing/misconfigured antivirus, system

patches, firewall, plan a backup/restore strategy

 Office required to reinstall infected desktop and all servers,

reinstall desktop security software on all systems (> 2 weeks)

 Data lost and restoration of recoverable data (> 3 weeks)

4/14/2015

SESSION 5 - DISASTER RECOVERY 6

slide-7
SLIDE 7

STAFF SYMPOSIUM IT TRACK

CryptoLocker: Office C

 Administrator opens file attachment from staff email and

leaves at the end of the day

 Cryptolocker encrypts 10,000 files on shares from two servers

  • ver 16 hours

 Trustee and staff discovery issues opening PDFs  Trustee has admin call STACS  < 15 minutes infected PC identified and removed from

network.

 Offsite backup service check to recovery files message reads

“Server not in contact for 533 days.”

 Administrator dismissed for gross negligence

4/14/2015

SESSION 5 - DISASTER RECOVERY 7

slide-8
SLIDE 8

STAFF SYMPOSIUM IT TRACK

Contributing and Distinguishing Factors

 User accounts configured with/without administrator rights  Systems with properly/poorly maintained antivirus & patches  Users quick/slow to notify system manager of unusual activity  System/office manager quick/slow to contact STACS for help  Backup and restoration process untested/tested & verified  Windows XP and 2003 Server still in use

4/14/2015

SESSION 5 - DISASTER RECOVERY 8

slide-9
SLIDE 9

STAFF SYMPOSIUM IT TRACK

Disaster and Recovery Scenarios

 Arbitrary deletion of file/folder discovered months later  SAN or NAS Failure / Server Failure

  • Multi disk failure (RAID with Dual Parity)
  • Controller failure leading to …

 Corruption of Virtual machine(s) over 3-5 days

  • (24-48 hours)

 System Infection

  • Requires a reformat of hard drives
  • Requires a system rebuild or full system restore
  • Requires full data restore – Data files and Databases

4/14/2015

SESSION 5 - DISASTER RECOVERY 9

slide-10
SLIDE 10

STAFF SYMPOSIUM IT TRACK

What to backup? The Obvious

 Bankruptcy data

  • Case software database
  • External case data files (PDFs)
  • Case Software and configuration files

 Expense Account data

  • Software and license key
  • Database encrypted export

4/14/2015

SESSION 5 - DISASTER RECOVERY 10

slide-11
SLIDE 11

STAFF SYMPOSIUM IT TRACK

What to backup? The Not So Obvious

 Electronic Disaster Recovery Kit

  • Licenses

 Microsoft Server and Seat licenses  Others purchased software license Keys

  • Ex: Acrobat, antivirus,

 Other Device Activation Keys

  • Ex: Check printer, firewall

 Backup solution license, serial numbers or keys

  • Hard to get copies of software
  • Encrypted Master Password list

4/14/2015

SESSION 5 - DISASTER RECOVERY 11

slide-12
SLIDE 12

STAFF SYMPOSIUM IT TRACK

What to backup? System & Software Configuration

  • Full System state (for bare metal restore)
  • Server Software configurations

 Web servers, ftp, email, exchange,  Exchange, IIS, SharePoint  Virtual Machines

  • Device Configurations

 Firewall configuration backup and firmware release  Wireless/Router configuration backup  Managed switch configuration – Especially with VLANs  NAS/SAN – especially RAID and logical volume setup  Hypervisor configuration

4/14/2015

SESSION 5 - DISASTER RECOVERY 12

slide-13
SLIDE 13

STAFF SYMPOSIUM IT TRACK

What to backup? User Data

 Email

  • Hosted Mailbox – outsourced to 3rd party
  • Hosted Temporarily – outsourced POPs/IMAPs
  • Exchange – centralized mailbox folders
  • Outlook – decentralized on user computer systems

 Desktop Files

  • User Profiles
  • Documents and Settings

4/14/2015

SESSION 5 - DISASTER RECOVERY 13

slide-14
SLIDE 14

STAFF SYMPOSIUM IT TRACK

What to backup? Organizational of data

 File Shares

  • Evaluate shares from all servers
  • Backup agents need access to restricted folders

 Accounting  Human Resources  System Admin (DR Kit)  Tax Returns

  • AccessEnum

 From Microsoft’s Sysinternals tool suite  Scans directories for file permissions

4/14/2015

SESSION 5 - DISASTER RECOVERY 14

slide-15
SLIDE 15

STAFF SYMPOSIUM IT TRACK

Q&A

 "Trustees, ask your system managers to answer these questions and show proof

  • f the answers. Ex: backup status report, list of files and folders backed up.

Review the list with you system manager to assure all critical data is backed up and recoverable if you lost everything last night. Ask what is not backed up, and review to make sure you can live without it. Otherwise get it on the backup list."

 Any suggestions on how to prepare a backup list? How to prepare a NOT backed

up list? I've been looking for some type of software that will prepare/save directory trees so that I can do this, but as yet have not found anything

  • acceptable. Any suggestions here? Sure I can come up with some general details,
  • servers, drives, folders, etc, but it sounds like you want more than that - as do I.

How about some specific suggestions on how to prepare these backup lists. Ideally, I would like a detail listing of everything and be able to flag what is or is not backed up, but I've been looking for the last few years and still haven't found a practical way to do this.

4/14/2015

SESSION 5 - DISASTER RECOVERY 15

slide-16
SLIDE 16

STAFF SYMPOSIUM IT TRACK

Types of Disaster Recovery Plans

 No disaster plan at all  No disaster plan, but good backup procedures  A disaster plan, with no resources in place  A ‘cold site’ disaster recovery solution  A ‘split site’ or ‘warm site’ disaster recovery solution  A ‘hot site’ disaster recovery solution

4/14/2015

SESSION 5 - DISASTER RECOVERY 17

slide-17
SLIDE 17

STAFF SYMPOSIUM IT TRACK

No disaster plan at all

 No disaster plan; good backup procedures  The absolute minimum companies must do – even

the smallest business – to prevent a disaster from wiping out business information is to back up the data on your computers daily and store the backups

  • ffsite at a secure archival company. Never store it at

employee’s homes.

 That way even if your hardware and software is

ruined, you can still replace it and load it up with all your irreplaceable data.

4/14/2015

SESSION 5 - DISASTER RECOVERY 18

slide-18
SLIDE 18

STAFF SYMPOSIUM IT TRACK

A disaster plan, with no resources in place

 Once you have a good backup and archival

procedure and your critical systems are fault tolerant, the next step is to put together procedures for remote disaster recovery.

 This simply means you ask and answer the question,

“What do we do if the computer center is utterly destroyed?”

 Put your plan is written form and store a copy offsite

4/14/2015

SESSION 5 - DISASTER RECOVERY 19

slide-19
SLIDE 19

STAFF SYMPOSIUM IT TRACK

A ‘cold site’ disaster recovery solution

 A simple yet effective business backup solution, a

cold site is simply a reserved area on a data center where your business can set up new equipment in the event of a disaster. This is a popular disaster recovery method because it tends to be less expensive than other options, yet still gives a company the ability to survive a true disaster.

4/14/2015

SESSION 5 - DISASTER RECOVERY 20

slide-20
SLIDE 20

STAFF SYMPOSIUM IT TRACK

A ‘split site’ disaster recovery solution

 If your organization is large enough, it may be feasible to house the IT

department across more than one location. In the event of a disaster to

  • ne site, operations can then reasonably simply shift to the other site and

any new equipment needed could be purchased as necessary as long as the backups were properly maintained. The advantage to this method is it eliminates the need for the major up-front costs of building a dedicated disaster center.

 Also know as a ‘warm’ site  As your organization will need to purchase or lease the equipment in the

warm site, this option does involve more set-up costs than a cold site, but has the advantage of being able to get your business systems up and running much faster. Even sites with multiple applications can generally be back to full operation within 24 hours.

4/14/2015

SESSION 5 - DISASTER RECOVERY 21

slide-21
SLIDE 21

STAFF SYMPOSIUM IT TRACK

A ‘hot site’ disaster recovery solution

 A hot site is a premium level of disaster recovery

where the business IT systems and up-to-date data are duplicated and maintained at a separate data center.

 In this scenario, a duplicate computer center is set

up in a remote location with communication lines set up and actively copying data at all times. The site has a duplicate of every critical server, with data that is up-to-date to within hours, minutes or even seconds.

4/14/2015

SESSION 5 - DISASTER RECOVERY 22

slide-22
SLIDE 22

STAFF SYMPOSIUM IT TRACK

Backup Plan

 Backup Schedule  Documented Recovery Process  Retention Periods  Recovery Time  Testing and Evidence

  • Real Evidence vs. Misleading Evidence

4/14/2015

SESSION 5 - DISASTER RECOVERY 23

slide-23
SLIDE 23

STAFF SYMPOSIUM IT TRACK

STACS Memo 8/24/2014

 If today at 5 PM I were to delete ALL the files on the network file shares,

could you

  • restore them from your backup?
  • How much data would be lost?
  • Is this data loss acceptable?

 When was the last time you checked to make sure your backups were

actually working?

 Trustees, ask your system managers to answer these questions and show

proof of the answers.

  • Ex: backup status report, list of files and folders backed up.
  • Review the list with you system manager to assure all critical data is backed up

and recoverable if you lost everything last night.

  • Ask what is not backed up, and review to make sure you can live without it.

Otherwise get it on the backup list.

4/14/2015

SESSION 5 - DISASTER RECOVERY 24

slide-24
SLIDE 24

STAFF SYMPOSIUM IT TRACK

Materials

 NACTT Disaster Recovery Template  Backup for Dummies book

4/14/2015

SESSION 5 - DISASTER RECOVERY 25

slide-25
SLIDE 25

STAFF SYMPOSIUM - IT TRACK

Server Virtualization

4/14/2015

SESSION 5 - DISASTER RECOVERY 26

slide-26
SLIDE 26

STAFF SYMPOSIUM IT TRACK

Users: Wouldn’t it be nice to…

 Have a backup copy of my computer when it

crashes?

 Recover my PC in minutes after it crashes?  Copy this freshly installed PC to another and another

and another…?

 Have access to my old PC so I can run the old

program that doesn’t run on my new PC?

 Try the new version of Windows without having to

wipe out my current setup?

4/14/2015

SESSION 5 - DISASTER RECOVERY 27

slide-27
SLIDE 27

STAFF SYMPOSIUM IT TRACK

Admins: Wouldn’t it be nice to…

Do all those User things plus…

 Have a backup server to test changes w/o messing

up the production system?

 Get rid of the half dozen servers and associated

cabling snake pit?

 Have a Linux/MacOS/Windows system to mess

around with?

 Run multiple Operating Systems at the same time?

4/14/2015

SESSION 5 - DISASTER RECOVERY 28

slide-28
SLIDE 28

STAFF SYMPOSIUM IT TRACK

4/14/2015

Virtualization

 Turning Hardware into Software

  • Running multiple computers on one piece of hardware

SESSION 5 - DISASTER RECOVERY 29

slide-29
SLIDE 29

STAFF SYMPOSIUM IT TRACK

Trustee A Production Virtualization Server

2 3 S v r 2 8 S v r L i n u x W i n d

  • w

s X P

Traditional Dedicated Computer Hardware (Copy image to a Virtual Machine)

Virtual Network Virtualization Software Windows 2003 File Server Windows 2008 SQL Server CPUs Memory Disk Storage N e t w

  • r

k C a r d Physical Hardware V i r t u a l H a r d w a r e V i r t u a l M a c h i n e s

Trustee B Production Virtualization Server (Trustee A Disaster Recovery Site)

W i n d

  • w

s 2 8 S Q L S e r v e r

2003 Svr

4/14/2015

SESSION 5 - DISASTER RECOVERY 31

slide-30
SLIDE 30

STAFF SYMPOSIUM IT TRACK

4/14/2015

Use Cases

 System Administration

  • Legacy application or system use
  • Native OS application use
  • Helpdesk/User Support
  • Test updates, patches and changes

 Application Development

  • Cross platform application testing

 Continuity of Operations

  • System backup and recovery
  • Portable Systems

SESSION 5 - DISASTER RECOVERY 32

slide-31
SLIDE 31

STAFF SYMPOSIUM IT TRACK

4/14/2015

Benefits

 Consolidates Hardware  Eases system administration  Reduce hardware costs  Simplifies Backup and Recovery  Continuity of Operations  Cloud Computing

  • Outsourcing system administration

SESSION 5 - DISASTER RECOVERY 33

slide-32
SLIDE 32

STAFF SYMPOSIUM IT TRACK

4/14/2015

Virtualization Solutions

 Desktop Software Solutions

  • Usually cheap or free
  • Some can create their own VMs

 Virtualization Server Solutions

  • Can stand alone or be client to a server
  • Gives a top-down view of multiple running VMs
  • Administer VMs distributed across the network

SESSION 5 - DISASTER RECOVERY 34

slide-33
SLIDE 33

STAFF SYMPOSIUM IT TRACK

Desktop vs. Server

Virtual Network V i r t u a l i z a t i

  • n

S

  • f

t w a r e

Windows 2003 File Server Windows 2008 SQL Server Linux Web Server Windows XP Test Desktop C P U s Memory D i s k S t

  • r

a g e N e t w

  • r

k C a r d

P h y s i c a l H a r d w a r e V i r t u a l H a r d w a r e

Virtual Machines

V i r t u a l N e t w

  • r

k

Virtualization Software W i n d

  • w

s 2 3 F i l e S e r v e r W i n d

  • w

s 2 8 S Q L S e r v e r L i n u x W e b S e r v e r W i n d

  • w

s X P T e s t D e s k t

  • p

CPUs Memory Disk Storage Network Card

Physical Hardware Virtual Hardware

G u e s t O S

  • n

V i r t u a l M a c h i n e s Host Operating System

4/14/2015

SESSION 5 - DISASTER RECOVERY 35

slide-34
SLIDE 34

STAFF SYMPOSIUM IT TRACK

4/14/2015

Desktop Software Solutions

 Windows

  • Microsoft HyperV

 Mac

  • Apple Parallels
  • VMWare Fusion

 Multi-Platform

  • VMware Desktop
  • Sun Virtual Box

SESSION 5 - DISASTER RECOVERY 36

slide-35
SLIDE 35

STAFF SYMPOSIUM IT TRACK

4/14/2015

Server Products

 VMware ESX Server  Microsoft Hyper Visor  High-end Server Vendors use proprietary solutions

(IBM)

SESSION 5 - DISASTER RECOVERY 37

slide-36
SLIDE 36

STAFF SYMPOSIUM IT TRACK

4/14/2015

Cost Items

 Virtualization Software

  • Free options are usually only community supported

 Hardware

  • More robust machines are required
  • Lots of RAM
  • Lots of Hard Disk space
  • Multiple Network Interface Cards (NIC)

 Operating System Software Licenses

  • Guest operating system must still be licensed and paid for

SESSION 5 - DISASTER RECOVERY 38

slide-37
SLIDE 37

STAFF SYMPOSIUM IT TRACK

Sample Small Office Solution

 6-8 Servers

  • AD/FS, MSSQL, IIS App server, MS Terminal Server, Redhat Dev

 Virtualization Software

  • VMWare vSphere 4 (this is the new version of ESX)
  • $ 1,800 (cost varies by number of CPUs, so a larger server would cost more)

 Hardware

  • Dell R710 server w/ 32GB of RAM and 1TB of RAID6 storage
  • $ 8,000 (this server is benchmarked for over 60VMs although 10 would be

more realistic in practice)

  • $ 2,100 (cost from Dell may be more or less depending on vendor)

 OS Software

  • Windows Server Enterprise (Open License allows up to 4 VMs)

 Total < $12,000.00

4/14/2015

SESSION 5 - DISASTER RECOVERY 39

slide-38
SLIDE 38

STAFF SYMPOSIUM IT TRACK

Gotchas

 Application compatibility w/ Windows OS

  • surprisingly there are apps that are not supported on both

Server Enterprise and Server Standard

 Application compatibility with VM

  • Many applications that require direct access to hardware

are not well suited to VM. An example would be a backup app that writes to a tape device.

 Hardware compatibility with VM

4/14/2015

SESSION 5 - DISASTER RECOVERY 40

slide-39
SLIDE 39

STAFF SYMPOSIUM IT TRACK

Networking

 Choice affects security and performance  Bridged

  • Guest VM gets its own IP address on physical network

 NAT’ed

  • Guest VM shares Host IP address on physical network

 Private Virtual Network

  • Guest VMs share a virtual network among each other

 Host Only

  • Guest VM and Host share a private virtual network

4/14/2015

SESSION 5 - DISASTER RECOVERY 41