[PPT] - CS615 - Aspects of System Administration Filesystems, Disks, Storage PowerPoint Presentation

SLIDE 1

CS615 - Aspects of System Administration Slide 1

CS615 - Aspects of System Administration Filesystems, Disks, Storage

Department of Computer Science Stevens Institute of Technology Jan Schaumann jschauma@stevens.edu http://www.cs.stevens.edu/~jschauma/615A/

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 2

CS615 - Aspects of System Administration Slide 2

Let’s review HW1

Find the right AMI: $ aws ec2 describe-images | grep -i omnios [...]

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 3

CS615 - Aspects of System Administration Slide 3

Let’s review HW1

Find the right AMI: $ aws ec2 describe-images --query Images[].ImageId --output text \

-filters "Name=name,Values=’OmniOS r151020*’"

ami-50ecc847

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 4

CS615 - Aspects of System Administration Slide 4

Let’s review HW1

Running an instance: $ aws ec2 run-instances --instance-type t1.micro \

-key-name stevens --image-id ami-50ecc847

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 5

CS615 - Aspects of System Administration Slide 5

Let’s review HW1

Save yourself some typing: $ alias instance=’aws ec2 run-instances --key-name stevens --image-id’ $ instance ami-50ecc847 --instance-type t1.micro

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 6

CS615 - Aspects of System Administration Slide 6

Let’s review HW1

Make it permanent: $ echo "alias instance=’aws ec2 run-instances --key-name stevens --image-id’" \ >> ~/.bashrc $ . ~/.bashrc $ alias alias instance=’aws ec2 run-instances --key-name stevens --image-id’ $

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 7

CS615 - Aspects of System Administration Slide 7

Let’s review HW1

Add the other common AMIs we’ll use: $ alias aws=’aws --output text’ del-volume=’aws ec2 delete-volume --volume-id’ instance=’aws ec2 run-instances --key-name stevens --image-id’ instances=’aws ec2 describe-instances’ term-instances=’aws ec2 terminate-instances --instance-ids’ start-fedora=’instance ami-0187f76b --instance-type t1.micro’ start-freebsd=’instance ami-d0b520b8 --instance-type t1.micro’ start-netbsd=’instance ami-569ed93c --instance-type t1.micro’ start-omnios=’instance ami-9fbbfaf5 --instance-type t1.micro’ start-ubuntu=’instance ami-6de0dd04 --instance-type t1.micro’ volumes=’aws ec2 describe-volumes’ $

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 8

CS615 - Aspects of System Administration Slide 8

Let’s review HW1

ssh to an instance: $ ssh -i ~/.ssh/ec2 root@<mumble>.compute-1.amazonaws.com

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 9

CS615 - Aspects of System Administration Slide 9

Let’s review HW1

Let’s save ourselves some typing: $ cat >>~/.ssh/config <<EOF > Host *.amazonaws.com > IdentityFile ~/.ssh/ec2 > User root > EOF $ ssh <mumble>.compute-1.amazonaws.com

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 10

CS615 - Aspects of System Administration Slide 10

Let’s review HW1

How do we know what host we’re connecting to? $ ssh ec2-54-227-16-184.compute-1.amazonaws.com The authenticity of host ’ec2-54-227-16-184.compute-1.amazonaws.com (54.227.16.184)’ can’t be established. ECDSA key fingerprint is 19:af:35:01:0b:2a:ee:3d:30:0f:69:11:cc:55:7c:20. Are you sure you want to continue connecting (yes/no)?

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 11

CS615 - Aspects of System Administration Slide 11

Let’s review HW1

$ instance ami-0187f76b # Fedora $ aws ec2 get-console-output --instance-id i-18fd6d98 [...] <14>Feb 6 18:37:59 ec2: ############################################################ <14>Feb 6 18:37:59 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS----- <14>Feb 6 18:37:59 ec2: 256 SHA256:3kQfQTV+sfk/BDa9jGdvvea8i28TwWOgWt8lZKdALgA /etc/ <14>Feb 6 18:37:59 ec2: 256 SHA256:R3/H4kFMu/JZZCxQL7ZYODZoKwGFNBN6Cpfbsx/PFDQ /etc/ <14>Feb 6 18:37:59 ec2: 2048 SHA256:WjIFQE4ZEQfSR0bfCVgRbR7Q0780RFug8CZNXcEam0I /etc <14>Feb 6 18:37:59 ec2: -----END SSH HOST KEY FINGERPRINTS-----

----BEGIN SSH HOST KEY KEYS-----

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP4RT8CDVexvK ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBTpcfKZcASSjicvMPAx6UAQSac9ZH0UeO+/A9m6EuFO ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC48feKNKyaS+c+zJmu6/L6o9L7VkW6YMuwu0//Uo5uS46Yh

----END SSH HOST KEY KEYS-----

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 12

CS615 - Aspects of System Administration Slide 12

Let’s review HW1

How do we know what host we’re connecting to? $ start-fedora $ ssh-keygen -l -f /dev/stdin <<<$(console i-0990f1eb069c853c4 | grep ^ecdsa) $ aws ec2 get-console-output --instance-id i-18fd6d98 | grep ^ecdsa > hostkey 256 19:af:35:01:0b:2a:ee:3d:30:0f:69:11:cc:55:7c:20 (ECDSA) $ ssh fedora@ec2-54-227-16-184.compute-1.amazonaws.com The authenticity of host ’ec2-54-227-16-184.compute-1.amazonaws.com (54.227.16.184)’ can’t be established. ECDSA key fingerprint is 19:af:35:01:0b:2a:ee:3d:30:0f:69:11:cc:55:7c:20. Are you sure you want to continue connecting (yes/no)? yes [fedora@ip-10-170-2-33 ~]$

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 13

CS615 - Aspects of System Administration Slide 13

Let’s review HW1

# uname -a SunOS ip-10-152-178-106.ec2.internal 5.11 omnios-r151020-b5b8c75 i86pc i386 i86xpv

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 14

CS615 - Aspects of System Administration Slide 14

Let’s review HW1

# ifconfig -a lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 xnf0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 2 inet 10.235.81.115 netmask ffffffc0 broadcast 10.235.81.127 ether 22:0:a:eb:51:73 lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1 inet6 ::1/128 xnf0: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 1500 index 2 inet6 ::/0 ether 22:0:a:eb:51:73

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 15

CS615 - Aspects of System Administration Slide 15

Let’s review HW1

# netstat -na | more [...] TCP: IPv4 Local Address Remote Address Swind Send-Q Rwind Recv-Q State

------------------- -------------------- ----- ------ ----- ------ -----------

127.0.0.1.4999 . 0 128000 0 LISTEN .111 .* 0 128000 0 LISTEN . . 0 128000 0 IDLE .111 .* 0 128000 0 LISTEN . . 0 128000 0 IDLE .46457 .* 0 128000 0 LISTEN .55986 .* 0 128000 0 LISTEN .22 .* 0 128000 0 LISTEN 10.110.94.225.22 155.246.89.107.46137 42304 47 128592 0 ESTABLISHED [...]

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 16

CS615 - Aspects of System Administration Slide 16

Let’s review HW1

# man df [...] # df [...] # df -haT [...] # df -i [...] # mount [...]

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 17

CS615 - Aspects of System Administration Slide 17

Let’s review HW1

# format [...] format> verify [...] # zpool list [...] # zfs list [...]

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 18

CS615 - Aspects of System Administration Slide 18

Filesystems, Disks, Storage

$ ssh linux-lab.cs.stevens.edu $ dh -hT How much disk space is available under /tmp?

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 19

CS615 - Aspects of System Administration Slide 19

Filesystems, Disks, Storage

$ ssh linux-lab.cs.stevens.edu $ dh -hT $ cd /tmp $ df -h . How much disk space is available under /tmp? Let’s fill it up.

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 20

CS615 - Aspects of System Administration Slide 20

Filesystems, Disks, Storage

$ ssh linux-lab.cs.stevens.edu $ dh -hT $ cd /tmp $ df -h . How much disk space is available under /tmp? Let’s fill it up. $ dd if=/dev/zero of=/tmp/big bs=1G [...] Now try to ssh to that host...

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 21

CS615 - Aspects of System Administration Slide 21

Filesystems, Disks, Storage

File sizes are not always what they seem to be. $ mkdir /tmp/${USER} $ export LARGE=/tmp/${USER}/large $ truncate -s $(df /tmp | awk ’/^\// { print $4; }’)0000 ${LARGE} $ ls -l ${LARGE} $ du ${LARGE} $ stat ${LARGE} $ cp ${LARGE} ${LARGE}2 $ du ${LARGE}2 $ cat ${LARGE} > ${LARGE}2 $ ls -l ${LARGE}* $ du ${LARGE}*

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 22

CS615 - Aspects of System Administration Slide 22

Filesystems, Disks, Storage

How many files can be created on /tmp? $ ssh linux-lab.cs.stevens.edu $ rm /tmp/${USER}/large2 $ dh -i /tmp

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 23

CS615 - Aspects of System Administration Slide 23

Filesystems, Disks, Storage

How many files can be created on /tmp? $ ssh linux-lab.cs.stevens.edu $ dh -i /tmp Let’s fill ’em up!

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 24

CS615 - Aspects of System Administration Slide 24

Filesystems, Disks, Storage

$ ssh linux-lab.cs.stevens.edu $ cd /tmp $ dh -i /tmp $ touch newfile $ cc -Wall ~jschauma/tmp/mkfiles.c $ ./a.out $ ls -ld /tmp/$(id -u) $ ls /tmp/$(id -u) | wc -l $ touch newerfile $ echo "hello hello hello" >> newfile $ rm -fr /tmp/$(id -u)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 25

CS615 - Aspects of System Administration Slide 25

Filesystems, Disks, Storage

Important lessons: File sizes are not always what they seem to be. Error messages aren’t always what they seem to be! All resources are finite.

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 26

CS615 - Aspects of System Administration Slide 26

Filesystems, Disks, Storage

basic disk concepts basic filesystem concepts file systems

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 27

CS615 - Aspects of System Administration Slide 27

Topics covered

basic disk concepts storage models disk interfaces physical disk structure partitions basic filesystem concepts file systems

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 28

CS615 - Aspects of System Administration Slide 28

Topics covered

basic disk concepts storage models disk interfaces physical disk structure partitions basic filesystem concepts RAID logical volume managment device formatting file systems

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 29

CS615 - Aspects of System Administration Slide 29

Topics covered

basic disk concepts storage models disk interfaces physical disk structure partitions basic filesystem concepts RAID logical volume managment device formatting file systems the UNIX filesystem or Berkeley Fast File System (FFS)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 30

CS615 - Aspects of System Administration Slide 30

Basic Disk Concepts Storage Models

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 31

CS615 - Aspects of System Administration Slide 31

Basic Disk Concepts: Storage Models

Direct Attached Storage (DAS) ssh lab ’df -hT /’

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 32

CS615 - Aspects of System Administration Slide 32

Basic Disk Concepts: Storage Models

Network Attached Storage (NAS) ssh lab ’df -hT /home/$(whoami)’

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 33

CS615 - Aspects of System Administration Slide 33

Basic Disk Concepts: Storage Models

Storage Area Networks (SAN)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 34

CS615 - Aspects of System Administration Slide 34

Basic Disk Concepts: Storage Models

Cloud Storage (Examples: EBS, S3)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 35

CS615 - Aspects of System Administration Slide 35

Basic Disk Concepts Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 36

CS615 - Aspects of System Administration Slide 36

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 37

CS615 - Aspects of System Administration Slide 37

Basic Disk Concepts: Disk Devices

Security affects everything. http://t.co/eM6XpATITQ

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 38

CS615 - Aspects of System Administration Slide 38

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 39

CS615 - Aspects of System Administration Slide 39

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 40

CS615 - Aspects of System Administration Slide 40

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 41

CS615 - Aspects of System Administration Slide 41

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 42

CS615 - Aspects of System Administration Slide 42

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 43

CS615 - Aspects of System Administration Slide 43

Basic Disk Concepts Disk Interfaces

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 44

CS615 - Aspects of System Administration Slide 44

Basic Disk Concepts: Disk Interfaces: SCSI

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 45

CS615 - Aspects of System Administration Slide 45

Basic Disk Concepts: Disk Interfaces: ATA

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 46

CS615 - Aspects of System Administration Slide 46

Basic Disk Concepts: Disk Interfaces: ATA

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 47

CS615 - Aspects of System Administration Slide 47

Basic Disk Concepts: Disk Interfaces: Fibre Channel

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 48

CS615 - Aspects of System Administration Slide 48

Basic Disk Concepts: Disk Interfaces: Fibre Channel

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 49

CS615 - Aspects of System Administration Slide 49

Basic Disk Concepts: Disk Interfaces: Fibre Channel

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 50

CS615 - Aspects of System Administration Slide 50

Basic Disk Concepts: Disk Interfaces: Fibre Channel

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 51

CS615 - Aspects of System Administration Slide 51

Basic Disk Concepts: Disk Interfaces: Fibre Channel

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 52

CS615 - Aspects of System Administration Slide 52

Basic Disk Concepts: Disk Interfaces: SANs

ATA over Ethernet (AoE): create low-cost SAN ATA encapsulated into Ethernet frames Fibre Channel over Ethernet (FCoE): consolidate IP and FC/SAN networks FC encapsulated into Ethernet frames *oE: no TCP/IP overhead restricted to a single Layer 2 network no inherent security features iSCSI SCSI encapsulated in TCP/IP packets

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 53

CS615 - Aspects of System Administration Slide 53

Basic Disk Concepts Physical Disk Structure

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 54

CS615 - Aspects of System Administration Slide 54

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 55

CS615 - Aspects of System Administration Slide 55

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 56

CS615 - Aspects of System Administration Slide 56

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 57

CS615 - Aspects of System Administration Slide 57

Basic Disk Concepts: Physical Disk Structure

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 58

CS615 - Aspects of System Administration Slide 58

Basic Disk Concepts: Physical Disk Structure

Hard drive performance determined by: seek time rotational latency internal data rate a few other negligible factors (external data rate, command overhead, access time, etc.)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 59

CS615 - Aspects of System Administration Slide 59

Basic Disk Concepts: Disk Devices

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 60

CS615 - Aspects of System Administration Slide 60

Basic Disk Concepts Partitions

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 61

CS615 - Aspects of System Administration Slide 61

Basic Disk Concepts: Partitions

Source: SGI Techpubs

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 62

CS615 - Aspects of System Administration Slide 62

Basic Disk Concepts: Partitions

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 63

CS615 - Aspects of System Administration Slide 63

Basic Disk Concepts: Partitions

Source: NetBSD Guide

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 64

CS615 - Aspects of System Administration Slide 64

Basic Disk Concepts: Partitions

NetBSD example (from disklabel(8)) Partition ’a’: / Partition ’b’: swap Partition ’e’: /home # size

ffset

fstype [fsize bsize cpg/sgs] a: 20972385 63 4.2BSD 4096 32768 1180 # (Cyl. 0- 20805) b: 1048320 20972448 swap # (Cyl. 20806 - 21845) c: 78140097 63 unused # (Cyl. 0- 77519) d: 78140160 unused # (Cyl. 0 - 77519) e: 56119392 22020768 4.2BSD 4096 32768 58528 # (Cyl. 21846 - 77519)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 65

CS615 - Aspects of System Administration Slide 65

Basic Disk Concepts: Partitions

NetBSD example (from disklabel(8)) Partition ’a’: / 10 GB Partition ’b’: swap Partition ’e’: /home 26 GB # size

ffset

fstype [fsize bsize cpg/sgs] a: 20972385 63 4.2BSD 4096 32768 1180 # (Cyl. 0- 20805) b: 1048320 20972448 swap # (Cyl. 20806 - 21845) c: 78140097 63 unused # (Cyl. 0- 77519) d: 78140160 unused # (Cyl. 0 - 77519) e: 56119392 22020768 4.2BSD 4096 32768 58528 # (Cyl. 21846 - 77519)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 66

CS615 - Aspects of System Administration Slide 66

Basic Disk Concepts: Partitions

Solaris example (from format(1m)): Current partition table (original): Total disk cylinders available: 38758 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks root wm 3 - 3764 3.62GB (3762/0/0) 7584192 1 swap wu 3765 - 4364 590.62MB (600/0/0) 1209600 2 backup wm 0 - 38757 37.26GB (38758/0/0) 78136128 3 unassigned wm (0/0/0) 4 unassigned wm (0/0/0) 5 unassigned wm (0/0/0) 6 unassigned wm (0/0/0) 7 home wm 4365 - 38757 33.06GB (34393/0/0) 69336288 8 boot wu 0 - 0.98MB (1/0/0) 2016 9 alternates wu 1 - 2 1.97MB (2/0/0) 4032

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 67

CS615 - Aspects of System Administration Slide 67

Basic Disk Concepts: Partitions

Linux example (from fdisk(8)): Disk /dev/sda: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 33 265041 83 Linux /dev/sda2 34 9726 77859022+ 83 Linux

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 68

CS615 - Aspects of System Administration Slide 68

Basic Disk and Filesystem Concepts: RAID and Logical Volumes

allow file systems to be larger than the physical size of a disk inrease I/O performance when striped fault tolerant when mirrored or plexed

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 69

CS615 - Aspects of System Administration Slide 69

Hooray! 5 Minute Break

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 70

CS615 - Aspects of System Administration Slide 70

Basic Filesystem Concepts Filesystem Layout

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 71

CS615 - Aspects of System Administration Slide 71

Basic Filesystem Concepts

All partitions – with the exception of the root (or /) partition – can be mounted anywhere in the filesystem hierarchy.

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 72

CS615 - Aspects of System Administration Slide 72

Basic Filesystem Concepts

All partitions – with the exception of the root (or /) partition – can be mounted anywhere in the filesystem hierarchy. The file /etc/fstab (see fstab(5)) specifies which disks / partitions to mount where: /dev/wd0a / ffs rw 1 1 /dev/cgd1a none swap sw 0 0 /dev/cgd0a /home ffs rw 1 2 /ignoreme /tmp mfs rw,-b4096,-f512,-s262144 0 0 kernfs /kern kernfs rw procfs /proc procfs rw,noauto ptyfs /dev/pts ptyfs rw 0 0

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 73

CS615 - Aspects of System Administration Slide 73

Basic Filesystem Concepts

All partitions – with the exception of the root (or /) partition – can be mounted anywhere in the filesystem hierarchy. The file /etc/fstab (see fstab(5)) specifies which disks / partitions to mount where:

# /etc/fstab: static file system information. # # Use ’vol_id --uuid’ to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults # / was on /dev/sda2 during installation LABEL=ROOT / ext3 errors=remount-ro,acl 1 # /boot was on /dev/sda1 during installation LABEL=BOOT /boot ext3 defaults,acl 2 # swap was on /dev/sda5 during installation UUID=9329ae83-289d-4c3d-8756-f707c4bbb312 none swap sw /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 deathstar.phy.stevens-tech.edu:/export/nfs-sw/opt /opt nfs ro,rsize=32768,intr,nolock 0 0 deathstar.phy.stevens-tech.edu:/export/srcit-dist /mnt/srcit-dist nfs ro,rsize=32768,intr,nolock 0 0 corsario.cs.stevens-tech.edu:/export/people /mnt/legacy/people nfs rw,rsize=32768,wsize=32768,intr,lock 0 0 corsario.cs.stevens-tech.edu:/export/faculty /mnt/legacy/faculty nfs rw,rsize=32768,wsize=32768,intr,lock 0 0

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 74

CS615 - Aspects of System Administration Slide 74

Basic Filesystem Concepts

All partitions – with the exception of the root (or /) partition – can be mounted anywhere in the filesystem hierarchy. To see what filesystems are currently mounted, run mount(8): /dev/wd0a on / type ffs (local) /dev/cgd0a on /home type ffs (local) mfs:276 on /tmp type mfs (synchronous, local) kernfs on /kern type kernfs (local) ptyfs on /dev/pts type ptyfs (local)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 75

CS615 - Aspects of System Administration Slide 75

Basic Filesystem Concepts

$ mount /dev/sda2 on / type ext3 (rw,errors=remount-ro,acl) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) varrun on /var/run type tmpfs (rw,nosuid,mode=0755) varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777) udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) fusectl on /sys/fs/fuse/connections type fusectl (rw) lrm on /lib/modules/2.6.28-17-generic/volatile type tmpfs (rw,mode=755) /dev/sda1 on /boot type ext3 (rw,acl) securityfs on /sys/kernel/security type securityfs (rw) automount(pid2623) on /home type autofs (rw,fd=4,pgrp=2623,minproto=2,maxproto=4) deathstar.phy.stevens-tech.edu:/export/nfs-sw/opt on /opt type nfs (ro,rsize=32768,intr,nolock,addr=155.246.89.4) deathstar.phy.stevens-tech.edu:/export/srcit-dist on /mnt/srcit-dist type nfs (ro,rsize=32768,intr,nolock,addr=155.246.89.4) corsario.cs.stevens-tech.edu:/export/people on /mnt/legacy/people type nfs (rw,rsize=32768,wsize=32768,intr,lock,addr=155.246.89.20) corsario.cs.stevens-tech.edu:/export/faculty on /mnt/legacy/faculty type nfs (rw,rsize=32768,wsize=32768,intr,lock,addr=155.246.89.20) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev) deathstar.phy.stevens-tech.edu:/export/home/kamberov on /home/kamberov type nfs (rw,sync,intr,vers=3,sloppy,addr=155.246.89.4) deathstar.phy.stevens-tech.edu:/export/home/mweiss on /home/mweiss type nfs (rw,sync,intr,vers=3,sloppy,addr=155.246.89.4) deathstar.phy.stevens-tech.edu:/export/home/jschauma on /home/jschauma type nfs (rw,sync,intr,vers=3,sloppy,addr=155.246.89.4)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 76

CS615 - Aspects of System Administration Slide 76

Basic Filesystem Concepts

Some of the different kinds of filesystems:

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 77

CS615 - Aspects of System Administration Slide 77

Basic Filesystem Concepts

Some of the different kinds of filesystems: “Regular” File Systems Journaling File Systems Network File Systems Various

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 78

CS615 - Aspects of System Administration Slide 78

Basic Filesystem Concepts The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 79

CS615 - Aspects of System Administration Slide 79

Basic Filesystem Concepts: The UNIX Filesystem

The filesystem is responsible for storing the data on the disk. So to read/write data, it needs to know in which physical blocks the actual data is located; ie how to map files to the disk blocks.

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 80

CS615 - Aspects of System Administration Slide 80

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 81

CS615 - Aspects of System Administration Slide 81

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 82

CS615 - Aspects of System Administration Slide 82

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 83

CS615 - Aspects of System Administration Slide 83

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 84

CS615 - Aspects of System Administration Slide 84

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 85

CS615 - Aspects of System Administration Slide 85

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 86

CS615 - Aspects of System Administration Slide 86

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 87

CS615 - Aspects of System Administration Slide 87

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 88

CS615 - Aspects of System Administration Slide 88

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 89

CS615 - Aspects of System Administration Slide 89

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 90

CS615 - Aspects of System Administration Slide 90

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 91

CS615 - Aspects of System Administration Slide 91

Basic Filesystem Concepts: The UNIX Filesystem

The filesystem is responsible for storing the data on the disk. So to read/write data, it needs to know in which physical blocks the actual data is located; ie how to map files to the disk blocks. Components of the Berkeley Fast Filesystem: set of inode storage cells df -i

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 92

CS615 - Aspects of System Administration Slide 92

Basic Filesystem Concepts: The UNIX Filesystem

The filesystem is responsible for storing the data on the disk. So to read/write data, it needs to know in which physical blocks the actual data is located; ie how to map files to the disk blocks. Components of the Berkeley Fast Filesystem: set of inode storage cells set of scattered “superblocks” newfs -N /dev/rdsk/c1t2160d0s0

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 93

CS615 - Aspects of System Administration Slide 93

Basic Filesystem Concepts: The UNIX Filesystem

The filesystem is responsible for storing the data on the disk. So to read/write data, it needs to know in which physical blocks the actual data is located; ie how to map files to the disk blocks. Components of the Berkeley Fast Filesystem: set of inode storage cells set of scattered “superblocks” map of disk blocks

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 94

CS615 - Aspects of System Administration Slide 94

Basic Filesystem Concepts: The UNIX Filesystem

The filesystem is responsible for storing the data on the disk. So to read/write data, it needs to know in which physical blocks the actual data is located; ie how to map files to the disk blocks. Components of the Berkeley Fast Filesystem: set of inode storage cells set of scattered “superblocks” map of disk blocks block usage summary fstyp -v /dev/rdsk/c1t2160d0s0 | more

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 95

CS615 - Aspects of System Administration Slide 95

Basic Filesystem Concepts: The UNIX Filesystem

The filesystem is responsible for storing the data on the disk. So to read/write data, it needs to know in which physical blocks the actual data is located; ie how to map files to the disk blocks. Components of the Berkeley Fast Filesystem: set of inode storage cells set of scattered “superblocks” map of disk blocks block usage summary set of data blocks

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 96

CS615 - Aspects of System Administration Slide 96

Basic Filesystem Concepts: The UNIX Filesystem

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 97

CS615 - Aspects of System Administration Slide 97

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode:

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 98

CS615 - Aspects of System Administration Slide 98

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 99

CS615 - Aspects of System Administration Slide 99

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 100

CS615 - Aspects of System Administration Slide 100

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions)

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 101

CS615 - Aspects of System Administration Slide 101

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions) file access and modification time

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 102

CS615 - Aspects of System Administration Slide 102

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions) file access and modification time file status modification time

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 103

CS615 - Aspects of System Administration Slide 103

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions) file access and modification time file status modification time number of links to the file

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 104

CS615 - Aspects of System Administration Slide 104

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions) file access and modification time file status modification time number of links to the file size of the file

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 105

CS615 - Aspects of System Administration Slide 105

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions) file access and modification time file status modification time number of links to the file size of the file disk device containing this file

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 106

CS615 - Aspects of System Administration Slide 106

Basic Filesystem Concepts: The UNIX Filesystem

Information stored in an inode: user owner and group owner ID’s file type access mode (permissions) file access and modification time file status modification time number of links to the file size of the file disk device containing this file $ stat /etc/passwd

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 107

CS615 - Aspects of System Administration Slide 107

Basic Filesystem Concepts: The UNIX Filesystem

File types: regular files $ stat /etc/passwd

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 108

CS615 - Aspects of System Administration Slide 108

Basic Filesystem Concepts: The UNIX Filesystem

File types: regular files directories $ stat /

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 109

CS615 - Aspects of System Administration Slide 109

Basic Filesystem Concepts: The UNIX Filesystem

File types: regular files directories special files $ file /dev/* | more

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 110

CS615 - Aspects of System Administration Slide 110

Basic Filesystem Concepts: The UNIX Filesystem

File types: regular files directories special files links $ touch /tmp/foo $ ln /tmp/foo /tmp/bar $ stat /tmp/foo /tmp/bar $ ln -sf /tmp/foo /tmp/bar $ stat /tmp/foo /tmp/bar

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 111

CS615 - Aspects of System Administration Slide 111

Basic Filesystem Concepts: The UNIX Filesystem

File types: regular files directories special files links sockets $ stat /dev/log

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 112

CS615 - Aspects of System Administration Slide 112

Basic Filesystem Concepts: The UNIX Filesystem

File types: regular files directories special files links sockets named pipes $ mkfifo /tmp/fifo $ cat /tmp/fifo > /tmp/out & $ stat /tmp/fifo | tee /tmp/fifo $ cat /tmp/out

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 113

CS615 - Aspects of System Administration Slide 113

Homework

Repeat the examples from class. Make sure you understand the commands and how they relate to the concepts we discussed. Repeat for a different OS, for example: ami-3b361952 – Fedora 23 ami-f709a29c – FreeBSD 10.2 ami-569ed93c – NetBSD 7.0 ami-50ecc847 – OmniOS 5.11 Remember to shut down your EC2 instances and to delete any unused ESB volumes! https://www.cs.stevens.edu/~jschauma/615/s16-hw2.html

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 114

CS615 - Aspects of System Administration Slide 114

Reading

http://is.gd/5mndwA http://is.gd/ig4QP5 http://is.gd/9YeIKh

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 115

CS615 - Aspects of System Administration Slide 115

Reading

Disk Interfaces: SCSI: http://en.wikipedia.org/wiki/Scsi scsi(4), scsictl(8); ATA: http://www.ata-atapi.com/ https://en.wikipedia.org/wiki/Advanced_Technology_Attachment https://en.wikipedia.org/wiki/Sata

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 116

CS615 - Aspects of System Administration Slide 116

Reading

Disk Interfaces: Serial attached SCSI: https://en.wikipedia.org/wiki/Serial_attached_SCSI Fibre Channel: https://hsi.web.cern.ch/HSI/fcs/fcs.html https://en.wikipedia.org/wiki/Fibrechannel AoE, FCoE, iSCSI: https://en.wikipedia.org/wiki/ATA_over_Ethernet https://en.wikipedia.org/wiki/FCoE https://en.wikipedia.org/wiki/ISCSI

Lecture 02: Filesystems, Disks, Storage January 30, 2017

SLIDE 117

CS615 - Aspects of System Administration Slide 117

Reading

Basic Disk Concepts: disklabel(8), fdisk(8) format(1m) RAID: https://en.wikipedia.org/wiki/RAID Basic Filesystem Concepts: http://is.gd/8KHnQj https://is.gd/wGgJ0e newfs(8) NFS: http://is.gd/70yqMZ

Lecture 02: Filesystems, Disks, Storage January 30, 2017