Linux Under the Hood Manual Pages & Info Pages Distribution - - PowerPoint PPT Presentation

linux under the hood
SMART_READER_LITE
LIVE PREVIEW

Linux Under the Hood Manual Pages & Info Pages Distribution - - PowerPoint PPT Presentation

Linux Under the Hood Manual Pages & Info Pages Distribution Differences Package Management Startup and Shutdown Pesky Permissions Common/Useful Tools and Services cron find rsync Manual Pages & Info Pages


slide-1
SLIDE 1

Linux – Under the Hood

Manual Pages & Info Pages Distribution Differences

  • Package Management
  • Startup and Shutdown

Common/Useful Tools and Services

  • cron
  • find
  • rsync

Pesky Permissions

slide-2
SLIDE 2

Manual Pages & Info Pages

  • Pretty much all linux commands have a manual page.
  • You can access them with

– man <command>

  • At the bottom of a page you will also often see a See also section like so

– SEE ALSO – Regular Manual Pages – awk(1), cmp(1), diff(1), find(1), gzip(1), perl(1), sed(1),

sort(1), xargs(1), zgrep(1), read(2), pcre(3), pcresyntax(3), pcrepattern(3), terminfo(5), glob(7), regex(7).

  • The numbers after each command refer to sections of the manual. For example if

you wish to view the page about regex which is in section 7 (sometimes commands will have man pages for the same command in multiple sections) you can use

– man 7 regex

  • In addition to the manual pages there are also info pages for a lot of common
  • commands. These can be accessed by using

– info <command>

slide-3
SLIDE 3

Distribution Differences – Package Managers

  • Debian / Ubuntu

– Uses DPKG (debian package) packages. – Uses the apt (Advanced Package Tool) repository system.

  • CentOS/Redhat (and Fedora)

– Uses the RPM (Redhat package manager) package manager. – CentOS & Redhat Use Yum (Yellowdog updater). – Fedora as of version 22 uses DNF (Dandified YUM) rather

than yum

– but the commands are in most cases identical to those for

yum.

slide-4
SLIDE 4

Distribution Differences – Package Managers

  • Managers vs Packages

– yum/dnf and apt both manage the finding of packages

and downloading them. They are also responsible for finding any packages the program your installing may also depend on and installing it too.

– rpm and dpkg are used to query and install/remove

the actual packages. They are what extract the archives and actually put the files in place and run scripts. These can also be used to query packages to list what files they provide etc….

Equivalent Ubuntu/Debian and Redhat/CentOS/Fedora Package Manager Commands

slide-5
SLIDE 5

Distribution Differences – System Services

  • init.d vs systemd

– init.d is a well known structure for scripts that start

and stop services. Depending on the age of your distribution you may or may not be using init.d

  • scripts. Whilst most (not all) distributions seem to be

moving to systemd some resist.

– The main script files are usually stored in /etc/init.d

The scripts in this directory are called with a start

  • r stop parameter which then does the needed work to

stop and start any particular service

slide-6
SLIDE 6

Distribution Differences – System Services

  • init.d

– initd has a concept of run levels for the system numbered 0 – 6 (usually)

  • Run level 0 – Shutdown
  • Run level 1 – Single User (recovery mode no services running)
  • Run level 2 – Ubuntu Multi user without network / Redhat Not used (user-definable)
  • Run level 3 – Full multi-user with network - text mode (no GUI/display manager)
  • Run level 4 – Not used – user definable
  • Run level 5 – Ubuntu - Run level 3 + display manager / Redhat - Full multi-user graphical

mode (with an X-based login screen)

  • Run level 6 – reboot

– The initial run level of a system using init.d scripts is set by the file

/etc/inittab

– The startup/shutdown order of scripts is defined by the directories /etc/rc.d/rc.

[0-6] which contain symbolic links to the scripts in /etc/init.d

slide-7
SLIDE 7

Distribution Differences – System Services

  • systemd
  • Uses the command systemctl to manage the starting and stopping of services.
  • Unlike init.d scripts systemd does not need to start everything in a specific order as it pre-

creates things like socket files so that the daemons can attach to them once they are started. Because of this systemd startup times are often extensively faster than those using init.d.

  • Systemd tasks are organised as units. There are several types

– Services (.service) – Mount points (.mount) – Devices (.device) – Sockets (.socket) – Timers (.timer)

  • The units are put into a control group named after the service. The system allows for a high degree
  • f parallelism.
  • Units are defined in files, normally these are located in /usr/lib/systemd/system but you can also

create them in /etc/systemd/system. Files created in the /etc directory will take prescience over the ones defined in /usr

  • For small tweaks to a service there is a drop in directory structure which can be used to tweak

settings.

slide-8
SLIDE 8

Distribution Differences – System Services

  • The following distrubibutions use systemd

– The versions listed are the first version where systemd was default, all

distributions require systemd except where noted.

– CentOS 7.14.04 – Debian v8 (Can run without systemd) – Fedora v15+ (current is 28, was the first to provide systemd) – Mint v18 – RedHat v7 – SUSE v12 – Ubuntu 13.04 (Can run without systemd)

  • Most distributions provide a copy of the init.d service command which translates

init.d style commands into systemd ones.

slide-9
SLIDE 9

Permissions

  • Standard Unix Permissions File/Dir

Properties

– Type of file – Permissions (User, Group Other,

ACL/Sec Context)

– Hard Links – File Owner (Name looked up from

their UID in /etc/passwd)

– File Group (Name looked up from

their GID in /etc/group)

– Timestamp (usually modified time) – Filename

slide-10
SLIDE 10

Permissions

  • The command stat shows you more detail about a

file including User id (Uid) and Group id (Gid)

  • It also shows the permissions as a bit mask

– 0664/-rw-rw-r-- – The first number is a special mode bit

normally this is just 0 unless your using sticky bits (which would be another talk!!)

– The 3 basic bits you really need to start with

are:

  • 4 – Read
  • 2 – Write
  • 1 – Execute (Note: on a directory this says you are

allowed to list the contents of the directory)

  • For those wanting more understanding there is an

info page which details all this

– info coreutils – Select Changing file attributes → Then chmod [lug@dave ~]$ stat hello File: hello Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd02h/64770d Inode: 105 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ lug) Gid: ( 1000/ lug) Access: 2018-10-23 21:51:56.219635662 +0100 Modify: 2018-10-23 21:51:56.219635662 +0100 Change: 2018-10-23 21:51:56.219635662 +0100 Birth: -

  • The 0644 is the same as the more visual indications of

permissions in this case in order.

– 0 – The object is a File – 6 – User Permission – so 4 & 2 = 6 which = read and write

for the user

– 4 – Group Permission – so 4 on its own = read for the

group

– 4 – Other Users Permission – so again 4 on its own =

  • ther users can read (or if a directory can list the

directory contents

slide-11
SLIDE 11

Permissions

  • Changing ownership

– You can change a files

  • wner using chown

– chown <user>

<file/directory

– An example would be chown

pi mydatafile

– You can also change the

group at the same time by adding a : and then the group name

– chown pi:pi mydatafile

  • Changing the Group (only)

– You can change the group of a file – chgrp dialout myscript – This would allow members of the dialout

group to have access to the file myscript with the appropriate permissions set on the file for that group.

  • Its important to make sure

your files are owned by the right user before setting the permissions !!!

slide-12
SLIDE 12

Permissions

  • To change a files permissions you can use

the chmod command.

– chmod can use both the numeric and rwx

versions of permissions. I suggest you start with the rwx method on and read up

  • n the other !

– In its basic form you can tell chmod to

change permissions for a file for the following

  • User (u)
  • Group (g)
  • Other (o)
  • All (a) – Affects all 3 of the above

– You tell it what to do to those entities

  • + Add the following permissions
  • Remove the following permissions
  • = Set to these permissions
  • Here are most of the permissions you can

adjust

– Read (r) – Write (w) – Execute (or in the case of

directories allow the user,group,other to list the contents of the directory) (x)

  • These are demonstrated on the next slide.

Its important to make sure you know if your adding a permission or removing it and more importantly setting it (=) as setting the incorrect permission can be a

  • problem. Sometimes its best to add the

permission rather than set, this will then retain the permissions it has and just add the additional option for read rather than overwriting to just read!

slide-13
SLIDE 13

Permissions

  • A simple example: to perform the

following actions on the file myscript

– give a user the read permission

  • chmod u+r myscript

– Remove the read permission

  • chmod u-r myscript

– Give the group read permission

  • chmod g+r myscript

– Remove the group read permission

  • chmod g-r myscript

– The same format goes for the other

users just replace the u or g with

  • You can also do combinations

– chmod ug+rwx myscript

  • This sets the user and group read,

write and execute permissions

– chmod ug+rwx myscript

  • This Adds the user and goup read,

write and execute permissions

  • Set all to add read and execute

– chmod a+rx myscript

  • Set all to only have read and execute

– chmod a=rx myscript

  • Finally a slightly more complex example.

Set the user to read + execute, the group to read and write and remove read, write and execute from other

– chmod u+rx,g=rx,o-rwx myscript

slide-14
SLIDE 14

Cron

  • Cron

– Used to schedule jobs to

be run regularly (usually scripts)

– Can be scheduled with up-

to 1 min accuracy

– Scripts normally start do

a job and quit. Jobs that start and keep running should be started as a system service using init.d or systemd (In systemd this is very easy)

  • To edit the crontab for your user

(which will run the script as your user) you should use the command

– crontab -e – This will fire up the default

editor (normally vim or vi) if your not a fan of this you can override this behaviour by setting a variable like so before

– export EDITOR=nano – Then run the crontab -e command

again it should now open up the crontab in the editor nano (replace nano with the editor of your choice). If your using bash you could add the export line to the file ~/.bash_profile which should be run each time you login

slide-15
SLIDE 15

Cron

  • On the right is a comment

taken from /etc/crontab

  • The header is actually nice

to paste into your crontab for your user as its actually a comment.

  • Whilst you can use the above

crontab file I would not recommend exiting the above file I recommend the use of crontab -e instead.

# For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed

slide-16
SLIDE 16

Cron

  • To schedule a script in this

case we will use the script name below as an example

– /home/pi/mybackup.sh

  • We need to edit the crontab

using

– crontab -e

  • To schedule the job to run

every day at midnight you would use the following line

0 0 * * * /home/pi/mybackup.sh # Every Monday at 1:30pm 30 13 * * 1 /home/pi/mybackup.sh # Run the script every minute 0-59/1 * * * * /home/pi/mybackup.sh # 2pm on the first day of January 0 14 1 jan * /home/pi/mybackup.sh

slide-17
SLIDE 17

Cron

  • There are 3 ways to setup a cron

– crontab -e

  • This edits a crontab for your user. You

can specify an alternate user by adding

  • u <username>

– /etc/cron.d/<filename>

  • This is a directory in which files can be

put to be used a cron entries (This is where rpm & dpkg packages would normally place entries for crons)

– /etc/crontab

  • This file is a central place and and

used to be the default for system crons and whilst it can be used I would recommend using one of the 2 above methods

  • One final thing for

crontab.

  • If you are performing

backups and don’t backup the whole server its always worth adding the following files and directories, including all sub directories and files, to your backup scripts.

– /etc/crontab – /etc/cron.* – /var/spool/cron

slide-18
SLIDE 18

Find

  • Locating items especially in large

file system tree’s can be difficult.

  • Find allows you to search with a lot
  • f flexibility and even do something

with what you have found

  • The format for the find command is as

follows

– find <options> <starting-path>

<expression>

  • Options are not normally needed they

are mainly related to how find should deal with symbolic links

  • Expression is essentially a

description of what your looking for. The options for this are extensive!!

  • A simple example

– find /var -iname “spool” – This will look for any file or

directory that's called spool

  • You can use wildcards in the iname too

– find /var -iname “*.log”

  • If you only want directories (d) or files

(f) you can use the -type option and this can be combined with other options

– find /var -type f -iname “*audit*.log”

  • Just be careful that you give the option

followed by its parameter before specifying the next option

slide-19
SLIDE 19

Find – An Intermission

  • One problem you can come across with

find is the errors if for example you don’t have permission to view a file/directory

  • You can use shell redirection to fix

this

– find var -type f -iname “*.log”

2>/dev/null

  • Any command run will have 1 input

(stdin) and 2 outputs (stdout (1) and stderr (2)). The above command tells the shell to redirect the contents of stderror to the file /dev/null (which just throws them away)

  • This is equally important when logging scripts
  • utput. If something outputs to stderr and you

dont redirect it it will display on the terminal and not be logged.

  • To log the 2 outputs you can do the following

(note that 1> can simply be changed to > for stdout)

– ./myscript 1> mylogfile.log 2>myerrors.log

  • Sometimes you would prefer to have both the

errors and the output together in this case you need to redirect stdoutput to the file first !! then you can redirect stderr to be added to stdout

– ./myscript > mylogfile.log 2>&1 – Putting the > after the 2>&1 would

actually not write to the log file!!

slide-20
SLIDE 20

Find

  • Ok Back to find
  • Find is able to do more than just list files if can run

commands on them.

– find linux-4.18.16 -type f -iname “*.c” -exec grep

  • nH ‘on fire’ {} \;
  • The above command will look for a file with the extension

.c and then for each file matched run the command grep on it looking for the phrase “on fire” the options for grep tell it to tell us the line number in the file and the file name for each match

slide-21
SLIDE 21

Find

  • Find is able to do more than just list files if can run commands on them.

– find linux-4.18.16 -type f -iname “*.c” -exec grep -nH ‘on fire’ {} \;

  • The above command will look for a file with the extension .c and then for each file matched run the command

grep on it looking for the phrase “on fire” the options for grep tell it to tell us the line number in the file and the file name for each match

  • You can also do basic logic between options. For example to find all header files with the extension .h and

all C files with the extension .c

– find linux-4.18.16 -type f \( -iname “*.h” -o -iname “*.c” \)

  • Note the \ before the open and close brackets. This is to tell bash to not to try and interpret them. This

command will in fact work without them but they are useful to know about so you can ensure your search is interpreted in the correct way.

  • How to find the newest files written to in a location (usual log files for me)

– touch now – find /var/log -newer now

  • The above creates an empty file now with a timestamp of now
  • The find command then looks in /var/log for any file which has a modification time stamp > than the one of

the file now.

slide-22
SLIDE 22

Rsync

  • Rsync is a fast tool for copying files. Its biggest benefit is that it

can compare files between 2 locations before copying and if the files are identical it can skip the file. This makes it ideal for tasks like backups.

  • Internally rsync makes 2 lists of files one of the source and one of the

destination it then compares file names, followed by file sizes and timestamps to work out what needs copying.

  • It is then able to break the files into chunks and only transfer the

differences between the 2 locations. This makes it highly efficient over networks but can use a lot of disk IO at each end. Normally this IO is still less than just re-copying the file each time would use.

  • Rsync has its own protocol but also works with ssh so you can easily

perform encrypted transfers.

slide-23
SLIDE 23

Rsync

  • The basic command for rsync is

– rsync <options> <source> <destination> – rsync -avh mysourcedir mydestdir

  • If you wish to use ssh then the directory changes.

– rsync <options> user@host:source destination – rsync <options> source user@host:destination

  • The options have the basic fuctions

– -a is a basic option known as archive mode and is the set of options “archive mode; equals -rlptgoD” – -v is increase verbosity – -h prints numbers in a human readable format

  • 2 other options which are of use

– --stats Gets rsync to provide statistics on the transfer of files – --progress

Gets rsync to output the progress of the file transfer

  • Rsync is a complete talk by it’s self I will perform some demo’s here if anyone is interested