Enabling CernVM for the Interactive Use Case Vasilis Nicolaou SFT - - PowerPoint PPT Presentation

Enabling µCernVM for the Interactive Use Case

Vasilis Nicolaou

SFT Group CERN vasilis.nicolaou@cern.ch Supervisor: Dr. Jakob Blomer

Summer, 2013

Overview

1

Web appliance agent for the µCernVM

2

Merging user databases

3

Merging RPM databases

4

Reimplementation of the cernvm-update script

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 2 / 25

Web appliance agent for the µCernVM Motivation

Motivation

The µCernVM will be used by physicists to access LHC data for analysis and development being the successor of the CernVM which had an open source appliance agent designed by rPath that is no longer

• available. A new one needs to be developed

to: Provide valuable information to reduce the users need of using shell commands Provide a user friendly layer for interacting with the system for simple tasks such as updating or powering off the system Provide an easy mechanism for expanding the web application by adding shell commands µCernVM architecture In summary: A read-only layer (R-O layer) and is mounted from the network A read/write layer (R/W layer) which is created on first boot

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 3 / 25

Web appliance agent for the µCernVM Motivation

Web appliance agent for the µCernVM

What is the web appliance agent? A web application that provides basic information related to the state of the µCernVM such as memory usage, update notifications and cron jobs. Supports basic user interaction for altering the state of the machine such as updating the underlying operating system.

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 4 / 25

Web appliance agent for the µCernVM User interaction

User interaction with the appliance agent

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 5 / 25

Web appliance agent for the µCernVM Extending the web appliance agent

Extending the web appliance agent as a user

API Management screen Information while API is enabled Helps the users with adding useful bash commands Supports deleting previously added bash commands Supports two formats of the generated HTML view; raw and table There is a technical report that contains more details on how to use the API and sample code

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 6 / 25

Web appliance agent for the µCernVM Extending the web appliance agent

Extending the web appliance agent as a user

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 7 / 25

Web appliance agent for the µCernVM Extending the web appliance agent

Extending the web appliance agent as a developer

The framework of the web application was developed according to the model-view-controller architecture It does not matter how you develop your application as long as it directs its HTML output in a certain way Every embedded application should load the Configuration module and create the view. When the application has some output to display it should call the view.setContent in

• rder to display the contents on the application reserved area

Example code

sys.path.append(os.environ[’MY_HOME ’]+ ’/ etc/config ’) sys.path.append(os.environ[’MY_HOME ’]+ ’/cgi -bin/chrome ’) from cern_vm import Configuration from view import View #[..]# config= Configuration () view = View(config.system.actions)

• view. setContent ("My

application ", "Hello World ") Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 8 / 25

Web appliance agent for the µCernVM Extending the web appliance agent

Extending the web appliance agent as a developer

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 9 / 25

Merging user databases

Merging user database

Shadow, passwd and group

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 10 / 25

Merging user databases

Merging user database

The user database consists of three files:

1 /etc/passwd contains basic information about the users such as their

username, id and login shell

2 /etc/group contains basic information about the groups such as their

id, members and permissions

3 /etc/shadow is where the user passwords are stored

The users can modify all the three files, which may cause conflicts

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 11 / 25

Merging user databases The problem of having two versions of user databases

The problem of having two versions of user databases

Conflicts with IDs, usernames, password inconsistency and changes in group members have to be resolved.

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 12 / 25

Merging user databases The current solution

The current solution

Merging /etc/group|passwd|shadow with the respective files from the R-O layer

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 13 / 25

Merging user databases Deploying the solution to the µCernVM

Deploying the solution to the µCernVM

Before any service starts, a series of scripts are executed to prepare the µCernVM by mounting the cvmfs and merging it with the R/W layer on the local machine. User database is critical and it must be ready before services and daemons run since they

• ften run under a system

user account.

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 14 / 25

Merging RPM databases

Merging RPM databases

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 15 / 25

Merging RPM databases

Merging RPM databases

Two RPM databases: One in the R-O file system and one generated on the first boot and placed in the R/W layer The R/W can be modified When a new RPM database is available on the R-O layer, a mechanism must exist to merge the two into one new R/W database

The R/W and R-O RPM databases Example rpm R/W and R-O diff after installing gedit

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 16 / 25

Merging RPM databases The algorithmic solution

The algorithmic solution simplified

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 17 / 25

Reimplementation of the cernvm-update script

Reimplementation of the cernvm-update script

The algorithm that merges the rpm databases is a vital part of the cernvm-update script which has the role of the update manager and it is responsible to leave the system in a state identical to what it would have been on a desktop Linux distribution. Usage Argument Action c check for new updates a Apply the new update if it exists p Patch the update with the user packages

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 18 / 25

Reimplementation of the cernvm-update script Fetching the update packs via mounting cvmfs

Fetching the update packs via mounting cvmfs

Configuration file

CVMFS_CACHE_BASE = $tmplocation /cache CVMFS_RELOAD_SOCKETS = $tmplocation /cache CVMFS_SERVER_URL =\ http :// hepvm.cern.ch/cvmfs/cernvm -devel.cern.ch CVMFS_HTTP_PROXY =DIRECT CVMFS_KEYS_DIR =/ mnt /.rw/aux

Calling the cvmfs2 tool

cvmfs2 -o config=" $configuration "\ $(attr -qg fqrn /mnt /.ro)\ " $tmplocation / mountpoint "\ >/dev/null

Unmounting and clean up

fusermount

• u $tmplocation / mountpoint

rm -rf $tmplocation

The remote cvmfs repository is mounted Check latest update-pack against the current system version When a new update exists the user may apply the update Sanity checks take place Once everything is OK, the system will be updated on next reboot

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 19 / 25

Reimplementation of the cernvm-update script Applying for the updates

Applying for the updates

’Applying’ in this context does not mean that the updates are installed. The system is left in a state that when it will be rebooted the update process will be initiated. Remove the .cvmfs_snaphot Set in /mnt/.rw/ucontext/context file the version that the system is going to be updated to On reboot the rebase will take place meaning that new files will replace old

• nes.

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 20 / 25

Reimplementation of the cernvm-update script Patching the R/W layer with the system updates

Patching the R/W layer with the system updates

There are rules on which files are replaced: Normally most files under /var and /etc are kept However there is a problem with some of those files such as the user database consisting

• f /etc/passwd|group|shadow

and /var/lib/rpm/* . Such files cannot just be replaced or be left as they are, a merging strategy is needed Process Summary

The R-O RPM database is copied to a temporary directory and the RPM replicas generated at pre-update stage are installed by referencing the temporary database The database then replaces the one under /var/lib/rpm A new empty rpm database is created under a temporary directory The post installation scripts that were chosen from pre-update stage are executed via calling rpm -i that points to the temporary database requesting that no dependencies are checked

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 21 / 25

Reimplementation of the cernvm-update script User interaction with the update mechanism

User interaction with the update mechanism

Use the cernvm-update script Use a special button to press and apply the updates Get notifications from the desktop environment or the live feed inside the web interface

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 22 / 25

Reimplementation of the cernvm-update script Notifying the users for new updates

Notifying the users for new updates

A cronjob that runs every hour was developed to notify users for new updates. Does not pop up annoying notifications Pops up at most one notification each day except if there is a newer update than the last notification stated Pops up notification for failure to reach the server if the server was unavailable for more than 1 day

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 23 / 25

Reimplementation of the cernvm-update script Further reading

Further reading

Technical report of the cernvm appliance agent API Report written for the University of Manchester as a master thesis for the MEng Software Engineering program Summer student report

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 24 / 25

Questions

Vasilis Nicolaou (CERN) Enabling µCernVM for IUC Summer 2013 25 / 25