Testing OSG Software Mtys Selmeci OSG Software Lead Developer OSG - - PowerPoint PPT Presentation

▶
testing osg software
SMART_READER_LITE
LIVE PREVIEW

Testing OSG Software Mtys Selmeci OSG Software Lead Developer OSG - - PowerPoint PPT Presentation

Oct 31, 2022 242 likes •517 views

Testing OSG Software Mtys Selmeci OSG Software Lead Developer OSG All Hands Meeting SLAC 10 April 2014 Overview Automated Testing osg-test Automated test runs Manual Testing Developer Acceptance Integration Test Bed (ITB)

slide-1
SLIDE 1

Testing OSG Software

Mátyás Selmeci


OSG Software Lead Developer OSG All Hands Meeting · SLAC · 10 April 2014

slide-2
SLIDE 2

Testing OSG Software 10 April 2014

Overview

Automated Testing

  • sg-test

Automated test runs

Manual Testing

Developer Acceptance Integration Test Bed (ITB)

Testing and Releases Future

2

slide-3
SLIDE 3

Testing OSG Software 10 April 2014

  • sg-test

3

slide-4
SLIDE 4

Testing OSG Software 10 April 2014

  • sg-test
  • Software that tests OSG software stack
  • Tries to mimic what admins do

– Install RPMs · configure (commands, files) – Run services · use client tools – Clean up · remove RPMs

  • Works best on virtual machine (VM)

– Needs administrator privileges (root) – Avoid conflicts with existing software

  • One run tests all installed components

4

slide-5
SLIDE 5

Testing OSG Software 10 April 2014

Test Coverage

HTCondor GRAM GridFTP Gratia probes VOMS VOMS Admin GUMS BeStMan XRootD Fetch CRL RSV gLExec

5

slide-6
SLIDE 6

Testing OSG Software 10 April 2014

Test Run Output

6

FAIL: test_03_myproxy_init (osgtest.tests.test_48_myproxy.TestMyProxy) Traceback (most recent call last): · · · EXIT STATUS: 1 STANDARD OUTPUT: MyProxy v5.9 Jul 2012 PAM SASL KRB5 LDAP VOMS OCSP · · · STANDARD ERROR: [none]

  • BAD SKIPS:

test_04_my_proxy_retrieval (osgtest.tests.test_48_myproxy.TestMyProxy) MyProxy creation failed

  • Ran 271 tests in 2367.078s

FAILED (failures=2, badSkips=1, okSkips=15)

slide-7
SLIDE 7

Testing OSG Software 10 April 2014

Automated Tests

7

slide-8
SLIDE 8

Testing OSG Software 10 April 2014

Automation Goals

One run tests one environment Want many runs, varying:

Operating system Installed package(s) Repositories used

Fully automated Harness power of HTC Clear reporting

8

slide-9
SLIDE 9

Testing OSG Software 10 April 2014

Testing in CHTC

  • Use VM universe feature of HTCondor to

run osg-test as opportunistic VM jobs

  • Supported by developers and admins at

Center for High Throughput Computing

  • Workflow adjusts to input parameters:

– All 6 OSG operating systems (EL5 & EL6) – 6 package sets (e.g., CE, BeStMan SE, GUMS) – 9 repository combos (e.g., release, 3.1 ⇒ 3.2)

  • Automated nightly runs or on demand

9

slide-10
SLIDE 10

Testing OSG Software 10 April 2014

Test Workflow

10

Prepare Workflow Report Results

  • sg-test

Extract

  • sg-test

Extract

  • sg-test

Extract

  • sg-test

Extract

⁞

slide-11
SLIDE 11

Testing OSG Software 10 April 2014

Benefits of VM Tests

  • “Nightlies”: 324 test runs in 2–4 hours
  • Customized run: 1–4 hours
  • Has found real issues

– OpenJDK 7 dependency changes – OpenSSL changes to minimum proxy lengths – OpenSSL packaging changes – jGlobus / BeStMan problems with certificates

11

slide-12
SLIDE 12

Testing OSG Software 10 April 2014

Report Sample

12

Cent 5 RH 5 SL 5 Cent 6 RH 6 SL 6 Everything 241 14 1 241 14 1 241 14 1 242 14 0 241 14 1 242 14 0 HTCondor 86 169 1 86 169 1 86 169 1 88 168 0 88 168 0 88 168 0 GridFTP 70 186 0 70 186 0 70 186 0 71 185 0 71 185 0 71 185 0 BeStMan 79 177 0 78 177 1 79 177 0 80 176 0 80 176 0 80 176 0 VOMS 93 163 0 93 163 0 93 163 0 94 162 0 94 162 0 94 162 0 GUMS 79 177 0 78 177 1 79 177 0 80 176 0 80 176 0 80 176 0

3.1 Production ⇒ 3.1 Testing [1/9]

http://vdt.cs.wisc.edu/tests/latest.html

slide-13
SLIDE 13

Testing OSG Software 10 April 2014

Manual Testing

13

slide-14
SLIDE 14

Testing OSG Software 10 April 2014

Why Test by Hand?

Lack coverage in osg-test yet

We keep adding tests …

If test cases are hard to code

Client and server on separate machines Simulating user input to web pages

If test covers a one-time change Exploratory testing

Unexpected output Unusual messages in log files

14

slide-15
SLIDE 15

Testing OSG Software 10 April 2014

Developer Testing

Check for basic errors

Packaging mistakes Software does not work at all

Usually install isolated software

E.g., not a whole Compute Element

Generally very lightweight

15

slide-16
SLIDE 16

Testing OSG Software 10 April 2014

Sample Dev Test

16

Carl, testing new GridFTP UDT feature:

# yum install osg-gridftp Loaded plugins: priorities, ...

  • sg-development | 1.9 kB 00:00

... Complete! # service globus-gridftp-server start Starting globus-gridftp-server: [ OK ] $ echo hi > /tmp/test.in $ globus-url-copy -udt file:///tmp/test.in \
 gsiftp://$HOSTNAME/tmp/test.out $ cat /tmp/test.out hi

slide-17
SLIDE 17

Testing OSG Software 10 April 2014

Acceptance Testing

Must pass before release Install software on bare VM

FermiCloud is awesome – thanks, guys!

Some integration, but not a site Kinds of testing

Check specific features or changes Check for regressions Look around for new issues Maybe do multi-machine testing

17

slide-18
SLIDE 18

Testing OSG Software 10 April 2014

Sample Defect #1

18

Brian L., testing an unrelated change:

# osg-koji setup Koji configuration file '...' already exists. Recreate it? [y/n] ? n Error downloading DOEGrids certs tarball (from https://pki1.doegrids.org/Other/doegrids.tar to /.../doegrids.tar): <urlopen error [Errno 110] Connection timed out> # echo $? 1 #

slide-19
SLIDE 19

Testing OSG Software 10 April 2014

Sample Defect #2

19

Neha, testing other changes to fetch-crl:

# grep dir /etc/fetch-crl.conf fodir = /etc/grid-security/certificates # service fetch-crl-boot start Running fetch-crl on boot: [ OK ] # echo $?

We emailed the fetch-crl developer about the failure to warn about an unused config option; still trying to convince them that this is a bug …

slide-20
SLIDE 20

Testing OSG Software 10 April 2014

ITB Testing

Install software on a test site

Same architecture as real site Relatively few worker/data nodes Takes some real production jobs

Lets us test in real environment Very few ITB sites today …

E.g.: UChicago (Suchandra), OU (Horst) Soon: Madison (OSG Software/Release)

20

slide-21
SLIDE 21

Testing OSG Software 10 April 2014

ITB Defect

21

Horst, testing Globus 5.2.5 updates:

# osg-control list Configured services:

  • c

condor-cron e edg-mkgridmap f fetch-crl-boot fetch-crl-cron g globus-gatekeeper globus-gridftp-server ...

slide-22
SLIDE 22

Testing OSG Software 10 April 2014

Testing is a Process

22

slide-23
SLIDE 23

Testing OSG Software 10 April 2014

Testing and Releases

  • 1. Build ⇒ Development
  • 2. Developer testing
  • 3. Development ⇒ Testing
  • 4. Acceptance and ITB testing
  • 5. Testing ⇒ Pre-Release
  • 6. One final run of osg-test
  • 7. Pre-Release ⇒ Production

23

slide-24
SLIDE 24

Testing OSG Software 10 April 2014

Future Work

  • sg-test

Expand coverage and expressiveness Add different modes: e.g., all vs. fast Better support runs on existing installs

Automated Runs

Update for OS, package, & repo changes Reports: “EL5 HTCondor jobs are broken”

Test Cases

Continue adding manual testing recipes Better tie tests to documentation

24

slide-25
SLIDE 25

Testing OSG Software 10 April 2014

Teams

OSG Software

Tim Cartwright Brian Lin Carl Edquist Edgar Fajardo Igor Sfiligoi Mat Selmeci

  • OSG Release

Tim Theisen Brian Lin Suchandra Thapa Xin Zhao

25

slide-26
SLIDE 26

Testing OSG Software 10 April 2014

Contacts

OSG Software & Release teams

  • sg-software@opensciencegrid.org

Tim Cartwright (Software mgr)

cat@cs.wisc.edu

Tim Theisen (Release mgr)

tim@cs.wisc.edu

OSG GOC (when defects slip by!)

goc@opensciencegrid.org

26