Building the Easy Button: Automating SAS Program Batch Runs Nancy - - PowerPoint PPT Presentation

▶
building the easy button automating sas program batch runs
SMART_READER_LITE
LIVE PREVIEW

Building the Easy Button: Automating SAS Program Batch Runs Nancy - - PowerPoint PPT Presentation

Feb 13, 2024 240 likes •976 views

Building the Easy Button: Automating SAS Program Batch Runs Nancy Brucken inVentiv Health June 28, 2017 Agenda Motivation Driver programs Windows batch files Building a batch file Future considerations

slide-1
SLIDE 1

Building the Easy Button: Automating SAS Program Batch Runs

Nancy Brucken inVentiv Health

June 28, 2017

slide-2
SLIDE 2

2

Agenda

  • Motivation
  • Driver programs
  • Windows batch files
  • Building a batch file
  • Future considerations
  • Summary
slide-3
SLIDE 3

3

Motivation

slide-4
SLIDE 4

4

Running a SAS Program

slide-5
SLIDE 5

5

Running Multiple SAS Programs

Program 1 Program 2 Program 3

slide-6
SLIDE 6

6

Running Many SAS Programs

  • One person

− Open program 1 and run − Open program 2 and run − And so on…. − Takes forever!

  • Multiple people

− Each person runs multiple programs − Distributes the work − What if someone isn’t available? − What about dependencies?

slide-7
SLIDE 7

7

Problem Scope

  • Delivery for “medium-sized” clinical study

− 30 SDTM domains − 25 analysis datasets − 60 TLF programs

slide-8
SLIDE 8

8

Dependencies

  • One program relies on the output of another program
  • Example- pharmaceutical industry Analysis Dataset Model

(ADaM)

ADSL 1 record/subject Demographics Treatment arm Population flags Covariates Other analysis variables Partial ADSL Run ADSL, then run

  • ther dataset

programs

slide-9
SLIDE 9

9

How do I automate?

slide-10
SLIDE 10

10

Where’s my Easy button?

slide-11
SLIDE 11

11

Driver Programs

slide-12
SLIDE 12

12

Driver Program

  • SAS program that runs other SAS programs

%include ‘C:\mysas\adsl.sas’; %include ‘C:\mysas\adae.sas’; %include ‘C:\mysas\adlb.sas’; . . . . . %include ‘C:\mysas\advs.sas’;

slide-13
SLIDE 13

13

Creating a Driver Program

  • Manually type in all of the programs to run

Or

  • Get the computer to do it for you!
slide-14
SLIDE 14

14

Program to Build Driver Program

filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; file mybatch; put '%include ' file_name ';'; run;

slide-15
SLIDE 15

15

Program to Build Driver Program

filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; file mybatch; put '%include ' file_name ';'; run;

Pipe routes output of

  • perating system

command to SAS dataset

slide-16
SLIDE 16

16

Program to Build Driver Program

filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; file mybatch; put '%include ' file_name ';'; run;

/b limits output to just file names

slide-17
SLIDE 17

17

Program to Build Driver Program

filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; file mybatch; put '%include ' file_name ';'; run;

Read in list of file names

slide-18
SLIDE 18

18

Program to Build Driver Program

filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; file mybatch; put '%include ' file_name ';'; run;

Build %INCLUDE statements

slide-19
SLIDE 19

19

Resulting Program

%include adsl.sas ; %include adae.sas ; %include adlb.sas ; %include admh.sas ; %include advs.sas ;

slide-20
SLIDE 20

20

Advantages of Driver Programs

  • Easy to create
  • Easy to control dependencies

− Specify run order

  • If one program errors out, the entire job stops
slide-21
SLIDE 21

21

Disadvantages of Driver Programs

  • Programs run sequentially, so do not take advantage of multi-

server SAS Grid environments

  • Large batch jobs may take a long time to run

− Can you disconnect your PC and still leave the job running?

  • Datasets, macro variables and options created in one program

will carry through into subsequent programs if not explicitly deleted or cleared

  • If one program errors out, the entire job stops
slide-22
SLIDE 22

22

Batch Files

slide-23
SLIDE 23

23

Batch Files

  • Files containing operating system commands
  • Supported by many operating systems

− Unix − MVS − Windows

  • Operating system-specific
slide-24
SLIDE 24

24

Typical Batch File Structure

Initialization Commands

slide-25
SLIDE 25

25

Initialization Section

  • Specify folder paths
  • Direct LOG and LST files
  • Specify location of SAS config file
  • Specify location of SAS executable

Place at the top of the batch file so they are only referenced once!

slide-26
SLIDE 26

26

Sample Initialization Section

@echo off set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“

  • CONFIG

"\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg"

slide-27
SLIDE 27

27

Sample Initialization Section

set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“

  • CONFIG

"\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg"

Path to program location

slide-28
SLIDE 28

28

Sample Initialization Section

set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“

  • CONFIG

"\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg"

Path to log file location

slide-29
SLIDE 29

29

Sample Initialization Section

set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“

  • CONFIG

"\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg"

Path to LST file location

slide-30
SLIDE 30

30

Sample Initialization Section

set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“

  • CONFIG

"\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg"

Location of SAS executable

slide-31
SLIDE 31

31

Sample Initialization Section

set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“

  • CONFIG

"\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\s asv92_i3spctx.cfg"

Location of SAS config file

slide-32
SLIDE 32

32

Command Section

  • Commands to run each program
  • Includes environment variables from initialization section
slide-33
SLIDE 33

33

Sample Command Section

@echo on %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst%

slide-34
SLIDE 34

34

Sample Command Section

%sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst%

Command to run SAS, including location of config file

slide-35
SLIDE 35

35

Sample Command Section

%sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst%

Path to program location

slide-36
SLIDE 36

36

Sample Command Section

%sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst%

Program name

slide-37
SLIDE 37

37

Sample Command Section

%sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst%

Log file location

slide-38
SLIDE 38

38

Sample Command Section

%sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst%

LST file location

slide-39
SLIDE 39

39

Advantages of Batch Files

  • Each program runs independently
  • Datasets, options and macro variables set in one program do

not carry over into subsequent programs

  • If one program errors out, the remaining programs will still run
  • Under many environments, you can start/schedule a batch job

and log off without having to wait until it finishes

  • Easy to control dependencies
slide-40
SLIDE 40

40

Disadvantages of Batch Files

  • Run sequentially

− Do not take advantages of multi-core Grid servers

  • If one program errors out, the remaining programs will still run
  • May be more difficult to set up than driver programs

− Need to identify locations of SAS executable and config files

  • Command syntax varies across operating systems
slide-41
SLIDE 41

41

SAS Grid Batch Files

  • Take advantage of SAS Grid multi-core architecture

− Parallel processing

  • Still some control over dependencies
slide-42
SLIDE 42

42

Sample Initialization Section

@echo off set pth=\\prod-server\client\product\project\Stat\programs\tables\ set plog=-log \\prod- server\client\product\projec\Stat\Programs\Tables\Log set plst=-print \\prod- server\client\product\projec\Stat\Programs\Tables\Lst\ set gridcfg=-gridconfig “\\prod- server\SAS\BatchFile\sasgsub94.cfg" set SAS= C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts "'%plog% %plst%'" -gridsubmitpgm

slide-43
SLIDE 43

43

Sample Initialization Section

@echo off set pth=\\prod- server\client\product\project\Stat\programs\tables\ set plog=-log \\prod- server\client\product\projec\Stat\Programs\Tables\Log set plst=-print \\prod- server\client\product\projec\Stat\Programs\Tables\Lst\ set gridcfg=-gridconfig “\\prod- server\SAS\BatchFile\sasgsub94.cfg" set SAS= C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts "'%plog% %plst%'" -gridsubmitpgm

Same as before, though have to specify UNC path, as drive letters not available to Grid server

slide-44
SLIDE 44

44

Sample Initialization Section

@echo off set pth=\\prod-server\client\product\project\Stat\programs\tables\ set plog=-log \\prod- server\client\product\projec\Stat\Programs\Tables\Log set plst=-print \\prod- server\client\product\projec\Stat\Programs\Tables\Lst\ set gridcfg=-gridconfig “\\prod- server\SAS\BatchFile\sasgsub94.cfg" set SAS= C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts "'%plog% %plst%'" -gridsubmitpgm

Location of SAS Grid GSUB config file for batch processing

slide-45
SLIDE 45

45

Sample Initialization Section

@echo off set pth=\\prod-server\client\product\project\Stat\programs\tables\ set plog=-log \\prod- server\client\product\projec\Stat\Programs\Tables\Log set plst=-print \\prod- server\client\product\projec\Stat\Programs\Tables\Lst\ set gridcfg=-gridconfig “\\prod- server\SAS\BatchFile\sasgsub94.cfg" set SAS= C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts "'%plog% %plst%'" -gridsubmitpgm

Command to run SAS program on Grid

slide-46
SLIDE 46

46

Sample Command Section

@echo on %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas" %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

slide-47
SLIDE 47

47

Sample Command Section

@echo on %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas" %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

Command to run SAS, including locations of log and LST files

slide-48
SLIDE 48

48

Sample Command Section

@echo on %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas" %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

Path to program location

slide-49
SLIDE 49

49

Sample Command Section

@echo on %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas" %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

Program name

slide-50
SLIDE 50

50

Problem

@echo on %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas" %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

  • Batch file submits everything at once to SAS Grid servers
  • What about dependencies?

− DM has to run first, followed by SV − Then everything else can run

slide-51
SLIDE 51

51

  • GRIDWAIT option
  • Add to the end of a program call
  • Forces Grid server to wait to execute the next program until that

program is finished

slide-52
SLIDE 52

52

  • GRIDWAIT example

@echo on %sas% "%pth%dm.sas“ -gridwait %sas% "%pth%sv.sas“ -gridwait %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

slide-53
SLIDE 53

53

Building a Batch File

slide-54
SLIDE 54

54

Sample Initialization Section

@echo off set pth=\\prod-server\client\product\project\Stat\programs\tables\ set plog=-log \\prod- server\client\product\projec\Stat\Programs\Tables\Log set plst=-print \\prod- server\client\product\projec\Stat\Programs\Tables\Lst\ set gridcfg=-gridconfig “\\prod-server\SAS\BatchFile\sasgsub94.cfg" set SAS= C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts "'%plog% %plst%'" -gridsubmitpgm Executes once, at the start of the program

slide-55
SLIDE 55

55

Sample Command Section

@echo on %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas" %sas% "%pth%ae.sas" %sas% "%pth%cm.sas" Generate a separate line for each program in the batch file

slide-56
SLIDE 56

56

Get Directory Listing

filename dirfile pipe "dir \\prod- isilon\SASDataHome\SASProduction\Client\Project\Stat\Programs\ SDTM\*.sas /on /n"; Produces the following: 05/16/2017 09:48 AM 2,898 AE.sas 05/16/2017 09:48 AM 2,399 CE.sas Etc. /on = sort files in alphabetical order /n = generates 5-column directory listing format

slide-57
SLIDE 57

57

Set Batch File Location

filename batchout "\\prod- isilon\SASDataHome\SASProduction\Client\Project\Stat\Programs\ utl\batch_files\run_sdtm.bat";

slide-58
SLIDE 58

58

Build File List

data check1; infile dirfile truncover; input fileinfo $200.; fileinfo = compbl(fileinfo); if substr(fileinfo, 3, 1)='/' then filename = cats('%sas% "%pth%', scan(fileinfo, 5, ' '), '"'); else delete; run; Remove extra blanks, extract filename from string, preface with ‘%sas% “%pth’, and add “ at the end

slide-59
SLIDE 59

59

Generate batch file

data _null_; set check1; file batchout; if _n_=1 then do; put "REM: A BATCH PROGRAM TO RUN SDTM PROGRAMS"; put / '@echo off'; put / "set pth=\\prod- server\SASDataHome\SASProduction\Client\Project\Stat\Programs\S DTM\";

slide-60
SLIDE 60

60

Generate Batch File (cont.)

put "set plog=-log \\prod- server\SASDataHome\SASProduction\Client\Project\Stat\Program s\SDTM\log"; put "set plst=-print \\prod- server\SASDataHome\SASProduction\Client\Project\Stat\Program s\SDTM\lst\"; put 'SET gridcfg=-gridconfig “\\prod- server\SAS\BatchFile\sasgsub94.cfg"';

slide-61
SLIDE 61

61

Generate Batch File (cont.)

*** Note that the SET SAS= statement generated is not correct- need to fix the quotes before PLOG after batch file is generated; put / 'SET SAS=C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts' ' %plog% %plst%''" -gridsubmitpgm'; put / '@echo on'; end; /* Initialization Section */ else put filename; run;

slide-62
SLIDE 62

62

Sort Order / Dependencies

Program yields initialization section plus these commands: @echo on %sas% "%pth%ae.sas" %sas% "%pth%cm.sas" %sas% "%pth%dm.sas“ %sas% "%pth%sv.sas"

slide-63
SLIDE 63

63

Sort Order / Dependencies

Reorder programs and add –gridwait options @echo on %sas% "%pth%dm.sas“ -gridwait %sas% "%pth%sv.sas“ -gridwait %sas% "%pth%ae.sas" %sas% "%pth%cm.sas"

slide-64
SLIDE 64

64

Executing Batch Files

  • Navigate to batch file in Windows Explorer and double-click on

the filename to launch

  • .bat extension is automatically recognized as Windows batch

file

slide-65
SLIDE 65

65

Missing Logs/Output?

  • Large batch jobs can overwhelm SAS Grid Platform Load Share

Facility (LSF) job scheduler

− Insert sleep commands periodically in batch job to slow down the processing − Syntax: sleep <time in seconds> − sleep 20 has been sufficient for us

slide-66
SLIDE 66

66

Future Plans

slide-67
SLIDE 67

67

Future Plans

  • Fix quote problem in batch file generation program
  • Investigate ways to branch batch jobs and handle more

complex dependencies

− Example: Dataset containing baseline values from multiple other datasets

slide-68
SLIDE 68

68

Summary

slide-69
SLIDE 69

69

Summary

  • Batch files are extremely helpful for automating the running of

multiple programs

  • Ensure logs are written to the correct location
  • Ensure programs are run in the correct order
  • True batch files run completely independently
  • Launch or schedule a batch file and….
slide-70
SLIDE 70

70

Find your Easy Button!

slide-71
SLIDE 71

71

Contact Information

Nancy Brucken inVentiv Health Nancy.Brucken@inventivhealth.com 734-887-0255

slide-72
SLIDE 72

72

Questions?