SMF Records/How to Extract Values from SMF Record Fields

How to Extract Values from SMF Record Fields

Summary

Processing the SMF record data with a REXX program in a batch environment consists roughly of the following steps:

Read the original SMF dataset (DUMPIN DD) with the IBM IFASMFDP program (see L11 or L12 )
Select the SMF record data you would like to process with a corresponding SYSIN DD Statement (see L11 or L12 for parameters)
Write formatted output into a new temporary dataset (DUMPOUT DD)
Convert the DUMPOUT dataset with the IBM IDCAMS REPRO Command into a readable format for REXX using a work dataset (WORKSMF DD)
Read all records from the WORKSMF dataset into a stem. variable in the memory of the REXX program
Process the required data and put the resulting comma-separated values file into another stem. variable for output
After finishing processing write this stem. variable into an output dataset (OUTDS DD) to store the resulting report

The Job Control Language

To process SMF record data with a REXX program in a batch environment there are a couple of prerequisites. First of all a formatted dump from the SMF dataset is needed, which can be produced with the IBM IFASMFDP program (see L11 or L12 ). Be sure to have at least read access to the SMF datasets in your installation. Unfortunately the resulting dataset from IFASMFDP has a RECFM(VBS) and is not readable directly in REXX. It has to be converted into another work dataset with RECFM(VB) and a suitable LRECL and BLKSIZE. This is done with the IBM IDCAMS REPRO command. Both the IFASMFDP and the REPRO commands are called from within the REXX script while the necessary DD statements are in the JCL itself.

//JOBNAME  JOB 'ACCNR',.,CLASS=L,MSGCLASS=J,
//          NOTIFY=&SYSUID,MSGLEVEL=(1,1),LINES=200000
//*
//*--------------------------------------------------------------------
//* ==> SET SMFOUT TO SMF WORK DATASET NAME
//* ==> SET REPOUT TO REPORT DATASET NAME
//*--------------------------------------------------------------------
//SET1    SET  SMFOUT=HLQ.XXX.READSMF.WRK
//SET2    SET  REPOUT=HLQ.XXX.SMF19.DASDREP.SYSTEM
//*--------------------------------------------------------------------
//* ==> DELETE WORK AND REPORT DATASETS
//*--------------------------------------------------------------------
//KILL01   EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
 DELETE HLQ.XXX.READSMF.WRK
 DELETE HLQ.XXX.SMF19.DASDREP.SYSTEM
 SET MAXCC=0
/*
//*--------------------------------------------------------------------
//* ==> START ISPF AND EXECUTE THE REXX
//*--------------------------------------------------------------------
//READSMF   EXEC  PGM=IKJEFT01,REGION=0M
//*
//ISPPROF  DD UNIT=SYSDA,SPACE=(TRK,(9,1,4)),
//  DCB=(LRECL=80,BLKSIZE=3120,RECFM=FB,DSORG=PO),
//  DISP=(NEW,PASS)
//*
//*--------------------------------------------------------------------
//ISPMLIB  DD DISP=SHR,DSN=SYS1.ISP.SISPMENU
//         DD DISP=SHR,DSN=SYS1.ISF.SISFMLIB
//ISPPLIB  DD DISP=SHR,DSN=SYS1.ISP.SISPPENU
//         DD DISP=SHR,DSN=SYS1.ISF.SISFPLIB
//ISPSLIB  DD DISP=SHR,DSN=SYS1.ISP.SISPSENU
//         DD DISP=SHR,DSN=SYS1.ISF.SISFSLIB
//ISPTLIB  DD DISP=SHR,DSN=SYS1.ISP.SISPTENU
//         DD DISP=SHR,DSN=SYS1.ISF.SISFTLIB
//ISPLOG   DD DUMMY
//* ---------------------------------------------------------------
//* ==> DUMPIN: SMF DATASET
//* ==> SYSIN:  RECORD TYPE, DAY AND TIME SELECTION
//* ---------------------------------------------------------------
//DUMPIN    DD    DISP=SHR,DSN=HLQ.SYSTEM.SMFDS.DJJMMDD
//DUMPOUT   DD    DISP=(,PASS),SPACE=(CYL,(800,200)),UNIT=SYSDA,
//          DSN=&&DP
//SYSIN     DD    *
INDD(DUMPIN,OPTIONS(DUMP))
OUTDD(DUMPOUT,TYPE(19))
DATE(2013334,2013334)
START(0000)
END(0100)
/*
//*--------------------------------------------------------------------
//SYSEXEC   DD    DISP=SHR,DSN=HLQ.XXX.ISPF.EXEC
//WORKSMF   DD    DSN=&SMFOUT,
//  DCB=(LRECL=32756,BLKSIZE=32760,RECFM=VB,DSORG=PS),
//  DISP=(NEW,CATLG),
//  UNIT=USEPLA,
//  SPACE=(CYL,(100,10),RLSE)
//OUTDS     DD    DSN=&REPOUT,
//  DCB=(LRECL=800,BLKSIZE=31200,RECFM=FB,DSORG=PS),
//  DISP=(NEW,CATLG),
//  UNIT=USEPLA,
//  SPACE=(CYL,(10,10),RLSE)
//SYSTSPRT  DD    SYSOUT=*
//SYSPRINT  DD    SYSOUT=*
//*--------------------------------------------------------------------
//SYSTSIN   DD    *
  ISPSTART CMD(SMF19)
/*

Chart 1: JCL for Data Selection and REXX Execution

In the example provided the first step KILL01 just deletes the work dataset and the resulting report dataset, because they are defined in the READSMF step as DISP=(NEW,CATLG).

In the READSMF step the program IKJEFT01 is executed to provide the necessary TSO/ISPF environment for calling the REXX program in batch.

All ISP* DD Statements belong to this environment.
The DD statements DUMPIN, DUMPOUT, SYSPRINT and SYSIN belong to the program IFASMFDP called at the beginning of the process_smf_data procedure in the REXX script.
- In particular the SYSIN DD statement provides the possibility of shrinking the data to only the really necessary amount for processing. In this example data is limited to only the SMF19 Records of day 334 in 2013 within the timeframe 00:00 AM to 01:00 AM. To avoid too long execution times and errors because of redundant data or insufficient storage, it is strongly recommended to reduce the amount of data processed at this point to the data really needed.
- The DUMPOUT DD statement defines only a temporary dataset with DSN=&&DP, because as mentioned before it is only needed for convertion into a work dataset with the REPRO Command called from within the REXX Script with "REPRO IFILE(DUMPOUT) OFILE(WORKSMF)".
- The SYSPRINT DD statement is used to show the number of read and written SMF records by the program IFASMFDP in the job output

The SYSEXEC DD statement defines the dataset which contains the REXX script.
The WORKSMF DD statement defines the source dataset for the REXX processing. All SMF records contained in this dataset are put into the stem variable SMF.. SMF. is then processed by the REXX code and the resulting values are written into another stem variable DASD_OUT. in comma-separated values format. In fact here they are actually semicolon-separated values because of the spreadsheet number format in some countries.
The OUTDS DD statement defines the output dataset where the stem variable DASD_OUT. is written at the end of the REXX processing with the command "Execio * Diskw OUTDS (finis stem dasd_out."
The SYSTSPRT DD statement defines the part in the job output where the REXX script can write out messages i.e. with the say command. This is extremely useful for both documentation and debugging purposes.
The SYSTSIN DD statement contains the ISPSTART CMD(SMF19) command to start the REXX script with the member name SMF19

The REXX Script

The REXX scripts in this book are all more or less built on the same structure:

Part 1 consists of a initialize procedure which only writes a border line into the job output and a process_smf_data procedure which is the main program of the REXX
Part 2 consists of a function to compute the field offsets relative to the beginning of the record and some functions to convert different data formats into processable formats for output.

Because the function part (Part 2) is the same in all following examples it is only shown once in this complete example. It is not necessary to repeat it in all following examples. Nevertheless the universal function part is necessary and should be copied at the end of each following REXX example where indicated.

/*REXX-----------------------------------------------------------------
 DASD Report from SMF19 Records
 ---------------------------------------------------------------------*/
call initialize
call process_smf_data
exit 0
/*---------------------------------------------------------------------
 ---------------------------------------------------------------------*/
initialize: procedure expose __.
  __. = ''
  __.0border = copies('-',70)
  return
/*---------------------------------------------------------------------
 ---------------------------------------------------------------------*/
process_smf_data: procedure expose __.
  "CALL 'SYS1.LINKLIB(IFASMFDP)'"
  "REPRO IFILE(DUMPOUT) OFILE(WORKSMF)"
  "EXECIO * DISKR  WORKSMF ( FINIS STEM SMF."
  drop DASD_OUT.
  count = 1
  j = 1
  say date() time() "DASD Report => read " smf.0 "input records"
  /*-------------------------------------------------------------------
   -------------------------------------------------------------------*/
  do i = 1 to smf.0
     line      = smf.i
    /*-----------------------------------------------------------------
     The standard SMF record header with subtypes . . .
     We're ignoring SMFxLEN (Offset 00) and SMFxSEG (Offset 02) since
     the IDCAMS PRINT utility doesn't include the record length and
     segment descriptor fields.
     ----------------------------------------------------------------*/
    SMFxFLG = binary_b(line,offset(4),1)
    rectype = binary_d(line,offset(5),1)  /* SMFxRTY: Record type    */
    say 'Line:' right(i,6,'0') 'Record Type:' rectype
    /*-----------------------------------------------------------------
     -----------------------------------------------------------------*/
    if (rectype = 19) then do
      SMFxTME = smftime(line,offset(6),4)
      SMFxDTE = smfjdate(line,offset(10),4)
      olddate = substr(smfxdte,3,2)||,         /* YYYY.DDD in YYDDD   */
                substr(smfxdte,6,3)
      Datum   = date('S',olddate,'J')          /* Date Conversion     */
      SMF19SID = ebcdic(line,offset(14),4)    /* SMF19SID: System ID  */
      SMF19VOL = ebcdic(line,offset(20),6)    /* SMF19VOL: VOLSER     */
      SMF19CUU = binary_x(line,offset(64),2)  /* SMF19CUU: UCB Number */
      SMF19SPC = binary_d(line,offset(52),2)  /*SMF19SPC: unalloc Cyls*/
      SMF19SPT = binary_d(line,offset(54),2)  /*SMF19SPT: unalloc Trks*/
      SMF19TRK = binary_d(line,offset(124),4) /* SMF19TRK: total Trks */
      Cyls_total = SMF19TRK / 15              /* 15 Trks per Cylinder */
      select
        when SMF19TRK = 16695 then dasd_model = '3390 Model 1'
        when SMF19TRK = 33390 then dasd_model = '3390 Model 2'
        when SMF19TRK = 50085 then dasd_model = '3390 Model 3'
        when SMF19TRK = 150255 then dasd_model = '3390 Model 9'
        when SMF19TRK = 491400 then dasd_model = '3390 Model 27'
        when SMF19TRK = 982800 then dasd_model = '3390 Model 54'
        otherwise dasd_model = 'other'
      end
                                             /* 849960 Bytes per Cyl  */
      GB_total  = (cyls_total * 849960) / (1024 * 1024 * 1024)
      GBt_total = translate(GB_total,,'.',',')     /*  Point to Comma */
      GB_free   = ((SMF19SPC * 15) + SMF19SPT) * (gb_total / SMF19TRK)
      GBt_free  = translate(GB_free,,'.',',')      /*  Point to Comma */
      GB_alloc  = gb_total - gb_free
      GBt_alloc = translate(GB_alloc,,'.',',')     /*  Point to Comma */
      j=j+1
      dasd_out.j = datum||';'||SMFxTME||';'||SMF19SID||';'||SMF19VOL||,
                   ';'||SMF19CUU||';'||SMF19SPC||';'||SMF19SPT||,
                   ';'||cyls_total||';'||dasd_model||';'||gbt_free||,
                   ';'||gbt_alloc||';'||gbt_total
    /*-----------------------------------------------------------------
     -----------------------------------------------------------------*/
      count = count + 1
    end
  end
  dasd_out.0 = count
  say dasd_out.0' Records written'
  dasd_out.1 = 'DATE;TIME;SYS;VOLSER;',
               'UCB;UNALLOC CYLS;UNALLOC TRKS;TRKS TOTAL;',
               'DASD Model;GB free;GB alloc;GB total'

  "Execio * Diskw OUTDS (finis stem dasd_out."

  return
/*---------------------------------------------------------------------
 Functions:
 ---------------------------------------------------------------------*/
/*---------------------------------------------------------------------
 Function: OFFSET
 Input: Field offset (decimal) per SMF Reference (SA22-7630, Chapter 14)
 Output: Input offset minus three
   To get the correct offset into the SMF input line, subtract three
   bytes. These three bytes consist of:
   2 bytes, to account for RDW not being present
   1 byte, because in Rexx, indices begin at position 1, not zero
 ---------------------------------------------------------------------*/
offset: procedure
  arg this_offset
  return (this_offset-3)
/*---------------------------------------------------------------------
 Function: Binary_d
 Returns : Decimal
 ---------------------------------------------------------------------*/
binary_d: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  this_field = substr($dumpline,$offset,$field_length)
  translated_field = x2d(c2x(this_field))
  return (translated_field)
/*---------------------------------------------------------------------
 Function: Binary4_d  --> for negative Values in 4 Byte binary fields
 Returns : Decimal
 ---------------------------------------------------------------------*/
binary4_d: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  this_field = substr($dumpline,$offset,$field_length)
  translated_field = x2d(c2x(this_field),8)
  return (translated_field)
/*---------------------------------------------------------------------
 Function: Binary_h
 Returns : Decimal
 ---------------------------------------------------------------------*/
binary_h: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  this_field = substr($dumpline,$offset,$field_length)
  translated_field = x2d(c2x(this_field))
  return (translated_field)
/*---------------------------------------------------------------------
 Function: Binary_x
 Returns : Hexadecimal
 ---------------------------------------------------------------------*/
binary_x: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  this_field = substr($dumpline,$offset,$field_length)
  translated_field = c2x(this_field)
  return (translated_field)
/*---------------------------------------------------------------------
 Function: Binary_b
 Returns : Binary
 ---------------------------------------------------------------------*/
binary_b: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  this_field = substr($dumpline,$offset,$field_length)
  translated_field = x2b(c2x(this_field))
  return (translated_field)
/*---------------------------------------------------------------------
 Function: Packed
 ---------------------------------------------------------------------*/
packed: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  translated_field = binary_x($dumpline,$offset,$field_length)
  return (translated_field)
/*---------------------------------------------------------------------
 Function: EBCDIC
 Returns:  EBCDIC
 ---------------------------------------------------------------------*/
ebcdic: procedure expose __.
  parse arg $dumpline,$offset,$field_length
  this_field = substr($dumpline,$offset,$field_length)
  return (this_field)
/*---------------------------------------------------------------------
 Function: Smftime
 Returns: hh:mm:ss
 ---------------------------------------------------------------------*/
smftime: procedure
  parse arg $dumpline,$offset,$field_length
  _time     = binary_d($dumpline,$offset,$field_length)
  hundreths = _time % 100
  hh        = hundreths % 3600
  hh        = RIGHT("0"||hh,2)
  mm        = (hundreths % 60) - (hh * 60)
  mm        = RIGHT("0"||mm,2)
  ss        = hundreths - (hh * 3600) - (mm * 60)
  ss        = RIGHT("0"||ss,2)
  this_time = hh||":"||mm||":"||ss
  return (this_time)
/*---------------------------------------------------------------------
 Function: Smfjdate
 Returns: Julian date yyyy.ddd
 Per SMF documentation, SMFxDTE is the date when the record was moved
 into the SMF buffer, in the form 0cyydddF where
   c   is 0 for 19xx and 1 for 20xx
   yy  is the current year (0-99)
   ddd is the current day (1-366)
   F   is the sign)
 ---------------------------------------------------------------------*/
smfjdate: procedure
  parse arg $dumpline,$offset,$field_length
  this_field = c2x(substr($dumpline,$offset,$field_length))
  parse value this_field with 1 . 2 c 3 yy 5 ddd 8 .
  if (c = 0) then
    yyyy = '19'||yy
  else
    yyyy = '20'||yy
  julian_date = yyyy||'.'||ddd
  return (julian_date)

Chart 2: REXX SMF19 to create a .csv file with the values of interest

Let's now go through the main procedure process_smf_data.

As described in the JCL part the program IFASMFDP is called to get a formatted dump and the REPRO command is used to convert the dump output into the correct dataset format.
"EXECIO * DISKR WORKSMF ( FINIS STEM SMF." read all records of the working dataset into memory
The drop DASD_OUT. just clears the output variable DASD_OUT.
The counter variables count and j are then initialized to a value of 1 and a header with date, time and the number of read SMF records is written into the job output (SYSTSPRT).
After this initial work processing a conditional loop for each record begins

At this point you should check the description of the SMF19 record mapping(see L11 or L12 ) to track the different offsets and length of the used record fields. In most cases we used the field names from this description.

Now for each record the record header and type is extracted and a line is written to the job output containing the line number and the record type
If the record type is 19 some directly usable values such as Time, Date, SYSID, Volser, UCB Number, the unallocated cylinders and tracks and the total tracks on the volume can be extracted using appropriate conversion functions
From the total tracks on the volume one can derive the total cylinders used with the following formula:

Cyls_total = SMF19TRK / 15

From the size of SMF19TRK it can also be concluded to which DASD Model the volume belongs. The variable dasd_model is filled in the appropriate select statement.
Knowing that a 3390 device contains 849960 bytes per cylinder the total gigabytes of the volume can be obtained with the formula:

GB_total = (Cyls_total * 849960) / (1024 * 1024 * 1024)

The following translate function in the REXX is used to mask the decimal point with a comma. REXX needs to compute with decimal points but the output dataset should contain a comma instead because of the spreadsheet number format in many countries.
The GB free and GB allocated of the volume can be derived from these formulas:

GB_free = ((SMF19SPC * 15) + SMF19SPT) * (GB_total / SMF19TRK)

GB_alloc = GB_total - GB_free

Here we finish the record processing, increment the counter j by 1, write all native and derived values into the stem. variable dasd_out.j and increment the counter count by 1.
After processing the last record the value of count is written to dasd_out.0 and the report header line is written to dasd_out.1.
With "Execio * Diskw OUTDS (finis stem dasd_out." the whole report is then written to the output dataset defined for OUTDS.

← Preface · SMF Type 19 Direct Access Volume →