C-Kermit Case Study #22

[ Previous ] [ Next ] [ Index ] [ C-Kermit Home ] [ Kermit Home ]

Article: 12223 of comp.protocols.kermit.misc
From: fdc@watsun.cc.columbia.edu (Frank da Cruz)
Newsgroups: comp.protocols.kermit.misc
Subject: Case Study #23: Modem Scripting Tutorial
Date:Fri Feb 23 20:02:02 EST 2001
Organization: Columbia University

It's been a while since scripting tutorials have been posted here, but with C-Kermit 8.0 nearing release, with all its new scripting features, now seems like a good time to resurrect the practice.

Does your organization have a big modem pool? Of course (like any other big university) ours has has a pool of many hundreds of lines. As you know, phone lines aren't cheap so the annual cost of running a big modem pool is painful. Yet, even in this age of DSL and cable modems, the demand for modems never stops increasing, and therefore so do the complaints about busy signals, and the pressure to increase the capacity, and the counterpressure to contain contain costs. (Eventually some day every room will have an Internet jack and modems will have the same historical status as card punches and Teletypes, but that's another discussion -- for now DSL in the home is often not available, usually has a long wait when it is, and it's expensive).

One consequence of all this is the need to monitor and test the modem pool to make sure it's working at its best -- no broken or poorly-performing lines or servers -- plus the degree to which it is available, i.e. the answer-to-call ratio. There's a script for doing this in the Kermit script library, but it's undergone quite a few refinements in recent months as we reconfigure our modem pool, change telephone service providers, etc, and many of the refinements nicely illustrate new features of C-Kermit 8.0 (and the forthcoming next release of Kermit 95).

  http://www.columbia.edu/kermit/ck80.html

The script is given a list of phone numbers and a list of terminal-server names. Each list can have one or more items. Obviously the script also needs to know the details of the calling computer: serial device and/or modem name, speed, etc. The script prompts for password (because it's a bad idea to store passwords in files) and runs in a loop, performing the following actions:

1. Calls the next phone number from the list (which is circular). If the call is not answered or the line is busy, the appropriate error is logged and this step is repeated.

2. Gets the terminal server prompt, gives any necessary configuration commands to the terminal server, and then requests a Telnet connection to a host computer. If this fails, an error is logged and the script hangs up and goes back to step 1. If the terminal server announces its name and the line number for the call, these are captured.

3. Logs in to the host computer.

4. Starts a Kermit server on the host, uploads a 100K precompressed file and downloads the same file.

5. Says BYE to the Kermit server, closing the connection.

6. Gets statistics about the call from the modem, including block-error and retrain counts and final session modulation speeds.

7. Writes a log record showing the date and time, phone number, terminal server name and line, initial connection speed. final input and output modulation speeds, block-error and retrain counts, and the upload and download throughput in characters per second.

Here is bit of a log:

Date  Time  Number    Line       Speed  Speed-After  Blers Retr Upload  Dnload
----------  -------   --------   -----  -----------  ----- ---- ------  ------
02/22 0905  5551234   ccts5:080  49333  50666/28800      0    0   2951    5184
02/22 0907  5551235   ccts3:002  49333  49333/31200      1    0   3255    5143
02/22 0909  5551236   ccts2:096  49333  49333/31200      6    0   3243    5031
02/22 0911  5551237   ccts5:051  49333  50666/28800      1    0   3002    5220
02/22 0913  5551238   ccts5:069  49333  50666/28800      0    0   3008    5234

Naturally, every imaginable kind of error is caught and logged:

02/22 0917  5551240   ccts3:076  49333  49333/31200      5    0   3217    5068
02/22 0919  5551241   ccts5:112  49333      UNKNOWN    UNK  UNK   3010    5215
02/22 1055  5551241   ccts4:UNK  31200      UNKNOWN    UNK  UNK FAILED  FAILED
02/22 1119  5551243  FAILED: "BUSY"
02/22 1509  5551244  FAILED: "NO CARRIER"
02/22 1509  5551245   ccts5:183  49333  50666/31200      1    0   2925    5201

The script can be halted between calls by the operator from the keyboard, but you don't need to halt it to get at the log, which is opened and closed around each write so it can be examined, copied, uploaded, or whatever on platforms that might not allow shared access to open files.

The log format is such that headings and so forth can be separated out (in UNIX) with:

  grep ^[0-9] modem.log > entries

leaving one line per call:

  wc -l entries                   # number of calls
  grep -v FAILED entries | wc -l  # number that were answered

The ratio of these two numbers gives the answer-to-call ration. (C-Kermit 8.0 also has its own internal GREP and line-counting commands in case you're not using UNIX.)

A log of successful calls can be sorted in various ways because the columns are fixed. For example, sorting on the "Line" column puts multiple calls to the same number together. The numeric data can be analyzed statistically too. For example, you can get the minimum, maximum, and mean upload or download speed and you can correlate it with telephone or line number to see (for example) if certain lines, terminal servers, or phone numbers perform better or worse than others over many calls (this would also jump out at you when eyeballing a sorted log).

The updated modem-test script is here:

  http://www.columbia.edu/kermit/ftp/scripts/ckermit/modemtest2

Let's look at a few pieces of it:

  dcl \&n[] = 5551234 5551235 5551236 5551237 5551238 5551239 5551240 -
              5551241 5551242 5551243 5551244 5551245 5551246 5551247

  dcl \&p[] = ccts1 ccts2 ccts3 ccts4 ccts5 ccts6

This declares an array \&n[] and loads it with 16 phone numbers, and another array \&p[], loading it with six terminal-server names. To change the script to use different lists of numbers and names requires changing only these two statements -- everything else adjusts itself automatically.

The terminal-server prompt turns out to be the terminal-server name followed by a ">" character, such as "ccts3>", so we construct a parallel array, \&q[], of prompts:

  dcl \&q[\fdim(&p)]
  for \%i 1 \fdim(&p) 1 {
    .\&q[\%i] := \&p[\%i]>
  }

The size of this array must be included in its declaration, since we are not initializing it in the declaration. But unlike in other languages (such as C), the size need not be a constant. In this case we say that the size is the "dimension" (size) of the array \&p[]. Then we loop through the new array, making the appropriate assignments.

Here's where we dial, illustrating (a) how we dial a number from the array; (b) how we handle failures based on Kermit's \v(dialstatus) variable (the \v(dialresult) variable is the modem's call-result message string):

  dial \&n[\%i]  ; \%i is the loop variable.
  if fail {
      switch \v(dialstatus) {
        :8, logrecord {FAILED: Timed out}, break
        :9, logrecord {FAILED: User canceled}, stop
        :10, logrecord {FAILED: Modem not ready}, break
        :default, logrecord {FAILED: "\v(dialresult)"}, break
      }
      continue
  }

The terminal servers in question are Ciscos. Upon connection you can either send PPP negotiation data or else request a command prompt by sending a carriage return. The script sends a carriage return. The server prints a screenful of text and then the prompt:

  ccts4 line 17

  Welcome to blah blah blah
  blah blah blah
  blah blah blah blah

  ccts4>

The text might or might not contain a line like this:

  ccts4 line 17

If it does, we need to pick up the line (port) number, but if it doesn't, we don't want to get stuck waiting for it. So instead, we scoop up everything up to and including the prompt:

  clear input                  ; Clear the INPUT buffer
  .\%x := 0                    ; (explained below)
  for \%j 1 10 1 {             ; Try 10 times to get terminal server herald
      output \13               ; Send Carriage Return (ASCII 13)
      minput 10 \fjoin(&q,,2)  ; Look for any of the prompts
      if success break         ; Quit this loop if we get one
  }
  if > \%j 10 {                ; This means we didn't get one.
      logrecord {FAILED: No terminal server prompt}
      continue                 ; Go make another call.
  }

The magic command in this section is:

  minput 10 \fjoin(&q,,2)

The function \fjoin() is new to C-Kermit 8.0. It replaces itself with a text string consisting of all the elements of the given array, separated (in this case) by spaces (the "2" is a formatting code). Therefore, this statement is equivalent to:

  minput 10 ccts1> ccts2> ccts3> ccts4> ccts5> ccts6>

This is not simply a notational convenience. It's one of the features that allows our script to be table-driven, in the sense that we only have to change the array initialization to make the script use different server names or phone numbers, and even different numbers of them, without having to change other lines in the script, such as the MINPUT that looks for all the possible prompts.

If the MINPUT command succeeded, we have the entire output of the terminal server in our \v(input) variable. Now we can scan it for the line-number text:

  .tmp := \v(input)  ; Copy the \v(input) variable
  .\%x = \v(minput)  ; The item that MINPUT matched (1, 2, 3, ...)

The next statement searches for the line-number message. \&p[\%x] is the name of this server, corresponding to the prompt. So, for example, if the prompt was "ccts4>", the server name is "ccts4" and this statement searches the text we just read for the string "ccts4 line ".

  .\%y = \findex({\&p[\%x] line },\m(tmp))

Next, we initialize the tsport (terminal-server port) variable to "UNK" in the desired line was not found:

  .tsport = UNK

Then if the "ccts4 line " string was found, we use another function, \fword(), to get the third "word" from the string that starts with "ccts4 line ". This is the line number. Then we left-pad (\flpad()) it with 0's so it is exactly 3 digits:

  if > \%y 0 {
    .tsport := \flpad(\fword(\s(tmp[\%y]),3),3,0)
  }

Now we know the server name and the port number:

  .tsport := \&p[\%x]:\m(tsport)

which, continuing with our example, becomes:

  ccts4:017

The rest is fairly straightforward: the regular OUTPUT / INPUT / IF FAIL sequence familiar to all Kermit script writers, to accomplish login, starting the Kermit program on the far end, transferring files, and logging out. There is one new feature in the data-transfer phase that puts some status information in the file-transfer display:

  SET TRANSFER MESSAGE text

in which the text can contain variables. In this case it's used to show the call sequence number, the phone number, the terminal server name and port, and the time the transfer started (so you can easily tell if it's stuck):

  set xfer message \m(seq): \v(dialnumber) (\m(tsport)) \v(time)

e.g.

  Last Message: 712: 5554321 (ccts3:020) 16:02:56

After logout comes another interesting part, where we query the modem for statistics and capture them. Of course the details depend on the modem. For US Robotics modems we "output ATI6\13" and then parse the results as follows:

  set flag off
  while not flag {
     minput 10  Blers  Retrains  Speed  OK
     switch \v(minput) {
       :1, clear input, input 2 \10, .blers := \fword(\v(input),1), break
       :2, clear input, input 2 \10, .retrains := \fword(\v(input),5), break
       :3, clear input, input 2 \10, .ospeed := \fword(\v(input),1,,/), break
       :4, set flag on, break
     }
  }

In other words, we look for any of "Blers", "Retrains", "Speed", and "OK" (which terminates the report). If we get OK, we're done. Otherwise we read the rest of the line ("input 2 \10", i.e. read up to a linefeed) and then use \fword() to extract the desired word.

The script requires about 2 minutes per call, so if all goes well it collects about 720 records per day, using a 100K test file and calling a 56K modem pool. It's best to collect several samples per server port, which could take a day or two or more, depending on the size of your pool and its hunt strategy.

Finally, obtaining statistics from the results is easier than ever using C-Kermit 8.0's new floating-point arithmetic and S-Expression syntax:

  http://www.columbia.edu/kermit/ftp/scripts/ckermit/stats

All we need to do is filter out the error records, as shown at the top, and then use UNIX 'cut' to isolate the desired two columns of numbers. If the servername:port column is one of them, we can simply filter out the non-numeric characters, so (for example) ccts4:017 becomes 4017. Records that have ccts4:UNK are discarded completely since we don't know the port number.

Here we get statistics for upload and download speed versus server:port:

  grep ^[0-9] modem.log | grep -v FAILED > tmp

  cut -c21-31,65-70 < tmp > up
  cut -c21-31,72-78 < tmp > dn

  tr -d "[a-z:]" < up | grep -v UNK > up2
  tr -d "[a-z:]" < dn | grep -v UNK > dn2

  stats up2
  stats dn2

The last one ("stats dn2") prints something like this:

  Points: 382
                         X         Y
  Minimum:           2001.00   1875.00
  Maximum:           5192.00   5282.00
  Mean:              3642.26   5016.66
  Variance:       1811436.52 157155.99
  Std Deviation:     1345.90    396.43

  Correlation coefficient:        0.12

- Frank

[ Top ] [ Previous ] [ Next ] [ Index ] [ C-Kermit Home ] [ Kermit Home ]