My primary responsibilities for the Air Force’s Global Positioning System (GPS)
were maintaining the legacy, mainframe computer systems.
I was responsible for installing all the operating system maintenance and releases.
I did this by first installing the changes on an offline system, and then migrating
the changes to the support system for a test period and then later to the operational
systems. Actual implementation of these upgrades usually took less than thirty minutes
with minimal operational impact, since I would take an extensive amount of time preparing
for the implementation.
I was responsible for disk storage management, organizing and allocating storage space
for the various user groups that used GPS. Back in the early days of GPS, this included
spreading out “high-use” data over separate storage units insuring that there was no
degradation of access. As time passed, significant improvements to data access speeds
meant that people do not even consider it a factor any more. However, in most systems
adjusting data storage can still improve efficiency.
I was responsible for evaluating several third party products for use in GPS.
These include, but are not limited to, Innovation’s Fast Dump/Restore (FDR),
Computer Associates’ Top Secret (CATS), Tape Library Management System (TLMS),
XCOM, and Auto-Operator. These products remained in effect until the Air Force
deactivated the legacy GPS system in early 2008.
Shortly after I arrived at Falcon AFS (now known as Schriever AFB), I was tasked
to evaluated several system security applications and recommend the one that
best solve the security concerns of the Air Force for the legacy Global
Positioning System (GPS). The Air Force was interested in implementing a closed
security system and, at the time, the government considered IBM’s RACF an open
security system. I eventually recommended a product from Computer Associates
called “Top Secret” (that is the product name, not a security classification).
To avoid any security issues when discussing this product, we used the acronym
CATS, since GPS was only a secret system. The Air Force accepted my
recommendation, ordered the product, and instructed me to implement CATS.
After I implemented CATS, The Air Force assigned the additional duty of Security
Administrator to me, and tasked me to define over 25 user groups and create over
250 initial user accounts. From that day on, I was the GPS Primary Security
Administrator.
I created a process to move the security audit logs to magnetic tape, organizing
them by year and month. This process was automatic and triggered whenever one of
the alternating log files reached full capacity. I also wrote the user interface
to retrieve the audit data that was quick and easy to use.
I was the driving force behind an effort to incorporate the GPS application into
this new security system. Previously, the GPS application had its own internal
security system that was less than adequate, and required that I manually enter
user passwords that would match their system passwords. The DAA accreditation
process issued a waver allowing this practice, until I configured the security
system to accept information about GPS specifically, and worked directly with
the application maintainers to modify the GPS application to access the system
security application. Once we implemented these changes, DAA was able to remove
the waver and declare GPS fully accredited.
In the early days of GPS, a single test environment shared data with the
operational environment. This created a security nightmare, trying to insure
that the test environment did not inadvertently, or intentionally, introduce
test data into the operational environment. I redesigned the test environment to
isolate it from the operational environment and updated it to multiple test
environments. Each test environment had its own database that the testers could
synchronize periodically with the operational environment.
More recently, the Air Force grew concerned with the number of people listed on
a specific access control roster and asked our manager if we could do anything
to reduce this number. Our manager called together the team to discuss issues
and possible solutions, and one common issue was that we could not reduce the
list, as long as the testers had access to their daily job. I finally suggested
taking away the testers access, and the team got very quite. Our manager asked
“Then how are the testers supposed to do their job?” I replied, “I’ll write a
user interface.” Our manager quickly concluded the meeting by saying, “Let me
know when you’re done.” Two months later, I release the first version of the
Test Package Submitter.
Although I never actually held the title of Database Administrator, I was
responsible for initializing the databases for new operational releases for the
test environments. The database for the legacy GPS system was not a relational
database like Oracle or even Microsoft Access. Each database was a collection of
over 2000 Virtual Storage Access Method (VSAM) datasets. Each new database would
require quite a bit of storage space and since I was the Direct Access Storage
Device (DASD, hard drives) manager, it was my job to create the new databases.
Since I was also the Security Administrator, I was able to define the security
restraints for the new database at the same time. With several years experience,
I was able to spread out the high-use dataset on several DASD units, allocate,
and populate all these files and have the new database ready for the
configuration managers in about half a day.
I was also skilled at manipulating the raw data inside these files using REXX
scripts. I designed several of these scripts, each designed to locate the data
of interest, convert it from internal format into human readable format, and
display it to the user. When the user determined they wanted to modify the data,
the script would convert the human input format back into internal format, and
replace the existing data. The best example of this is the KSS software tool I
developed.
I wrote the first personnel file for the 1002 Space Systems Support Squadron,
in 1985/6, using dBase II, and later converted it to dBase III.
I wrote a Weekly Activity Report (WAR) application for the 1002 Space Systems
Support Squadron in dBase III that replace the process they were doing in Word.
Their old process used to take months to percolate to upper management. In that
amount of time, the information grew so old that it was no longer important. My
new database version allowed supervisors to automatically merge subordinate
inputs, and pass it to their superiors in minutes. My application reduced the
entire process to less than half a day. Managers now had real time data at their
fingertips.
On a personal level, I have created a database family tree web site that I am
still developing. It currently has over three hundred families. Development has
slowed down a bit since I am now more interested in finding a new challenging
career, but I am still working on a login routine that will allow my family
members to edit their own data. Access to my family tree database is currently
by invitation only, and is password protected to reduce the risk of identity
theft.
Many years ago, I also exploited the capabilities of dBase. I wrote an
application that I planned to market, so to prevent a user from copying it to
multiple systems; I devised a method of capturing the user’s system information,
encrypting it, and storing it in unused areas of the database. If the
application detected that it was accessing an illegal copy of the database on an
unregistered system, it simple reverted to its unregistered version.
I also designed a true binary field for dBase databases, that would store and
access logical information at the bit level. The dBase logical field used an
entire byte to store logical data, and at the time (late 70’s), this seemed like
a waste to me. 5¼-inch floppies were the main external storage media, with a max
capacity of 180K or 360K. I figured out that I could store eight true binary
fields in a single byte and save seven bytes per record. Since my database had
32 logical fields, I would save 28 bytes per record. Since my database had over
6000 records, I would save over 164K, nearly a full floppy. I also developed all
the code to access the true binary fields exactly the same way users accessed
the dBase logical fields and I developed the database conversion routines to
switch from logical to true binary fields. I contacted Aston-Tate to propose my
changes, but Computer Associates had recently bought them out. When I contacted
Computer Associates, they were not interested in my modifications.
In addition to creating the new databases for the test packages, I was also
responsible for setting up the initial test package environments. I would always
start with an exact copy of the operational environment and integrate all the
modifications for the delivered release, getting the environment ready for the
configuration managers. This insured that the configuration managers could
validate the tests later.
My knowledge of configuration management helped me design and develop the Test
Package Submitter tool. I designed it to enforce configuration management
control over the various test package environments. Although testers could make
temporary configuration changes, the default configuration was completely under
CM control.
System Initialization and Shutdown routines
When the Air Force first implemented automation software on all the
systems at Schriever, they tasked a small team of Software Engineers to
consolidate and standardize several core routines that all rooms would use. I
was not a member of that team, and in fact, my supervisor specifically
instructed me not to work on the core routines. Therefore, for the next three
months I worked on other automation routines, learning as much as I could about
the new software. When the SE team delivered their core routines, I started by
testing their system initialization routine on our support system. It failed
miserably on several attempts. While I waited for them to return to fix it, I
decided to look at their code and try to figure out why it was not working. What
I found was a mess. The routine had over 1800 lines of nested IF-THEN-ELSE logic
up to 12 layers deep. I tried to find the section for the GPS systems, but I
could not. I finally pushed their routine aside and made a list of the tasks
that the initialization routine should start. I then added the commands that
would start those tasks. As I reviewed my list, I realized that I could write a
routine to read my list in a simple loop. In less than an hour, I had my routine
and it was less than 100 lines of code. I created the task lists for the other
GPS systems and implemented my routine. When the SE team returned the next day,
I offered to let them have my routine for the other rooms, but they refused.
About a month later they returned and asked if they could use my routine in the
other rooms. They were spending a lot of time fixing and maintaining their
routine in the other rooms. My routine has not required any maintenance since I
implemented it.
Their shutdown routine was just as bad so I modified a copy of my initialization
routine to create a new routine to bring down each system systematically. It
also has not required any maintenance since implementation.
Mission Package Switchovers
The legacy GPS mission package is an application that runs on one of
the IBM mainframe systems at Schriever AFB. In the event of a failure of part of
that system, the Air Force has a process called a switchover that transfers the
mission application to the passive system. This process had over 60 steps, which
the computer operators and Air Force crew had to execute and coordinate
manually. Even when they executed every step correctly, the entire process took
over 20 minutes. When there were mistakes or an operator was being very careful
not to make a mistake, it easily extended the time to over 30 minutes.
I decided to automate this process, which at first seemed simple enough. I soon
discovered that I had to design many new techniques to coordinate automated
processes across multiple systems. This was back in the late 1980’s and we did not
have TCPIP linking our systems together. Our systems had a connection known as a
loosely coupled connection. These systems shared disk drives and JES2 resources,
only. When I completed my automated process, the operators simply enter the
command to start the process, answered a couple questions, and approximately 8
minutes later the mission switchover was complete.
Transitions to the remote backup site
I used the new techniques I designed for the switchover process and
incorporated them into the transition process. The process was similar, but the
connection with the remote system was even less than the loosely coupled systems
sitting next to each other. All I had were the SNA communication lines that we
used for file transfers. Well since one of the first steps of the transition
process was to turn off files transfers, I had the full bandwidth at my
disposal. The only part of the transition process I could not automate was the
remote site, communication line, switch, which happened out in the world
somewhere. Actually, it happened at two separate locations and I never got the
opportunity to visit them. I begged my supervisor to send me to their locations
and fix their GPS communication processes, but he simply reminded me that it was
not my job.
The Test Package Submitter tool
If you have read my software tools section, you may wonder why I am also
listing this tool under automation. It is because this tool depends on the
automation software. Almost half of the tool resides in the automation software
realm. This was done specifically for security reasons, since the automation
software is a system level application accessible only by system programmers and
engineers. TPS users are not even aware that the automation software does most
of the work for them. The automation software actually generates their JCL and
submits it for them. The user interface of TPS simply generates a parameter list
and passes that list to the automation software. The simple act of passing this
parameter list, is the security barrier between users who do not require access
to sensitive data and the sensitive data.
JCG - The Job Card Generator tool
Any one who has worked on mainframe computer systems has probably had
to deal with Job Control Language (JCL) and Job Cards. Unfortunately, most users
do not fully understand either. Invariably, I used to get 3 to 4 calls a week
(usually in the early morning hours) from my co-workers about jobs they were
submitting that simply disappear. The culprit was always an illegal job card,
and usually a "programmer name" field that was too long. Once the system detects
an illegal job card, it simply discards the entire job stream. The system does
not even try to determine if there is a user, who might want to know about the
failure. Therefore, I had an ulterior motive for creating this tool.
On the surface this tool sounds like it would be fairly simple, but I wanted it
to work in any situation. I started with the IBM manuals, and I created a user
interface that included all the potential parameters that were available for the
job card. I organized them into separate pages with the most common parameters
on the main page, and the less popular parameters on optional sub-pages. Each
parameter had its own entry field, which allowed for independent validation. The
user could also press the help key to get all the information about parameter,
directly from the IBM manuals, why they might want to user the parameter, and
what limitations the parameter has. Since local restrictions, dictated in the
JES2 parameters file, affect many job card parameters, the tool also queried
this file to insure local restrictions were enforced.
In its simplest use, a user, editing a JCL stream in the ISPF editor, could
simply enter the word JOBCARD in the editor's command line, and a new default
job card would appear at the beginning of the JCL stream, replacing the old one
if one existed. A user could get a customized job card by entering "JOBCARD
EDIT" on the command line, which brought up the user interface.
Other software tool developers could also call upon the power of JCG to create
job cards for their tools. Many developers wanted to generate JCL streams in the
background, without bothering their users with the details of JCL. JCG gave them
the opportunity to create custom job cards for their jobs without storing
numerous versions in the user’s profile. These developers could also offer their
users the opportunity to access the JCG user interface with default parameters
specifically for their software tool.
ORI - The Operational Release Interface tool
ORI was another simple tool with a major impact. In the GPS system,
there are three basic operational release environments: a future/test
environment, a current operational environment, and a previous release
environment. Prior to ORI a user would have to logon and painstakingly
pre-configure the environment they wanted, and then logoff and back on to get
their session configured properly. With ORI, they simply selected the
operational release they wanted and ORI instantly and dynamically reconfigured
their session for them. Tool developers could now customize Operational Release
dependent tools. Analysts could continue working on unresolved issues in a
previous release. Testers could test new releases without the worry of
interfering with the operational baseline. Developers quickly converted existing
tools to use ORI and most tools soon displayed the ORI tag line informing the
user of their configured environment. All new tools with release dependencies
were required to incorporate ORI.
DPF - The Database Preparation Facility tool
I actually did not design DPF, but instead converted an existing
system known as DBPF into the structured format of the Software Tool Management
System. Most of DBPF is still in its original form inside the tool, just
packaged and documented more accurately and completely. Since DBPF was one of
those old systems that required users to logoff to reconfigure their sessions, I
incorporated ORI into the new tool design. DPF also generates and submits JCL
for its users, so I also incorporated JCG into the new design.
KSS - The KSS tool
KSS is not an acronym. It is actually the name of a VSAM database
file used in the GPS system. It contains many constants used by GPS. When the
developers designed the new release called Accuracy Improvement Initiative (AII),
they added several new “constants” to this file in their development
environment. Unfortunately, after they delivered AII, we discovered that these
“constants” were actually variables in the real world. Configuration Managers
used DPF to create at least 24 different KSS configurations for each test and
operational environment. The size of each of these KSS files is several
megabytes, increasing disk storage requirements. Configuration Managers also
created a separate file to document the differences between the different KSS
configurations hoping that testers could use it to select the KSS they were
supposed to use.
Therefore, I dove into the raw data structure of the KSS file, located the new
“constant” fields and wrote a simple user interface to reconfigure this file on
the fly. The KSS tool converts internal data formats into human readable formats
to show the user what is currently in the file, and converts human input data
into internal formats before inserting it into the file.
The KSS tool allowed each environment to return to a single KSS file.
TPS - The Test Package Submitter tool
The Test Package Submitter tool is truly a culmination of all the
skills I have learned over the years. It all started with an Air Force security
concern. Since I was their system security administrator, they included me in
all their meetings. Higher authorities had instructed the Air Force to reduce
the number of names on a particular access list and most of those names were
contractor testers. The Air Force in turn asked our manager for a solution. Our
manager got a small team of contractors together and the brainstorming
commenced. We identified many individual issues and proposed solutions for each,
but one underlying problem continued to emerge – as long as the testers had
access to their test package JCL, their names had to remain on the list. I
finally suggested that we remove the testers’ access from their JCL. Silence
filled the room, and our manager asked “Then how are the testers supposed to do
their job?” I responded with “I’ll write a user interface tool… They’ll love
it.” My brain was already designing the interface that would make the testers’
daily job easier, and my manager could see it in my face. He simply responded
“Great! Let’s get started, and let me know when you’re done.”
The basic design
is simple; the user interface collects input from the testers, creates a
parameter list, and passes it to the system automation software. The system
automation software generates the test package JCL and submits it for the
testers.
During the collection process, the interface validates each user input
parameter. Testers only need to enter the parameters once, simplifying the
testers’ job. Previously the tester needed to search through their entire JCL
stream finding all occurrences of all the parameters where the values had to be
changed. Not only was this task tedious and prone to human error, it was time
consuming.
The tool also verified the existence of all files and members of partitioned
datasets prior to generating their parameter list, including files that may have
existed previously and subsequently been inadvertently deleted. If the tool
finds any errors, it generates a detailed report for the Configuration Managers.
I made the tool aware of local policy.
- Testers were not
supposed to submit their test packages on the operational system, so I designed
the tool to detect which system is the current operational system, and prevent
the testers from submitting their test package there.
- Although I had previously redesigned the test environments so
that multiple test packages could run simultaneously, there were still some
restrictions. One involved the use of live resources. The local policy stated
that no more than one test package, using live resources, could execute on a
system at a time. I designed TPS to be aware of this and prevent it.
- Another local policy stated that no more than one test
package, using actual crypto, could execute on a system at a time. I designed
TPS to be aware of and prevent this as well.
I designed TPS to reconfigure itself for the various user groups. The tester
group included the maintainers, integrators, verifiers, and regression testers.
The configuration group included both the level 1 and level 2 configuration
managers. The analyst group included the analysts and systems personnel. In
addition, a special tool maintainer group that included me and one other
co-worker. As tool maintainers, we could simulate any other group or specific
user, to test the functionality of the tool.
I designed TPS to enforce configuration management of the test environments.
I designed TPS to utilize many existing tools, like JCG, ORI, KSS, and NGD (a
tool designed and developed by one of my co-workers). It even utilizes the
cataloged JCL streams that the users are used to seeing, when they are
available. If these JCL streams are not available, TPS simply and quietly
generates them in the background – a task that, in the past, the configuration
managers had to do manually.
Once we implemented TPS, the system automation software was the only entity
authorized to submit test package JCL. We revoked the testers’ authority and
they no longer needed to maintain a JCL library. We removed their names from the
access list and the issue was resolved.
I was responsible for maintaining the computer operators’ procedures and
insuring their accuracy. Since the turnover rate for operators was fairly high,
they had one primary rule about working on GPS; “Always follow the procedures.”
They also had a secondary rule; “If the procedures don’t work, call Mike.” The
accuracy of these procedures came from my knowledge of GPS, but making them easy
to understand came mostly from new operators who were in training. Additionally,
since I had a keen understanding of the systems automation software, I would
automate many of these procedures.
I was also responsible for maintaining many of the Air Force crew position
procedures, specifically any procedure that required coordination with the
computer operators. These procedures usually included some action that the
operators accomplished while the crew waited. The crew did not have visibility
to the operators’ actions and would typically grow impatient. When I first
started rewriting these procedures, I would include expected wait times and
enlightened the crew to what the operators were doing and why they needed to
wait. I even invited the crew and operators to visit each other during procedure
training exercises to see exactly what the other group is doing during those
waiting periods.
