Fine Tuning z/OS
Software Maintenance
Strategies and Tips
for Improving Your Maintenance Experience
Management Summary
Over one half of a half century – give or take – has passed since the first
mainframe business applications hit the first data centers. Hardware and Software, platforms and
paradigms, computing languages, methodologies, software: journals, tools, technologies,
and vendors, development gurus and mavens have come and gone in that time. But applications have been built. And if they were built to spec - and if the
business processes these applications supported are still a viable economic
concern, then the same applications may still be running your business - some
for years, even decades. They represent (in accounting terms) “intangible assets”
of significant value, Intellectual Property – automating, extending and encapsulating
other assets of your business. Their
book value - if viewed from the standpoint of "income generated over
time" is substantial. And software like
all corporate assets requires reinvestment if it is to be sustained, for
continuous improvement and growth to support your changing business model. This
reinvestment is named, "Maintenance".
The IEEE definition of maintenance is as follows [1]: “Software
maintenance is the process of modifying a software system or component after
delivery to correct faults, improve performances or other attributes, or adapt
to a changed environment.”
What maintenance provides was stated many years ago by Barry Boehm [2]: “...
The majority of software costs are incurred during the period after the
developed software is accepted. These costs are primarily due to software
maintenance, which here refers both to the activities to preserve the
software’s existing functionality and performance, and activities to increase
its functionality and improve its performance throughout the life cycle.” And Boehm could have finished with, “…and compounding the business value of
your software – saving you time and money.”
Just as you wouldn’t consider ignoring advances in tools, technologies and
software development process (or any other aspect of your business, I’m
betting), there are three areas of investment that will help you fine tune the
maintenance of the applications that run your business:
1. Investment in people
2. Investment in process
3.
Investment in tools
Figure 1 – Three key
organizational areas to renovate.
Investment in People
At the end of the day, your business benefits – or is handicapped by the quality
of the work produced by people, your people.
And the quality of the software maintenance performed on your business
applications by your people correlates with the caliber of three kinds of
knowledge used every by the maintenance staff in your shop:
1. Application Domain Knowledge - or the understanding of how software
intersects with your business – to some degree industry vertical knowledge but
to a large degree the unique business rules and processes that differentiate
your corporation, setting it apart from competitors in the market. The trustees of application domain knowledge
are the business users – and ensuring that the recurrent analysis needs of
software maintenance staff for authoritative application domain knowledge are
satisfied is a fine-tuning key to success. Ñ
2. Technical Knowledge – the deep mastery of the complete technology
environment underlying your application software stack – as well as of the
specific maintenance tools and processes employed at your shop should be an
obvious factor. The complexity of
production software – especially “Legacy System” code is well known. But surprisingly, maintenance and support is
often seen as a less-than-prestigious assignment, given to those not
technically up to doing new development.
This would be a recipe for disaster.
Imagine your most novice programmer, on-call in the data center when a production run ABENDs – and all heck breaks loose. Providing maintenance teams with premier
learning technical learning opportunities is a fine-tuning key to success. Ñ
3. Application
Semantics – refers to the
custom code that realizes the unique business rules and idiosyncratic in-house
processes that keep your company running. In most shops the original developers of the
application have long since moved on or retired. So, “learning” an application’s “meta-model”,
its vocabulary, understanding the coding patterns and application organization,
visualizing its data flow and processing paths, being able to predict the state
of variable values, etc becomes the first – and usually the largest challenge
your maintenance teams face. And the
value of your software investment as it is maintained over time – is determined
by the quality and accuracy of understanding of the application semantics. .
Providing maintenance teams with static and dynamic maintenance tools
that simplify this arcane task is a fine-tuning key to success. Ñ
By investing in people, the suggestion is to setup a software maintenance
center of excellence around these three areas of knowledge acquisition,
formalizing taxonomies of learning, setting measurable goals, objectives,
etc. Some of which would be general to
all maintenance, such as ongoing education in z/OS skills and infrastructure
and including any specific education on the tools and processes you adopt –
others of which would be specific to your business areas and the specific
code-base a team supports. Specific
knowledge transfer would include:
·
Periodic meetings with
users and Subject Matter Experts (SMEs) to transfer
domain knowledge of the system
·
Time set aside
specifically for improving application understanding and for building the kinds
of procedural models individuals need before diving into any code fix or
enhancement.
Is this really necessary?
No – it’s not. But
in that regard, neither should doctors bother to learn about new medical
advances, nor should auto mechanics learn how to use today’s computerized
diagnostic tools to tune car engines more quickly and accurately, etc.
As an aside, it is my experience as a z/OS consultant that
I’ve never once heard a complaint from some developer about how they’d been
over-trained or over-prepared for an assignment or project. To the contrary -
what used to be taken for granted in most companies (technical education, built
into the work year) has pretty much dissolved into "on the job
training" - a euphemism for, "catch as catch can” learning – informal
and almost always incomplete.
Not to put too fine a point on it there is most
definitely a technical and potential loss of business income and opportunity
when under-trained developers are swinging well-above their
software-programming weight class.
Consider anyone from a mechanic to a surgeon. How comfortable would you be with their work
knowing that there are gaps in their knowledge and capabilities? Put another way, how would you feel about a
dental apprentice, just about to do your root canal?
Investment in Process
The need for a methodology – or even multiple methodologies
(one for each of the four distinct categories of maintenance described in table
1 below) that address how to tackle the diverse and complex issues presented by
z/OS software maintenance in a systematic way would seem to be obvious. And yet a formalized process for ‘how to
maintain an application at XYZ Corp.” is rarely found – and almost never
considered as important as the System Development Life Cycle (SDLC) used to
develop new systems.
Admittedly, you might find documentation; informal “cookbooks”
and other collateral based on lessons learned for doing certain aspects of
maintenance work (i.e. steps for handling data center ABENDs and on-call
situations, etc.) And if you’re lucky
enough to have a true Software Maintenance Methodology for handling Adaptive
Maintenance in your shop I congratulate you.
But you’re not in the majority.
Like the applications they support, your shop’s software
maintenance methodology would – if it existed - also fall into the category of "Intellectual
Capital". And investment in
correcting, adapting and perfecting your maintenance processes – revising best
practices, plowing lessons learned back into the software maintenance
methodology is a fine-tuning key to success. Ñ
FYI, there is an abundance of industry research, white
papers, studies, etc. on maintenance methodologies published and available from
such organizations as the IEEE, ACM, etc. Meaning, if you don’t currently have a
maintenance methodology you do not have to start building one from
scratch.
It’s important to point out that we do not advocate any
particular methodology, nor do we believe that a rigid and heavy methodology is
“the answer” to wrestling software maintenance problems to the mat. In our experience, if a methodology becomes onerous,
or is pushed dogmatically, development teams will either find a way out of
using it (and often end up “winging things” – rarely a good situation), or it
will become yet another cost issue – not a solution to the problem it was
intended to solve.
What we’re recommending in this paper is that you provide
enough systematic structure that developers do not have to make determinations
and judgment calls during projects that end up as “wheel spinning” or worse, as
strategic mistakes that cost the organization time and money (this is especially
for emergency maintenance tasks – or data center “hangs” and bug fixes).
One strong process-specific fine-tuning recommendation is
to appoint a single person to be the software maintenance architect at your
shop – and task that person with responsibility for establishing organizational
controls: developing and refining the maintenance plan, publishing a dictionary
of your firm’s software maintenance terms and vocabulary, creating and managing
your software maintenance methodology, surveying and analyzing the maintenance
problem factors affecting maintenance in your shop (i.e. SME knowledge, maintainability
metrics, reconciling problem factors from the user’s versus developer’s
perspectives, quality of documentation, programmer effectiveness/motivation/turnover,
the age and condition of your business applications, the integrity of
production data, machine availability, etc. ), and uncovering and putting into
practice worthwhile technical insights, and empirical data points available
from journals, conferences, etc. Ñ
Anecdotally, one shop we are aware of – through the
research their Software Maintenance Architect undertook – found that by
“batching” adaptive maintenance requests, they ended up improving the
efficiency and throughput of their certain projects by almost 20%. Not a number to sneeze at these days
(actually, not a number to sneeze at any days).
A final reason for appointing a single person is that the
above-mentioned technical insights and data points are often published in the
arcane lexicon of formal research.
Searching for an individual with the right mix of practical production
experience and 800 Verbal
Tools
As fate would have it, the software stacks running on –
and attached to your z/OS have evolved and grown in size and complexity just as
corporate belt-tightening due to our most recent recession has inversely reduced
staff and time to microns of availability.
So from a technology perspective, tools are no longer a Nice-to-Have,
they're a necessity.
Using the same manual processes and approaches throughout
the maintenance lifecycle that would have been used 10, 20 or even 30 years ago
ensures that things don't get done efficiently - not done to spec – or that they
cost more than necessary – perhaps all of the above. Consider as an example how you would debug a
situation where incorrect values showed up in a Java/web page that rendered
data from in an IMS database accessed through a CICS transaction using a Web
Service. How to track a problem like
this down? Where even to start?
Fortunately, the software industry has moved
beyond
Outsourced Maintenance
Adding to the drivers for taking a deeper
look at the maintenance being done to the software assets that run your
business is the trend over the last ten years towards outsourcing
maintenance. Which adds the additional
wrinkles of distance (latency), communication (the lexicon of business is
subtly complex – amplifying errors due to misunderstanding and
misinterpretation), loss of control (over the maintenance practices of your
outsourcer) and is exacerbated by the loss of in-house application/business
expertise.
Overcoming these additional obstacles
requires tightening down on project management (through increased project
governance – either with the help of more rigorous methodology, specially
designed project governance products or probably both), adopting technology
that provides pragmatic scoping of maintenance projects, and real-time
collaborative team-development functionality, and making all of your remaining
staff better at their jobs by adopting the above keys to success.
Why Invest in Fine-Tuning Software
Maintenance?
Research shows that the total investment in
maintenance activities is between one and nine orders of magnitude more than
what is invested in software development and delivery.
[Effective Software Maintenance, Stanislaw Jarzabek,
Auerbach Publications]. Even
taking the lowest of these estimates, it’s clear that incremental improvements
in your shops maintenance organization, its people, processes and tools could
easily be the single best financial investment you make.
In this detailed white paper we offer specific recipes
for improving or better utilizing Knowledge, Process and Tools that will keep
your development firing on all cylinders - as they continue to manage the
complexities and continue to do the heavy lifting that maintenance has always
been, in I/T.
Executive Summary – Keys to Success
Software is the foundation of
today’s business. While analysis,
design, construction and deployment are all key phases of the development
lifecycle, Software Maintenance, which ensues the moment your application goes
into production, and continues until it is sunset, is undoubtedly the most
important single stage.
A successful application
development area needs a dynamic maintenance plan, a dedicated maintenance
supervisor and a first-string maintenance team to continually renew and sustain
the software, allowing you to exploit its business value, and prevent costly
data failures, defects and applications that flat-line in production.
Ñ Endorse the
business value of your production software and the maintenance process of that
software as a capital asset across your organization
Ñ Employ a software
maintenance methodology that takes into consideration the different categories
of software maintenance (see next section) – and undergoes periodic refinement
and renovation, based on industry research and your in-house “lessons learned”
Ñ Establish a
position overseeing software maintenance, with responsibility for the establishment
of your shop’s maintenance plan, the progress of maintenance practices, the
integration of the maintenance tools and methodologies used – both in house and
outsourced
Ñ Ensure that software
maintenance staff has easy access to application domain knowledge Subject
Matter Experts
Ñ Provide software
maintenance personnel with access to premier technical learning
Ñ Install key
software maintenance tools for: 1. application understanding/impact analysis,
and 2. Dynamic code analysis and testing, 3. Team collaboration, and 4. Source
Control Management
Fine Tuning z/OS
Maintenance – Deeper Dive
Let’s begin this section by more precisely defining
terms and vocabulary. The
|
|
Unscheduled |
Scheduled |
|
Reactive |
Emergency maintenance: Unscheduled corrective
maintenance performed to keep a system operational.” |
Corrective maintenance: Modification of a software product
performed after delivery to correct discovered faults. Adaptive maintenance: Modification of a software product performed
after delivery to keep a computer program usable in a changed or changing
environment. |
|
Proactive |
|
Perfective maintenance: Modification of a software
product performed after delivery to improve software quality, performance or
maintainability – which is a key factor in maintenance productivity. |
Table
1 – the IEEE Software Maintenance Categories
Various other organizations (the International
Organization for Standardardization – ISO for example) also have categorized the
types of software maintenance companies do – refactoring and extending Table
1’s categories and definitions. But
there is a high degree of overlap among all the experts in this area.
At the risk of breaking the above
disclaimer against adopting overly-onerous methodology, you might well consider
the IEEE’s list, just a starting point for your own shop’s maintenance research
and in-house process. Other experts and practitioners in this field have broken
down each of the steps by numerous other taxonomies. See Table 6 – Preventative Software Maintenance –
Classification Factors, extension to IEEE [4] Risto Vehvilainen, for an excellent example extending
IEEE’s categorization.
We continue now, with our People/Process/Tools
motif, but we’ll drill down to explore more technical and detailed topics.
People
Who are the people at your shop who
do maintenance? Through informal research
done by
|
Maintenance
Team Organization |
Used by
Percentage of Shops |
|
Developers who wrote the code
maintain it (primarily) |
7 |
|
Separate maintenance in-house team |
16 |
|
Local consultants |
8 |
|
Offshore/Outsourced maintenance |
40 |
|
Other (various combinations of the
above) |
29 |
Table
2 – Software Maintenance Team Break Down
The first thing you notice when you
start adding up the percentages, and slicing the maintenance organizations a
different way is that roughly 60% of the world’s software maintenance is done
by external development teams – not by a shop’s own technical staff. Because of this, it might be tempting to
declare victory over maintenance, but unfortunately the opposite is true.
Fine tuning software maintenance in
a world where developers who did not write the code, have never even seen the
code and your company’s programming nomenclature, may not know your company –
possibly do not even understand your vertical market space - which translates
into each application’s vocabulary meaningful variable and procedure names,
etc. – and don’t have the kinds of working relationships with SMEs and other critical staff, is both harder and a larger
imperative.
In light of this, we return to our initial
break People break-down, and offer additional keys to maintenance experience
success.
1. Application Domain Knowledge - The importance of the business and
programmer analyst. A long time ago
there were users – system analysts and coders - and problems. As the system analysts were overwhelmed in
translating business requirements to procedural specification, flow charts, Warnier – Orr diagrams, so they could be precisely
understood, and used by the coders.
Net? System Analysts – and the
business-process knowledge they acquired became a development bottleneck.
The solution that the industry evolved
into?
The coders – that stayed at various shops (or within a domain-range of
shops – say, insurance, in
In many respects, the current situation is analogous to
the late 1960’s/early 1970s – with a split between language coders - who can
spell PERFORM - and business/system/programmer analysts who understand how a
particular division of an insurance company adjudicates claims. The key to success for your company is to at
all costs – nurture and retain those on staff who have deep technical business
analysis knowledge – of how your current application stack operates. Ñ
There has been much talk for a long time about how to
replace the soon-to-retire aging COBOL population of the world (okay…so this
past recession has mitigated this problem to a large extent). But primarily, the issue is not COBOL
programming/technical skills. The
problem is how to know what the business term (and variable name) “coverage-code”
means changing data edits, or understanding which production job can be
released in the data center if a hang occurs and there are file update and
synchronization dependencies, and why (yes, “why”) a particular record value is
considered required. To
paraphrase a current credit card advertising campaign, this intersection of
business and technical data is: “priceless”.
For more on the topic of application domain knowledge
transfer – and especially the benefits and a structured process, see Pigoski [6], “Software
Maintenance Training: Transition Experiences.”
2. Technical Knowledge – Given the above breakdown of “who’s doing the maintenance on your
systems” finding ways to certify that the COBOL programmers who are making
changes to your general ledger programs got more than 3 weeks of Entry Level
Training would be a good start. A key to
success then is, for you to assume responsibility for certifying technical
credentials – of course for your in-house staff, but also for any vendor that
will end up with long term contracts and where you have already reduced
in-house technical and programmer/analyst staff. Ñ There are industry-supplied tests and certification exams
available in most of the z/OS disciplines (example, www.iccp.org )
Ensure that your own in-house staff
have unfettered access to technical learning content, and expertise –
especially any groups dedicated to Emergency Maintenance. Understand that we are not proposing a
limitless budget of expensive vendor-training classes, but that someone –
perhaps the Software Maintenance Architect is responsible to establish a set of
links to the kinds of quality materials available on the internet; from
Finally, it’s important to note that an opportunity for
saving time and money exists by educating maintenance developers well, in
traditional-z/OS specific areas of highest impact:
·
Internal Data
Representation problems – which
cause a significant percent of production ABENDs on z/OS
·
Language-dependent
issues – such as fall-thru problems when
mixing COBOL Sections and Paragraphs, flow-issues, when replacing Next Sentence
with Continue, etc.
·
All of the
MVS System Completion Codes – their common
causes and solutions: http://www-949.ibm.com/software/rational/cafe/thread/2241?tstart=15
·
Common
Business and z/OS Application Problem Areas – Date arithmetic and Y2K-windowing, static versus dynamic calls, etc.
·
The Compiler
Options – and their effect on testing,
performance and debugging: http://www-01.ibm.com/support/docview.wss?uid=swg27001475
·
Run-Time software
issues and/or common “tricky coding” Constructs – DB2 SQL problems mixing
It might be mentioned that “deep-dive” education is
required here. One definition of a
“Legacy System” is “a system that works”. Another is “a complex system (that
works)”. Assigning an intern to a
production support role for a complex batch system is tantamount to sending a
Little League pitcher to relieve a major league ball team, in the 9th
inning, with the bases loaded – in the world series.
3. Application
Semantics – the breakdown in
table 5 brings this key factor into crystal clear focus:
·
For in-house
maintenance done by developers who wrote the code – Application Semantics
should not be an issue (and if it is, you have bigger problems than just
Software Maintenance)
·
For Separate
maintenance teams from within your organization and for Local Consultants –
Application Semantics will be a straightforward learn. It is still recommended to use static and
dynamic code analysis tools to simplify and streamline the learning, reduce time
and risk and level the playing field (Static and Dynamic Analysis tools can
raise deep technical semantics and expose the use of key elements at a much
higher-level of abstraction than traditional, manual source-based unstructured search
activities.
·
For
Offshore/Outsourced maintenance – using static and dynamic analysis tools is
critical – and not just by lead project analysts during the “analysis and
scoping” phases of a project. It is
critical to have these tools available during technical construction and
testing phases of projects. Consider the
following:
o
You’re making
changes to a computation, and all of a sudden it occurs to you that you’re not
entirely sure how or where a variable’s value has been derived from the file
read to your statement. What do you do
at this point – keep typing? Of course
not. You drop back into
code/program/application analysis in order to refine or complete
the mental map of the logic you’re revising, before changing another line of
source code.
o
Or you’re changing
a performed (chained) routine and you need to know that fall-through will not
occur, or that it will not affect other nested paragraphs or functions.
o
Suppose you’re
testing a module at the source-level, and watching as the lines execute in
sequence. All of a sudden the “FALSE”
path of an IF statement is taken when you expected the “
·
Every developer
that’s ever modified code has experienced the above. Without Static and Dynamic Analysis tools
you’re back to using manual ISPF find commands and thumbing through program
listings – essentially “building a house with nails and hammers instead of
power tools”. Yes – it can be done. But much more slowly, much more
expensively. And it can only by
substantially more expensive veteran developers. Use the Static and Dynamic Analysis tools to
regain control over this key are of Software Maintenance – learning the
Application Semantics.
Investment in Process
A typical process-flow has been captured by IEEE in its IEEE-1219 [3] document,
and is reproduced below in Figure 2. It organizes
the software maintenance process – or lifecycle in seven steps. You may recognize some or all of these steps
as occurring in your shop’s approach to software. Or you may be have similar, fewer or more
steps depending on the formality of your methodology and how it is applied to by
different management teams.
There is much more detail in the IEEE document, such as input and output
deliverables, the activities grouped, related and supporting sub-processes, the
control, and a set of metrics. If you
haven’t built or acquired your own formal maintenance process, you will be
hard-pressed to find a better starting point.
Figure 2 the IEEE
Software Maintenance Lifecycle (process)
However, three points on this:
- As
described earlier, it is the experience of most of the shops we contacted
for this research that the needs of each category of maintenance
(emergency, corrective, adaptive and perfective) expand out to additional
sub-processes for most all of the steps in the IEEE process (see table 2).
- Your
shop’s own experience, priorities and prerogatives – even approach to
implementing methodologies always supersede a generic, one-size-fits-all
approach
- And
as lightly touched on earlier, working towards an iron-clad, deep
procedural methodology might be an indulgence that will not
return-on-your-investment as anticipated, and must be considered in the
context of your shop’s inclination towards: Results-based, versus
process-driven activities and project management.
Tools and the Steps in the Process
When considering the required, optional and effective use of tools
throughout the Software Maintenance Lifecycle, consider that there are two
categories:
1.
Tools that bring value to
each and every step – these include Cross-platform
collaboration products and source control management tools
2.
Tools that have step-specific
advantages – notably the static and dynamic
analysis tools we will dive into, shortly
Your shop might create its own maintenance-tools-process-driven matrix,
which will change over time, as new technologies and new problem domains appear
on the landscape, but a typical correlation of Software Maintenance Lifecycle
phases and tools that help fine-tune your efforts to save time and money is
shown below in Table 3. In the next
section we drill-down on the different tools and functionality and features –
as applied to Software Maintenance.
|
|
Tools Used at Specific
Lifecycle Steps |
Tools Used Throughout the
Lifecycle |
|
Problem/modification identification, classification, and prioritization. |
Dynamic Testing tools – for initial breakdown
of emergency and corrective maintenance problem definition |
Source
Code Management |
|
Project Scoping Analysis. |
Dynamic Testing and Static source
code analysis tools – for trustworthy/accurate
results and efficient/cost-effective analysis activities |
|
|
Design. |
n/a |
|
|
Implementation Construction |
Static source code analysis tools – for solving
inconsistencies in developer
understanding |
|
|
Regression testing System testing. |
1. Dynamic Testing tools for revealing problems
during test & debugging. 2. Static source code analysis tools for
solving problems, stemming from an inconsistent developer understanding
(which – if you think about it, is the source of the preponderance of
software problems) |
|
|
Acceptance testing. |
Dynamic Testing tools |
|
|
Delivery |
n/a |
Table 3 – Typical lifecycle
phases and tool usage correlation
Tools
So – the first consideration that must be addressed in this section is “why
tools?” – and the second is “which tools?”
For decades software developers thumbed through listings using paper clips
and sticky pads when they needed to bookmark variable references and ran out of
fingers, or on z/OS they used ISPF search operations, and eventually code went
into production – and worked. Let’s
discuss the value proposition for using the four categories of tools shown in
Table 3 – and discuss the must-have features that offer enough value for you to
invest and use them.
Source Code Management
This is the easy sub-category. Given the size,
complexity and dynamic nature of modern z/OS development scenarios – especially
with the introduction of offshore development, it is simply axiomatic that
without a robust Source Code Management (
Not only is it common for multiple versions of the same software to be
deployed at different sites, but for bug fixing and adaptive maintenance
automating the control of access to source versions is a long-established
best-practice, and doubtless you not only have Source Code Management up and
running at your shop, but you’ve customized the hell out of it by now, as well.
Team Collaboration
Unlike Source Control Management you may not have been introduced yet, to
Team Collaboration software, which is essentially a software dashboard
that:
·
Allows project leaders to define,
organize, and track and your software maintenance, support and development
teams and their projects (not program source, people and processes)
·
Provides simplified interaction
and communications between Subject Matter Experts (SMEs)
and the technical programmer/analysts and architects who interact with them –
this interaction can be captured electronically for reuse, refinement and
documentation.
·
Accommodates Management reporting
and governance of project milestones and deliverables
·
Enables project technical
developers to share development artifacts (source and model-based) in-context
of doing project work
By raising the team concept to the level of an automated entity and
providing integrated workflow among all stakeholders, software maintenance
projects (across all four categories) are freed (to degrees) from repetitive,
manual tasks, and allowed to focus on business and technical problems at
hand. At the risk of over-simplifying,
you can think of Team Collaboration as reducing the costs of administrivia and bureaucracy, which can be a significant
cost and barrier to productivity enhancements in an organization. Speaking personally, and as someone who spent
the better part of a decade enmeshed in software maintenance projects at
various Global 2000 shops, it is hard to argue against reducing mind-numbing
regimentation, in all its semblances.
Finally, revisiting our previous points on challenges introduced by offshore
maintenance projects, Team Collaboration was born to solve the latency,
communications and logistics issues that are part and parcel of half-a-world-away
shared project efforts (or even half-a-block away, the point being that
distributed development vs. central development brings organizational
challenges that must be addressed, to gain the understood benefits).
Static and Dynamic Code Analysis Tools
Analysis tools (both static and
dynamic) assist developers in “gaining intellectual control”
[Larry England,
Research has shown that there is
a diseconomy of scale in productivity when maintaining large
systems. This is due to the large number
(often in the hundreds to thousands) of variables-and-functions-of-interest
scattered throughout large amounts (often in the hundreds of thousands to
millions of lines) of software.
Historically only practiced
veteran z/OS developers – those with over 5 years of production “combat
experience” are capable of building precise, clean and faultless
internal/mental maps (thereby gaining “intellectual control”) of the code execution
paths, key variables, their state transitions, etc.
Over the past 20 years many of
these individuals have retired or moved on to management positions, and now many
maintenance teams are in the untenable position of using relatively inexperienced
developers to make technical judgment calls on changes to core systems –
systems that have only gotten older, more complex, brittle and (net) more
difficult to fathom than ever before.
This represents a remarkable level of risk. While all of the above-mentioned Keys to Success are cumulatively helpful, electronic
automated analysis software consolidates and amplifies your savings of time and
money on an order of magnitude.
Static
Code Analysis
One of the unintended
consequences of Y2K was the birth of Static Code Analysis tools. These were successfully used in Y2K Adaptive
Maintenance, as
Features
The information needs of
maintenance and support staff vary by the type of application learning/comprehension
model employed. It is commonly
understood that you “learn” a program either top-down (by studying
documentation, talking with SMEs, and ranging over
the functions an procedures stating from the entry point – one logical branch
at time, or bottom-up, starting with a specific function, record or variable –
often working from program listing cross-references, and tracing backwards and
upwards. By the way, if this sounds like
a meticulous, difficult, error-prone grind?
Too right.
It is, and it’s also the nucleus of software maintenance project. In reality – for projects or systems of
significance programmers switch between the two approaches, integrating
information into a final mental model of the application’s execution – and how
it relates to the business or situation at hand [7].
Tools then, that comprise features
automating or simplifying analysis top-down and bottom-up will
fast-track your teams to improved productivity and lowered risk. Such tools should (at a minimum) contain the
functionality described in Table 4 – and
come with a graphical (mouse-based) IDE – largely because the task of
understanding code is keyboard-less, and driving through diagrams and
hyper-links with a mouse instead of typing FIND commands into green-screen
panels is actually a critical element of the usage model.
The application semantics – or “metadata”
(i.e. information about information) surfaced by static analysis should
be contained in an easily search-able dictionary or database. The tool’s metadata, extracted and exposed as structured
top-down diagrams and bottom up element-level reports – cross-referenced and
hyper-linked must (must) be built from the source code itself – in order to
provide the maximum semantic integrity.
It should be pointed out that the
list of requirements in Table 4 is by no means exhaustive. A good Static Analysis tool can assist in
answering the majority of Where/What/When/How – and especially “Why” questions
that forms the provenance of Software Maintenance projects.
|
Feature/Function |
Description |
|
|
Top-down systems views |
Iconic/Graphical images – and
detailed reports, that correlate to system flow charts |
Rational Asset Analyzer |
|
Hyper-linked transaction and
call/calling program views |
Execution chains abstracted up to a graphical level with accompanying
detailed report analysis. Hyper-linked for top-down study. |
Rational Asset Analyzer |
|
Hyper-linked batch job views |
Individual steps in a job (“run-unit”), graphically exposed and
hyper-linked, with sub-elements that show (at a minimum) programs, utilities,
external files, etc. used by each job step. Accompanying detailed report
analysis. All hyper-linked for top-down study. |
Rational Asset Analyzer |
|
Hyper-linked Data Flow Diagrams |
Hyper-linked graphical depiction of file usage throughout a batch job or
transaction |
Rational Asset Analyzer |
|
Program or Module “Functional Decomposition” |
Different views or slices into programs breaking the |
Rational Asset Analyzer |
|
Impact Analysis |
“What If” analysis, before source modifications are made – at all levels
of scope: Program, Run-Unit, Transaction, System-wide |
Rational Asset Analyzer |
|
“Where Used” and “How Used” Analysis |
Ability to isolate variables and reveal only: statements that change the
variable value, statements that test or reference the variable value,
declarations, etc. |
Rational Asset Analyzer Rational Developer for System z |
|
N-way search capabilities |
Search in meta-model for elements Search in source for elements and element subset |
Rational Asset Analyzer Rational Developer for System z |
|
Customized code windows |
The ability to create on-the-fly combinations of editable windows in
order to provide the best source-level perspective an area of code |
Rational Developer for System z |
|
Subset capabilities |
Ability to group (subset) |
Rational Asset Analyzer Rational Developer for System z |
|
One-click Access declaration of variable |
Ability for the tools to “keep up with the developers exploratory
process” |
Rational Developer for System z |
|
Expand/Collapse code sections |
Isolate to manage-able code blocks |
Rational Developer for System z |
|
Extensible Query Language |
The ability to compose completely-custom queries against the meta data in
the tool’s dictionary or database |
Rational Asset Analyzer |
|
Integration with the “Development Workbench” |
The ability to use the results of all of the above analysis reporting
directly inside the edit/compile/debug tools |
Rational Developer for System z |
|
Code Quality Metrics or development “Best Practices” |
Show range violations (bad GO
TO structures), recursion detection.
List inefficient variable declarations, etc. |
Rational Asset Analyzer Application Performance Analyzer |
|
Perfective Maintenance |
Finding dead code, unused copybooks, etc |
Rational Asset Analyzer |
|
Support Emergency Corrective Maintenance analysis |
Show “where modified” code to trace down root-causes of S0C7 and S0C4,
etc |
Rational Asset Analyzer |
|
Code slicing and Business Rules Mining |
Separating business logic from persistence and presentation logic |
Rational Asset Analyzer |
Table 4 – Static Code Analysis – a
List of Recommended Features
Dynamic
Code Analysis
While Static Code analysis will
be an indispensable aid in helping to solve many classes of Software
Maintenance, for certain problems – especially some in the Emergency-Corrective
Maintenance category you will need another view of the production application –
a view that is provided by actually running the live code – at the source level
and following the execution sequence.
This is called Dynamic Code Analysis.
Dynamic Code Analysis tools allow
you to inspect – at the detail level – and without conjecture:
·
The instruction sequence – every statement “live”, stepped through and
animated running against the software stack that is the deployment platform
·
Variable values – in program storage, file buffers, database records temp storage, anything
the code has addressability to
This type of real-life monitoring
– at the source code level (which is the logical currency developers trade in)
is absolutely unrivaled, for solving some of the most obstinate and bewildering
software analysis problems – and is invaluable for corrective maintenance in
every form.
Table 7 lists some of the more
prominent and useful Dynamic Code Analysis features. Like source code management, it is likely
that you already own a product that can be used for Dynamic Code Analysis. But are you taking advantage of these tools
for Dynamic Code analysis (and not just debugging)? The table’s Feature/Function column lists
some of the ways that Dynamic Code analysis can provide teams with
automated-assistance, in areas that are traditionally labor intensive (and
error-prone, due to the size and scale of complexity as concerns z/OS Legacy
Systems).
Wrap-up
Software is like
other knowledge-based R&D products, an intangible corporate asset. The value of intangibles is based on the
income they are expected to generate in the future. Alfred P. Sloan (long-time president
and chairman of G.M.) once said, "Take my assets - but leave me my
organization and in five years I'll have it all back." We
could paraphrase this as, “Take my book-value assets – but leave me my
software and in 5 years I’ll have it all back.” That of course, pre-supposes you are
maintaining your software, over the 5 years – in a quality manner.
It is naïve to think that
organizations have buckets of money to throw at the dilemma of software
maintenance, however in this white paper we have attempted to provide solutions
that incorporate:
·
Internal
process reengineering
·
Web-based
knowledge acquisition
·
Modest
investment – or repurposing of development software with definitive maintenance
value
By implementing one or all
of these solutions, we are confident that you will see:
·
Lowered defects – which lowers costs, and raises the overall quality of your data
center operations
·
Improved productivity – that can helps balance the demand for changes and queued
enhancements with new systems functionality
·
Savings in hardware and software costs – through more effective and productive developer
usage models
·
An increase in the shelf-life (and respective business value) of your
z/OS applications –
a business value which unlike other intangibles grows over time
But at the very least, by
having read this far, you’re already more tuned in, and hopefully a little
wiser for having spent the ten minutes or so.
Net: Don’t’ let your z/OS legacy applications flat-line. There’s no “Dr. House” waiting ambling about
with last-second miracle cures. The good news? It
doesn’t require a diagnostic genius to make things better - just someone who
appreciates the value of “systems that work”.
Jon Sayles, October 2009
Thanks to the following
|
Feature/Function |
Description |
|
|
Dynamic, source-level
step-by-step animation |
Iconic/Graphical images – and
detailed reports, that correlate to system flow charts |
Debug Tool |
|
Present variable values: - Characters - Hexadecimal |
Execution chains abstracted up to a graphical level with accompanying
detailed report analysis. Hyper-linked for top-down study. |
Debug Tool |
|
Interrupt execution based on
convenient set of options: |
z/OS applications
are almost always too large to step through from their beginning. Various break points must be set in order
to make the tool usage practical:
Entry to module, value change, conditional value, etc. |
Debug Tool |
|
Selective subset of Variables
to “monitor” |
z/OS applications
have far too many variables to monitor all at once, and viewing just the
variables on the current line is too restrictive. You must be able to make persistent lists
of variables for monitoring |
Debug Tool |
|
“What if” Analysis |
Besides passively viewing the code execution flow and variable values at
run-time, developers often need to modify storage values, alter the
instruction sequence, iterate over
complex logic, in order to really understand the algorithm or code
pattern |
Debug Tool |
|
Access to z/OS Storage and Registers |
For debugging assembler programs, or for analyzing a failed instruction
at the machine code level |
Debug Tool |
|
GUI Integration with the development Workbench |
Far from cosmetic, graphical, integrated Workbenches that include Dynamic
and Static Analysis tools are simply expected. To understand this, consider how often you
open DOS prompts and type: copy c:\directory1\subdirectory1\myfile.dat to
d:\directory1\subdirectory1\myfile.dat these
days? Or do you usually Drag &
Drop files? J |
Debug Tool and Rational Developer for System z |
|
List programs called at run-time, during a specific transaction |
When learning an application (adaptive maintenance) or studying the root
cause of an ABEND situation (corrective maintenance) it is acutely important
to understand the precise execution route |
Debug Tool CICS Interdependency Analyzer |
|
The sequence of transactions within an application |
When learning an application or studying the root cause of an ABEND
situation it is acutely important to understand the precise execution route |
Debug Tool CICS Interdependency Analyzer |
|
List files and databases (Read or Updated) used at run-time during a
transaction |
Following the chain of input/edited/output files as they are read
from/written to throughout a transaction or batch application is one of the
most common maintenance strategies.
Simplifying/Automating this process alone can be worth the price of
any product |
Debug Tool CICS Interdependency Analyzer |
|
For Emergency (ABEND) Situations, provide initial debugging data points |
Integrated “fault analysis” where valuable research information, such the
exact statement in a program that caused an ABEND (and surrounding code) plus
the values in the variables under consideration can be critical in timely
diagnosis and correction of the problem |
Debug Tool + Fault Analyzer |
|
List program lines not tested – as a result of Regression Data |
A “test coverage” utility, that shows marked source of lines tested – and
lines untested |
Debug Tool |
|
Integration with static code analysis |
The ability to research problems alongside their discovery promotes
productivity, cuts down on errors, etc |
Rational Asset Analyzer |
Table 5 – Dynamic Code Analysis –
a List of Recommended Features
|
|
|
|
-------------------Software
Categories – IEEE Definitions------------------------ |
|||
|
Category |
Sub-Category |
General Considerations |
Corrective
Maintenance |
Adaptive
Maintenance |
Perfective
Maintenance |
Emergency Maintenance |
|
Software
Maintenance
Methodology |
Exists? Periodically
reviewed and revised? Project
teams trained in its use? Improved
by integrating industry research and best practices? |
The
steps and use of tools, in-house expertise, collaborative methods, etc. for
production bug-fixes |
The
steps and use of tools, in-house expertise, collaborative methods, etc for
adding code to production assets. |
The
steps and use of tools, in-house expertise, collaborative methods, etc. for
measuring performance, Areas to touch within an application and to avoid,
etc. |
Includes
sections on optimal use of tools, in-house expertise, collaborative methods,
etc. for emergency bug-fixes, ABENDs, production problems, etc. |
|
|
Software
Maintenance Tools |
Static
Code Analysis |
Tools
available that cover your software stack requirements? Teams
trained in the use of tools? |
Automated
analysis of application semantics |
Automated
analysis of application semantics |
Queries
that create reports highlighting potential candidates for performance
improvement |
Steps
and examples for use in production ABENDs – and in different post-production
bug fixes? |
|
Dynamic
Code Analysis |
Tools
available that cover your software stack requirements? Teams
trained in the use of tools? |
Tracing
code, testing, and learning application semantics |
Tracing
code, testing, and learning application semantics |
N/A |
Understanding
“WHAT” and “HOW” of ABENDs |
|
|
Collaborative
software version control |
Tools
available. If
outsourced maintenance, very useful to overcome challenges of communications,
time, and distance. Teams
trained in the use of tools? |
Real-time
fault analysis sharing |
Project
tracking, and sharing of design documentation and user communication |
Project
tracking, and sharing of design documentation and user communication |
Real-time
fault analysis information sharing |
|
|
Software
Maintenance Teams (People) |
Methodology
Training |
Technical
Training Plan |
Understanding
of Tools and process |
Understanding
of Tools and process |
Understanding
of Tools and process |
Understanding
of Tools and process |
|
Technical
(language and platform) training |
Access
to internet-based technical consumables: Redbooks, language documentation,
white papers, Q&A forums, etc. |
Easy
look-up of language elements and analysis meta-data – to support Application
Domain understanding |
Easy
look-up of language elements and analysis meta-data – to support Application
Domain understanding |
Access
to |
Understanding
of MVS System Completion Codes and common causes |
|
Table 7 –
Keys to success, over the cross-section of different categories of Software
Maintenance
|
Classification Factor |
Classification Example |
Importance |
Preventative Maintenance |
|
Process |
1.
Fault correction 2.
Small change 3.
Large change 4.
Maintenance Project 5.
Advise 6.
Study |
An
important instruction and therefore needed |
Study and
maintenance project |
|
Priority |
1.
Immediately 2.
Soon 3.
Planned
schedule 4.
On
suitable situation |
An
important instruction and therefore needed |
Planned
schedule |
|
Compulsion |
1.
Compulsory 2.
Compulsory
but with a flexible schedule 3.
Optional |
An rather
important instruction and therefore needed |
Optional |
|
Change |
1.
Code 2.
Specifications 3.
Procedures 4.
Instructions 5.
Descriptions |
An
important instruction and therefore needed |
Anything |
|
Target |
1.
Very
small, not registered as a separate request 2.
Small
registered as a separate request 3.
Moderate 4.
Large |
|
At least
moderate |
|
Influence
on software |
1.
Corrective 2.
Changing
functionality 3.
Changing
capacity or performance 4.
Improving
quality 5.
Applying
to new environment |
Interesting as additional Information from a reporting point of view |
|
|
Reason for
the change |
1.
Error:
software doesn’t work according to the specifications 2.
Need
to improve software cost/benefit analysis 3.
Changes
in the business of the client 4.
Need
to change technological environment |
Interesting as additional Information from a reporting point of view. |
Anything
except error |
|
Initiator
of the change |
1.
User 2.
User
liaison 3.
IT
management 4.
Maintenance 5.
Other
IT person |
Interesting as additional Information from a reporting point of view. Can be concluded from other
factors. |
IT
management or maintainer |
|
Role of
the maintainer |
1.
Reactive 2.
Proactive 3.
Co-operative |
Interesting as additional Information from a reporting point of view. Can be concluded from other
factors. |
Co-operation |
|
Planning
Scope |
1.
Immediate 2.
Short
range (a few months) 3.
Year(s) |
Not very interesting. Can be concluded from other factors. |
Year |
|
Preparation
before implementation |
1.
Not
needed 2.
On
request level 3.
On
application level |
Not very interesting. Can be concluded from other factors. |
On application
level |
|
Risks |
1.
Minor 2.
Moderate 3.
Major |
Useful as additional information |
At least
moderate risks |
|
Importance
for Business |
1.
No
special importance 2.
A
moderate importance 3.
Critical 4.
Strategic |
Useful as additional information |
At least
moderate importance |
Table 6 – Preventative Software Maintenance – Classification
Factors, extension to IEEE [4] Risto Vehvilainen.
References
[1] IEEE Std. 610.12, “Standard Glossary of Software Engineering
Terminology”, IEEE
Computer Society Press,
[2] Barry W. Boehm, Software Engineering Economics, Prentice Hall
[3] IEEE Std. 1219-1998, “Standard for Software Maintenance”, IEEE Computer
Society Press,
Los Alamitos, CA, 1998.
[4] Risto Vehvilainen, "What Is Preventive Software Maintenance?," icsm, pp.18, 16th IEEE
International Conference on Software Maintenance (ICSM'00), 2000)
[5] “Effort Estimation for
Corrective Software Maintenance”, SEKE '02,
[6] “Software Maintenance Training: Transition Experiences”, Thomas M. Pigoski and C. Steve Looney. 1063-6773/93 1993 IEEE
[7] “Programming Understanding Needs During
Corrective Maintenance of Large Scale Software”, Anneliese
von Mayrhause and A. Marie Vans. 0730-3157/97 1997 IEEE