Fine Tuning z/OS Software Maintenance

Strategies and Tips for Improving Your Maintenance Experience



Management Summary

Over one half of a half century – give or take – has passed since the first mainframe business applications hit the first data centers.  Hardware and Software, platforms and paradigms, computing languages, methodologies, software: journals, tools, technologies, and vendors, development gurus and mavens have come and gone in that time.  But applications have been built.  And if they were built to spec - and if the business processes these applications supported are still a viable economic concern, then the same applications may still be running your business - some for years, even decades.
They represent (in accounting terms) “intangible assets” of significant value, Intellectual Property – automating, extending and encapsulating other assets of your business.  Their book value - if viewed from the standpoint of "income generated over time" is substantial.  And software like all corporate assets requires reinvestment if it is to be sustained, for continuous improvement and growth to support your changing business model. This reinvestment is named, "Maintenance".


The IEEE definition of maintenance is as follows [1]: Software maintenance is the process of modifying a software system or component after delivery to correct faults, improve performances or other attributes, or adapt to a changed environment.

What maintenance provides was stated many years ago by Barry Boehm [2]: “... The majority of software costs are incurred during the period after the developed software is accepted. These costs are primarily due to software maintenance, which here refers both to the activities to preserve the software’s existing functionality and performance, and activities to increase its functionality and improve its performance throughout the life cycle.”
  And Boehm could have finished with, “…and compounding the business value of your software – saving you time and money.”

Just as you wouldn’t consider ignoring advances in tools, technologies and software development process (or any other aspect of your business, I’m betting), there are three areas of investment that will help you fine tune the maintenance of the applications that run your business:

        1. Investment in people

        2. Investment in process

        3. Investment in tools

Figure 1 – Three key organizational areas to renovate. 


Investment in People
At the end of the day, your business benefits – or is handicapped by the quality of the work produced by people, your people.  And the quality of the software maintenance performed on your business applications by your people correlates with the caliber of three kinds of knowledge used every by the maintenance staff in your shop:

1. Application Domain Knowledge - or the understanding of how software intersects with your business – to some degree industry vertical knowledge but to a large degree the unique business rules and processes that differentiate your corporation, setting it apart from competitors in the market.  The trustees of application domain knowledge are the business users – and ensuring that the recurrent analysis needs of software maintenance staff for authoritative application domain knowledge are satisfied is a fine-tuning key to success.  

2. Technical Knowledge – the deep mastery of the complete technology environment underlying your application software stack – as well as of the specific maintenance tools and processes employed at your shop should be an obvious factor.  The complexity of production software – especially “Legacy System” code is well known.  But surprisingly, maintenance and support is often seen as a less-than-prestigious assignment, given to those not technically up to doing new development.  This would be a recipe for disaster.  Imagine your most novice programmer, on-call in the data center when a production run ABENDs – and all heck breaks loose.  Providing maintenance teams with premier learning technical learning opportunities is a fine-tuning key to success.


3.  Application Semantics – refers to the custom code that realizes the unique business rules and idiosyncratic in-house processes that keep your company running.  In most shops the original developers of the application have long since moved on or retired.  So, “learning” an application’s “meta-model”, its vocabulary, understanding the coding patterns and application organization, visualizing its data flow and processing paths, being able to predict the state of variable values, etc becomes the first – and usually the largest challenge your maintenance teams face.  And the value of your software investment as it is maintained over time – is determined by the quality and accuracy of understanding of the application semantics.  .  Providing maintenance teams with static and dynamic maintenance tools that simplify this arcane task is a fine-tuning key to success. Ñ

By investing in people, the suggestion is to setup a software maintenance center of excellence around these three areas of knowledge acquisition, formalizing taxonomies of learning, setting measurable goals, objectives, etc.  Some of which would be general to all maintenance, such as ongoing education in z/OS skills and infrastructure and including any specific education on the tools and processes you adopt – others of which would be specific to your business areas and the specific code-base a team supports.  Specific knowledge transfer would include:

·              Periodic meetings with users and Subject Matter Experts (SMEs) to transfer domain knowledge of the system

·              Time set aside specifically for improving application understanding and for building the kinds of procedural models individuals need before diving into any code fix or enhancement.


Is this really necessary?

No – it’s not.  But in that regard, neither should doctors bother to learn about new medical advances, nor should auto mechanics learn how to use today’s computerized diagnostic tools to tune car engines more quickly and accurately, etc. 


As an aside, it is my experience as a z/OS consultant that I’ve never once heard a complaint from some developer about how they’d been over-trained or over-prepared for an assignment or project. To the contrary - what used to be taken for granted in most companies (technical education, built into the work year) has pretty much dissolved into "on the job training" - a euphemism for, "catch as catch can” learning – informal and almost always incomplete.   


Not to put too fine a point on it there is most definitely a technical and potential loss of business income and opportunity when under-trained developers are swinging well-above their software-programming weight class.  Consider anyone from a mechanic to a surgeon.  How comfortable would you be with their work knowing that there are gaps in their knowledge and capabilities?  Put another way, how would you feel about a dental apprentice, just about to do your root canal? 


Investment in Process

The need for a methodology – or even multiple methodologies (one for each of the four distinct categories of maintenance described in table 1 below) that address how to tackle the diverse and complex issues presented by z/OS software maintenance in a systematic way would seem to be obvious.  And yet a formalized process for ‘how to maintain an application at XYZ Corp.” is rarely found – and almost never considered as important as the System Development Life Cycle (SDLC) used to develop new systems. 


Admittedly, you might find documentation; informal “cookbooks” and other collateral based on lessons learned for doing certain aspects of maintenance work (i.e. steps for handling data center ABENDs and on-call situations, etc.)  And if you’re lucky enough to have a true Software Maintenance Methodology for handling Adaptive Maintenance in your shop I congratulate you.  But you’re not in the majority.


Like the applications they support, your shop’s software maintenance methodology would – if it existed - also fall into the category of "Intellectual Capital".  And investment in correcting, adapting and perfecting your maintenance processes – revising best practices, plowing lessons learned back into the software maintenance methodology is a fine-tuning key to success. Ñ


FYI, there is an abundance of industry research, white papers, studies, etc. on maintenance methodologies published and available from such organizations as the IEEE, ACM, etc.  Meaning, if you don’t currently have a maintenance methodology you do not have to start building one from scratch. 


It’s important to point out that we do not advocate any particular methodology, nor do we believe that a rigid and heavy methodology is “the answer” to wrestling software maintenance problems to the mat.  In our experience, if a methodology becomes onerous, or is pushed dogmatically, development teams will either find a way out of using it (and often end up “winging things” – rarely a good situation), or it will become yet another cost issue – not a solution to the problem it was intended to solve. 


What we’re recommending in this paper is that you provide enough systematic structure that developers do not have to make determinations and judgment calls during projects that end up as “wheel spinning” or worse, as strategic mistakes that cost the organization time and money (this is especially for emergency maintenance tasks – or data center “hangs” and bug fixes).


One strong process-specific fine-tuning recommendation is to appoint a single person to be the software maintenance architect at your shop – and task that person with responsibility for establishing organizational controls: developing and refining the maintenance plan, publishing a dictionary of your firm’s software maintenance terms and vocabulary, creating and managing your software maintenance methodology, surveying and analyzing the maintenance problem factors affecting maintenance in your shop (i.e. SME knowledge, maintainability metrics, reconciling problem factors from the user’s versus developer’s perspectives, quality of documentation, programmer effectiveness/motivation/turnover, the age and condition of your business applications, the integrity of production data, machine availability, etc. ), and uncovering and putting into practice worthwhile technical insights, and empirical data points available from journals, conferences, etc.   Ñ


Anecdotally, one shop we are aware of – through the research their Software Maintenance Architect undertook – found that by “batching” adaptive maintenance requests, they ended up improving the efficiency and throughput of their certain projects by almost 20%.  Not a number to sneeze at these days (actually, not a number to sneeze at any days).  A final reason for appointing a single person is that the above-mentioned technical insights and data points are often published in the arcane lexicon of formal research.  Searching for an individual with the right mix of practical production experience and 800 Verbal SAT scores will – our bet, at most net you one individual – guys like this don’t grow on trees.



As fate would have it, the software stacks running on – and attached to your z/OS have evolved and grown in size and complexity just as corporate belt-tightening due to our most recent recession has inversely reduced staff and time to microns of availability.  So from a technology perspective, tools are no longer a Nice-to-Have, they're a necessity. 


Using the same manual processes and approaches throughout the maintenance lifecycle that would have been used 10, 20 or even 30 years ago ensures that things don't get done efficiently - not done to spec – or that they cost more than necessary – perhaps all of the above.  Consider as an example how you would debug a situation where incorrect values showed up in a Java/web page that rendered data from in an IMS database accessed through a CICS transaction using a Web Service.  How to track a problem like this down?  Where even to start?


Fortunately, the software industry has moved beyond TSO-based and manual approaches to maintaining and supporting cross-platform or even complex single-platform applications.  Mature technologies and products in the areas of: 1. application understanding/impact analysis, and 2. Testing and dynamic code analysis, 3. Team collaboration, and 4. source control management have been around and been in wide-spread use.  Such technologies have proven to save companies time, money and ultimately to improve application quality and lower defect rates. Utilizing tools in the above four categories is a fine-tuning key to success. Ñ


Outsourced Maintenance

Adding to the drivers for taking a deeper look at the maintenance being done to the software assets that run your business is the trend over the last ten years towards outsourcing maintenance.  Which adds the additional wrinkles of distance (latency), communication (the lexicon of business is subtly complex – amplifying errors due to misunderstanding and misinterpretation), loss of control (over the maintenance practices of your outsourcer) and is exacerbated by the loss of in-house application/business expertise. 


Overcoming these additional obstacles requires tightening down on project management (through increased project governance – either with the help of more rigorous methodology, specially designed project governance products or probably both), adopting technology that provides pragmatic scoping of maintenance projects, and real-time collaborative team-development functionality, and making all of your remaining staff better at their jobs by adopting the above keys to success.

Why Invest in Fine-Tuning Software Maintenance?

Research shows that the total investment in maintenance activities is between one and nine orders of magnitude more than what is invested in software development and delivery.  [Effective Software Maintenance, Stanislaw Jarzabek, Auerbach Publications].   Even taking the lowest of these estimates, it’s clear that incremental improvements in your shops maintenance organization, its people, processes and tools could easily be the single best financial investment you make.

In this detailed white paper we offer specific recipes for improving or better utilizing Knowledge, Process and Tools that will keep your development firing on all cylinders - as they continue to manage the complexities and continue to do the heavy lifting that maintenance has always been, in I/T.     


Executive Summary – Keys to Success


Software is the foundation of today’s business.  While analysis, design, construction and deployment are all key phases of the development lifecycle, Software Maintenance, which ensues the moment your application goes into production, and continues until it is sunset, is undoubtedly the most important single stage.


A successful application development area needs a dynamic maintenance plan, a dedicated maintenance supervisor and a first-string maintenance team to continually renew and sustain the software, allowing you to exploit its business value, and prevent costly data failures, defects and applications that flat-line in production.



Ñ Endorse the business value of your production software and the maintenance process of that software as a capital asset across your organization


Ñ Employ a software maintenance methodology that takes into consideration the different categories of software maintenance (see next section) – and undergoes periodic refinement and renovation, based on industry research and your in-house “lessons learned”


Ñ Establish a position overseeing software maintenance, with responsibility for the establishment of your shop’s maintenance plan, the progress of maintenance practices, the integration of the maintenance tools and methodologies used – both in house and outsourced


Ñ Ensure that software maintenance staff has easy access to application domain knowledge Subject Matter Experts

Ñ Provide software maintenance personnel with access to premier technical learning


Ñ Install key software maintenance tools for: 1. application understanding/impact analysis, and 2. Dynamic code analysis and testing, 3. Team collaboration, and 4. Source Control Management


Fine Tuning z/OS Maintenance – Deeper Dive

Let’s begin this section by more precisely defining terms and vocabulary.  The Institute of Electrical and Electronics Engineers (IEEE [3]) has defined categories of corrective, adaptive, and perfective and emergency maintenance.






Emergency maintenance: Unscheduled corrective maintenance performed to keep a system operational.”

Corrective maintenance:  Modification of a software product performed after delivery to correct discovered faults.


Adaptive maintenance:  Modification of a software product performed after delivery to keep a computer program usable in a changed or changing environment.



Perfective maintenance: Modification of a software product performed after delivery to improve software quality, performance or maintainability – which is a key factor in maintenance productivity.

Table 1 – the IEEE Software Maintenance Categories


Various other organizations (the International Organization for Standardardization – ISO for example) also have categorized the types of software maintenance companies do – refactoring and extending Table 1’s categories and definitions.  But there is a high degree of overlap among all the experts in this area.


At the risk of breaking the above disclaimer against adopting overly-onerous methodology, you might well consider the IEEE’s list, just a starting point for your own shop’s maintenance research and in-house process. Other experts and practitioners in this field have broken down each of the steps by numerous other taxonomies.  See Table 6 – Preventative Software Maintenance – Classification Factors, extension to IEEE [4] Risto Vehvilainen, for an excellent example extending IEEE’s categorization.   


We continue now, with our People/Process/Tools motif, but we’ll drill down to explore more technical and detailed topics.



Who are the people at your shop who do maintenance?  Through informal research done by IBM for this white paper surveying (a sampling of 36 z/OS shops from the Global 2000), we list the following categories:


Maintenance Team Organization

Used by Percentage of Shops

Developers who wrote the code maintain it (primarily)


Separate maintenance in-house team


Local consultants


Offshore/Outsourced maintenance


Other (various combinations of the above)


Table 2 – Software Maintenance Team Break Down


The first thing you notice when you start adding up the percentages, and slicing the maintenance organizations a different way is that roughly 60% of the world’s software maintenance is done by external development teams – not by a shop’s own technical staff.  Because of this, it might be tempting to declare victory over maintenance, but unfortunately the opposite is true. 


Fine tuning software maintenance in a world where developers who did not write the code, have never even seen the code and your company’s programming nomenclature, may not know your company – possibly do not even understand your vertical market space - which translates into each application’s vocabulary meaningful variable and procedure names, etc. – and don’t have the kinds of working relationships with SMEs and other critical staff, is both harder and a larger imperative.


In light of this, we return to our initial break People break-down, and offer additional keys to maintenance experience success.


1. Application Domain Knowledge - The importance of the business and programmer analyst.  A long time ago there were users – system analysts and coders - and problems.  As the system analysts were overwhelmed in translating business requirements to procedural specification, flow charts, Warnier – Orr diagrams, so they could be precisely understood, and used by the coders.  Net?  System Analysts – and the business-process knowledge they acquired became a development bottleneck. 


The solution that the industry evolved into?  The coders – that stayed at various shops (or within a domain-range of shops – say, insurance, in Hartford, CT) became Programmer/Analysts.  Programmer/Analysts interfaced with users and systems analysts. The bottleneck eased.  And as they accumulated business knowledge, over the years their work became even more valuable and higher-quality.


In many respects, the current situation is analogous to the late 1960’s/early 1970s – with a split between language coders - who can spell PERFORM - and business/system/programmer analysts who understand how a particular division of an insurance company adjudicates claims.  The key to success for your company is to at all costs – nurture and retain those on staff who have deep technical business analysis knowledge – of how your current application stack operates. Ñ


There has been much talk for a long time about how to replace the soon-to-retire aging COBOL population of the world (okay…so this past recession has mitigated this problem to a large extent).  But primarily, the issue is not COBOL programming/technical skills.  The problem is how to know what the business term (and variable name) “coverage-code” means changing data edits, or understanding which production job can be released in the data center if a hang occurs and there are file update and synchronization dependencies, and why (yes, “why”) a particular record value is considered required.  To paraphrase a current credit card advertising campaign, this intersection of business and technical data is: “priceless”. 


For more on the topic of application domain knowledge transfer – and especially the benefits and a structured process, see Pigoski [6], “Software Maintenance Training: Transition Experiences.”



2. Technical Knowledge – Given the above breakdown of “who’s doing the maintenance on your systems” finding ways to certify that the COBOL programmers who are making changes to your general ledger programs got more than 3 weeks of Entry Level Training would be a good start.  A key to success then is, for you to assume responsibility for certifying technical credentials – of course for your in-house staff, but also for any vendor that will end up with long term contracts and where you have already reduced in-house technical and programmer/analyst staff.  Ñ There are industry-supplied tests and certification exams available in most of the z/OS disciplines (example, )


Ensure that your own in-house staff have unfettered access to technical learning content, and expertise – especially any groups dedicated to Emergency Maintenance.  Understand that we are not proposing a limitless budget of expensive vendor-training classes, but that someone – perhaps the Software Maintenance Architect is responsible to establish a set of links to the kinds of quality materials available on the internet; from IBM DeveloperWorks (all the Redbooks, language manuals, etc.) to collections of technical articles:  and so forth.


Finally, it’s important to note that an opportunity for saving time and money exists by educating maintenance developers well, in traditional-z/OS specific areas of highest impact:

·              Internal Data Representation problems – which cause a significant percent of production ABENDs on z/OS

·              Language-dependent issues – such as fall-thru problems when mixing COBOL Sections and Paragraphs, flow-issues, when replacing Next Sentence with Continue, etc.

·              All of the MVS System Completion Codes – their common causes and solutions:

·              Common Business and z/OS Application Problem Areas – Date arithmetic and Y2K-windowing, static versus dynamic calls, etc.

·              The Compiler Options – and their effect on testing, performance and debugging: 

·              Run-Time software issues and/or common “tricky coding” Constructs – DB2 SQL problems mixing AND/OR, Table Joins, etc. IMS problems with traversing Logical Databases, and Secondary index databases, and using multiple PCBs, etc.


It might be mentioned that “deep-dive” education is required here.  One definition of a “Legacy System” is “a system that works”.  Another is “a complex system (that works)”.  Assigning an intern to a production support role for a complex batch system is tantamount to sending a Little League pitcher to relieve a major league ball team, in the 9th inning, with the bases loaded – in the world series.


3.  Application Semantics – the breakdown in table 5 brings this key factor into crystal clear focus:

·              For in-house maintenance done by developers who wrote the code – Application Semantics should not be an issue (and if it is, you have bigger problems than just Software Maintenance)

·              For Separate maintenance teams from within your organization and for Local Consultants – Application Semantics will be a straightforward learn.  It is still recommended to use static and dynamic code analysis tools to simplify and streamline the learning, reduce time and risk and level the playing field (Static and Dynamic Analysis tools can raise deep technical semantics and expose the use of key elements at a much higher-level of abstraction than traditional, manual source-based unstructured search activities.

·              For Offshore/Outsourced maintenance – using static and dynamic analysis tools is critical – and not just by lead project analysts during the “analysis and scoping” phases of a project.  It is critical to have these tools available during technical construction and testing phases of projects.  Consider the following: 

o        You’re making changes to a computation, and all of a sudden it occurs to you that you’re not entirely sure how or where a variable’s value has been derived from the file read to your statement.  What do you do at this point – keep typing?  Of course not.  You drop back into code/program/application analysis in order to refine or complete the mental map of the logic you’re revising, before changing another line of source code. 

o        Or you’re changing a performed (chained) routine and you need to know that fall-through will not occur, or that it will not affect other nested paragraphs or functions. 

o        Suppose you’re testing a module at the source-level, and watching as the lines execute in sequence.  All of a sudden the “FALSE” path of an IF statement is taken when you expected the “TRUE” path. L Do you just ignore the problem (your procedural misunderstanding), and keep going?  Hopefully not.  You (once again) jump back into structural and program flow analysis.

·              Every developer that’s ever modified code has experienced the above.  Without Static and Dynamic Analysis tools you’re back to using manual ISPF find commands and thumbing through program listings – essentially “building a house with nails and hammers instead of power tools”.  Yes – it can be done.  But much more slowly, much more expensively.  And it can only by substantially more expensive veteran developers.  Use the Static and Dynamic Analysis tools to regain control over this key are of Software Maintenance – learning the Application Semantics.



Investment in Process

A typical process-flow has been captured by IEEE in its IEEE-1219 [3] document, and is reproduced below in Figure 2.  It organizes the software maintenance process – or lifecycle in seven steps.  You may recognize some or all of these steps as occurring in your shop’s approach to software.  Or you may be have similar, fewer or more steps depending on the formality of your methodology and how it is applied to by different management teams.


There is much more detail in the IEEE document, such as input and output deliverables, the activities grouped, related and supporting sub-processes, the control, and a set of metrics.  If you haven’t built or acquired your own formal maintenance process, you will be hard-pressed to find a better starting point.



Figure 2 the IEEE Software Maintenance Lifecycle (process)


However, three points on this:

  1. As described earlier, it is the experience of most of the shops we contacted for this research that the needs of each category of maintenance (emergency, corrective, adaptive and perfective) expand out to additional sub-processes for most all of the steps in the IEEE process (see table 2).
  2. Your shop’s own experience, priorities and prerogatives – even approach to implementing methodologies always supersede a generic, one-size-fits-all approach
  3. And as lightly touched on earlier, working towards an iron-clad, deep procedural methodology might be an indulgence that will not return-on-your-investment as anticipated, and must be considered in the context of your shop’s inclination towards: Results-based, versus process-driven activities and project management.


Tools and the Steps in the Process

When considering the required, optional and effective use of tools throughout the Software Maintenance Lifecycle, consider that there are two categories:


1.                          Tools that bring value to each and every step – these include Cross-platform collaboration products and source control management tools

2.                          Tools that have step-specific advantages – notably the static and dynamic analysis tools we will dive into, shortly


Your shop might create its own maintenance-tools-process-driven matrix, which will change over time, as new technologies and new problem domains appear on the landscape, but a typical correlation of Software Maintenance Lifecycle phases and tools that help fine-tune your efforts to save time and money is shown below in Table 3.  In the next section we drill-down on the different tools and functionality and features – as applied to Software Maintenance.





Tools Used at Specific Lifecycle Steps

Tools Used Throughout the Lifecycle

Problem/modification identification, classification, and prioritization.

Dynamic Testing tools – for initial breakdown of emergency and corrective maintenance problem definition

Team collaboration


Source Code Management

Project Scoping


Dynamic Testing  and Static source code analysis tools – for trustworthy/accurate results and efficient/cost-effective analysis activities





Static source code analysis tools – for solving  inconsistencies in developer understanding

Regression testing

System testing.

1. Dynamic Testing tools for revealing problems during test & debugging.


2. Static source code analysis tools for solving problems, stemming from an inconsistent developer understanding (which – if you think about it, is the source of the preponderance of software problems)

Acceptance testing.

Dynamic Testing tools



Table 3 – Typical lifecycle phases and tool usage correlation




So – the first consideration that must be addressed in this section is “why tools?” – and the second is “which tools?” 


For decades software developers thumbed through listings using paper clips and sticky pads when they needed to bookmark variable references and ran out of fingers, or on z/OS they used ISPF search operations, and eventually code went into production – and worked.  Let’s discuss the value proposition for using the four categories of tools shown in Table 3 – and discuss the must-have features that offer enough value for you to invest and use them.




Source Code Management

This is the easy sub-category.  Given the size, complexity and dynamic nature of modern z/OS development scenarios – especially with the introduction of offshore development, it is simply axiomatic that without a robust Source Code Management (SCM) product that automates the version control process, and offers revision management, file locking, the ability to return to previous versions (by stripping out deltas) and version merging your development and maintenance activities would quickly (within months) grind to a halt.


Not only is it common for multiple versions of the same software to be deployed at different sites, but for bug fixing and adaptive maintenance automating the control of access to source versions is a long-established best-practice, and doubtless you not only have Source Code Management up and running at your shop, but you’ve customized the hell out of it by now, as well.


Team Collaboration

Unlike Source Control Management you may not have been introduced yet, to


Team Collaboration software, which is essentially a software dashboard that:

·              Allows project leaders to define, organize, and track and your software maintenance, support and development teams and their projects (not program source, people and processes)

·              Provides simplified interaction and communications between Subject Matter Experts (SMEs) and the technical programmer/analysts and architects who interact with them – this interaction can be captured electronically for reuse, refinement and documentation.

·              Accommodates Management reporting and governance of project milestones and deliverables

·              Enables project technical developers to share development artifacts (source and model-based) in-context of doing project work


By raising the team concept to the level of an automated entity and providing integrated workflow among all stakeholders, software maintenance projects (across all four categories) are freed (to degrees) from repetitive, manual tasks, and allowed to focus on business and technical problems at hand.  At the risk of over-simplifying, you can think of Team Collaboration as reducing the costs of administrivia and bureaucracy, which can be a significant cost and barrier to productivity enhancements in an organization.  Speaking personally, and as someone who spent the better part of a decade enmeshed in software maintenance projects at various Global 2000 shops, it is hard to argue against reducing mind-numbing regimentation, in all its semblances.


Finally, revisiting our previous points on challenges introduced by offshore maintenance projects, Team Collaboration was born to solve the latency, communications and logistics issues that are part and parcel of half-a-world-away shared project efforts (or even half-a-block away, the point being that distributed development vs. central development brings organizational challenges that must be addressed, to gain the understood benefits).


Static and Dynamic Code Analysis Tools

Analysis tools (both static and dynamic) assist developers in “gaining intellectual control” [Larry England, IBM, Santa Theresa Labs] over an application. 


Research has shown that there is a diseconomy of scale in productivity when maintaining large systems.  This is due to the large number (often in the hundreds to thousands) of variables-and-functions-of-interest scattered throughout large amounts (often in the hundreds of thousands to millions of lines) of software.


Historically only practiced veteran z/OS developers – those with over 5 years of production “combat experience” are capable of building precise, clean and faultless internal/mental maps (thereby gaining “intellectual control”) of the code execution paths, key variables, their state transitions, etc. 


Over the past 20 years many of these individuals have retired or moved on to management positions, and now many maintenance teams are in the untenable position of using relatively inexperienced developers to make technical judgment calls on changes to core systems – systems that have only gotten older, more complex, brittle and (net) more difficult to fathom than ever before.  This represents a remarkable level of risk.  While all of the above-mentioned Keys to Success are cumulatively helpful, electronic automated analysis software consolidates and amplifies your savings of time and money on an order of magnitude.


Static Code Analysis

One of the unintended consequences of Y2K was the birth of Static Code Analysis tools.  These were successfully used in Y2K Adaptive Maintenance, as January 1st 2000 came and went with virtually no apocalyptic z/OS melt-downs.  And it’s appropriate to point out that, every category of Software Maintenance is related to some aspect of the Y2K problem.  Ergo – if the use of Static Analysis tools was the central key to success for finding and fixing century date routines strewn throughout z/OS applications, then it stands to reason that applying them to corrective, adaptive, preventive and perfective maintenance makes business sense.  And passing up the opportunity to employ them means making up for their automated and comprehensive features with additional time (a not insignificant percentage increase), risk (through an incomplete of faulty analysis) and the money associated with additional time and risk.  So what is it that Static Code Analysis tools bring to the table?



The information needs of maintenance and support staff vary by the type of application learning/comprehension model employed.  It is commonly understood that you “learn” a program either top-down (by studying documentation, talking with SMEs, and ranging over the functions an procedures stating from the entry point – one logical branch at time, or bottom-up, starting with a specific function, record or variable – often working from program listing cross-references, and tracing backwards and upwards.  By the way, if this sounds like a meticulous, difficult, error-prone grind?  Too right.  It is, and it’s also the nucleus of software maintenance project.  In reality – for projects or systems of significance programmers switch between the two approaches, integrating information into a final mental model of the application’s execution – and how it relates to the business or situation at hand [7]. 


Tools then, that comprise features automating or simplifying analysis top-down and bottom-up will fast-track your teams to improved productivity and lowered risk.  Such tools should (at a minimum) contain the functionality described in Table 4  – and come with a graphical (mouse-based) IDE – largely because the task of understanding code is keyboard-less, and driving through diagrams and hyper-links with a mouse instead of typing FIND commands into green-screen panels is actually a critical element of the usage model. 


The application semantics – or “metadata” (i.e. information about information) surfaced by static analysis should be contained in an easily search-able dictionary or database.  The tool’s metadata,  extracted and exposed as structured top-down diagrams and bottom up element-level reports – cross-referenced and hyper-linked must (must) be built from the source code itself – in order to provide the maximum semantic integrity.


It should be pointed out that the list of requirements in Table 4 is by no means exhaustive.  A good Static Analysis tool can assist in answering the majority of Where/What/When/How – and especially “Why” questions that forms the provenance of Software Maintenance projects.





IBM Offering

Top-down systems views

Iconic/Graphical images – and detailed reports, that correlate to system flow charts

Rational Asset Analyzer

Hyper-linked transaction and call/calling program views

Execution chains abstracted up to a graphical level with accompanying detailed report analysis. Hyper-linked for top-down study.

Rational Asset Analyzer

Hyper-linked batch job views

Individual steps in a job (“run-unit”), graphically exposed and hyper-linked, with sub-elements that show (at a minimum) programs, utilities, external files, etc. used by each job step. Accompanying detailed report analysis. All hyper-linked for top-down study.

Rational Asset Analyzer

Hyper-linked Data Flow Diagrams

Hyper-linked graphical depiction of file usage throughout a batch job or transaction

Rational Asset Analyzer

Program or Module “Functional Decomposition”

Different views or slices into programs breaking the

Rational Asset Analyzer
Rational Developer for System z

Impact Analysis

“What If” analysis, before source modifications are made – at all levels of scope: Program, Run-Unit, Transaction, System-wide

Rational Asset Analyzer

“Where Used” and “How Used” Analysis

Ability to isolate variables and reveal only: statements that change the variable value, statements that test or reference the variable value, declarations, etc.

Rational Asset Analyzer

Rational Developer for System z

N-way search capabilities

Search in meta-model for elements

Search in source for elements and element subset

Rational Asset Analyzer

Rational Developer for System z

Customized code windows

The ability to create on-the-fly combinations of editable windows in order to provide the best source-level perspective an area of code

Rational Developer for System z

Subset capabilities

Ability to group (subset)

Rational Asset Analyzer

Rational Developer for System z

One-click Access declaration of variable

Ability for the tools to “keep up with the developers exploratory process”

Rational Developer for System z

Expand/Collapse code sections

Isolate to manage-able code blocks

Rational Developer for System z

Extensible Query Language

The ability to compose completely-custom queries against the meta data in the tool’s dictionary or database

Rational Asset Analyzer

Integration with the “Development Workbench”

The ability to use the results of all of the above analysis reporting directly inside the edit/compile/debug tools

Rational Asset Analyzer

Rational Developer for System z

Code Quality Metrics or development “Best Practices”

Show range violations (bad GO TO structures), recursion detection.  List inefficient variable declarations, etc.

Rational Asset Analyzer

Application Performance Analyzer

Perfective Maintenance

Finding dead code, unused copybooks, etc

Rational Asset Analyzer

Support Emergency Corrective Maintenance analysis

Show “where modified” code to trace down root-causes of S0C7 and S0C4, etc

Rational Asset Analyzer

Code slicing and Business Rules Mining

Separating business logic from persistence and presentation logic

Rational Asset Analyzer

Table 4 – Static Code Analysis – a List of Recommended Features



Dynamic Code Analysis

While Static Code analysis will be an indispensable aid in helping to solve many classes of Software Maintenance, for certain problems – especially some in the Emergency-Corrective Maintenance category you will need another view of the production application – a view that is provided by actually running the live code – at the source level and following the execution sequence.  This is called Dynamic Code Analysis.


Dynamic Code Analysis tools allow you to inspect – at the detail level – and without conjecture:

·              The instruction sequence – every statement “live”, stepped through and animated running against the software stack that is the deployment platform

·              Variable values – in program storage, file buffers, database records temp storage, anything the code has addressability to


This type of real-life monitoring – at the source code level (which is the logical currency developers trade in) is absolutely unrivaled, for solving some of the most obstinate and bewildering software analysis problems – and is invaluable for corrective maintenance in every form.


Table 7 lists some of the more prominent and useful Dynamic Code Analysis features.  Like source code management, it is likely that you already own a product that can be used for Dynamic Code Analysis.  But are you taking advantage of these tools for Dynamic Code analysis (and not just debugging)?  The table’s Feature/Function column lists some of the ways that Dynamic Code analysis can provide teams with automated-assistance, in areas that are traditionally labor intensive (and error-prone, due to the size and scale of complexity as concerns z/OS Legacy Systems).




Software is like other knowledge-based R&D products, an intangible corporate asset.  The value of intangibles is based on the income they are expected to generate in the future. Alfred P. Sloan (long-time president and chairman of G.M.) once said, "Take my assets - but leave me my organization and in five years I'll have it all back."   We could paraphrase this as, “Take my book-value assets – but leave me my software and in 5 years I’ll have it all back.”  That of course, pre-supposes you are maintaining your software, over the 5 years – in a quality manner.

It is naïve to think that organizations have buckets of money to throw at the dilemma of software maintenance, however in this white paper we have attempted to provide solutions that incorporate:

·              Internal process reengineering

·              Web-based knowledge acquisition

·              Modest investment – or repurposing of development software with definitive maintenance value

By implementing one or all of these solutions, we are confident that you will see:

·              Lowered defects – which lowers costs, and raises the overall quality of your data center operations

·              Improved productivity – that can helps balance the demand for changes and queued enhancements with new systems functionality

·              Savings in hardware and software costs – through more effective and productive developer usage models

·              An increase in the shelf-life (and respective business value) of your z/OS applications – a business value which unlike other intangibles grows over time

But at the very least, by having read this far, you’re already more tuned in, and hopefully a little wiser for having spent the ten minutes or so.  Net: Don’t’ let your z/OS legacy applications flat-line.  There’s no “Dr. House” waiting ambling about with last-second miracle cures.  The good news?  It doesn’t require a diagnostic genius to make things better - just someone who appreciates the value of “systems that work”.

Jon Sayles, October 2009

Thanks to the following IBM professionals for the help they gave in reviewing this: Larry England/IBM, Dave Myers/IBM, Saleem Padani/IBM, Tom Ross/IBM





IBM Offering

Dynamic, source-level step-by-step animation

Iconic/Graphical images – and detailed reports, that correlate to system flow charts

Debug Tool

Present variable values:

  - Characters

  - Hexadecimal

Execution chains abstracted up to a graphical level with accompanying detailed report analysis. Hyper-linked for top-down study.

Debug Tool

Interrupt execution based on convenient set of options:

z/OS applications are almost always too large to step through from their beginning.  Various break points must be set in order to make the tool usage practical:  Entry to module, value change, conditional value, etc.

Debug Tool

Selective subset of Variables to “monitor”

z/OS applications have far too many variables to monitor all at once, and viewing just the variables on the current line is too restrictive.  You must be able to make persistent lists of variables for monitoring

Debug Tool

“What if” Analysis

Besides passively viewing the code execution flow and variable values at run-time, developers often need to modify storage values, alter the instruction sequence, iterate over  complex logic, in order to really understand the algorithm or code pattern

Debug Tool

Access to z/OS Storage and Registers

For debugging assembler programs, or for analyzing a failed instruction at the machine code level

Debug Tool

GUI Integration with the development Workbench

Far from cosmetic, graphical, integrated Workbenches that include Dynamic and Static Analysis tools are simply expected.  To understand this, consider how often you open DOS prompts and type: 

copy c:\directory1\subdirectory1\myfile.dat   to d:\directory1\subdirectory1\myfile.dat these days?  Or do you usually Drag & Drop files?  J

Debug Tool and Rational Developer for System z

List programs called at run-time, during a specific transaction

When learning an application (adaptive maintenance) or studying the root cause of an ABEND situation (corrective maintenance) it is acutely important to understand the precise execution route

Debug Tool

CICS Interdependency Analyzer

The sequence of transactions within an application

When learning an application or studying the root cause of an ABEND situation it is acutely important to understand the precise execution route

Debug Tool

CICS Interdependency Analyzer

List files and databases (Read or Updated) used at run-time during a transaction

Following the chain of input/edited/output files as they are read from/written to throughout a transaction or batch application is one of the most common maintenance strategies.  Simplifying/Automating this process alone can be worth the price of any product

Debug Tool

CICS Interdependency Analyzer

For Emergency (ABEND) Situations, provide initial debugging data points

Integrated “fault analysis” where valuable research information, such the exact statement in a program that caused an ABEND (and surrounding code) plus the values in the variables under consideration can be critical in timely diagnosis and correction of the problem

Debug Tool + Fault Analyzer

List program lines not tested – as a result of Regression Data

A “test coverage” utility, that shows marked source of lines tested – and lines untested

Debug Tool

Integration with static code analysis

The ability to research problems alongside their discovery promotes productivity, cuts down on errors, etc

Rational Asset Analyzer

Table 5 – Dynamic Code Analysis – a List of Recommended Features






-------------------Software Categories – IEEE Definitions------------------------







Corrective Maintenance

Adaptive Maintenance

Perfective Maintenance

Emergency  Maintenance






Periodically reviewed and revised?


Project teams trained in its use?


Improved by integrating industry research and best practices?

The steps and use of tools, in-house expertise, collaborative methods, etc. for production bug-fixes

The steps and use of tools, in-house expertise, collaborative methods, etc for adding code to production assets.

The steps and use of tools, in-house expertise, collaborative methods, etc. for measuring performance, Areas to touch within an application and to avoid, etc.

Includes sections on optimal use of tools, in-house expertise, collaborative methods, etc. for emergency bug-fixes, ABENDs, production problems, etc.

Software Maintenance Tools

Static Code Analysis

Tools available that cover your software stack requirements?


Teams trained in the use of tools?

Automated analysis of application semantics

Automated analysis of application semantics

Queries that create reports highlighting potential candidates for performance improvement

Steps and examples for use in production ABENDs – and in different post-production bug fixes?

Dynamic Code Analysis

Tools available that cover your software stack requirements?


Teams trained in the use of tools?

Tracing code, testing, and learning application semantics

Tracing code, testing, and learning application semantics


Understanding “WHAT” and “HOW” of ABENDs

Collaborative software version control

Tools available.


If outsourced maintenance, very useful to overcome challenges of communications, time, and distance.


Teams trained in the use of tools?

Real-time fault analysis sharing



Project tracking, and sharing of design documentation and user communication

Project tracking, and sharing of design documentation and user communication

Real-time fault analysis information sharing

Software Maintenance Teams


Methodology Training

Technical Training


Understanding of Tools and process

Understanding of Tools and process

Understanding of Tools and process

Understanding of Tools and process

Technical (language and platform)  training

Access to internet-based technical consumables: Redbooks, language documentation, white papers, Q&A forums,  etc.

Easy look-up of language elements and analysis meta-data – to support Application Domain understanding

Easy look-up of language elements and analysis meta-data – to support Application Domain understanding

Access to IBM Redbooks, conference slides, etc. on technology optimization

Understanding of MVS System Completion Codes and common causes

Table 7 – Keys to success, over the cross-section of different categories of Software Maintenance







Preventative Maintenance


1.        Fault correction

2.        Small change

3.        Large change

4.        Maintenance Project

5.        Advise

6.        Study

An important instruction and therefore needed

Study and maintenance project


1.        Immediately

2.        Soon

3.        Planned schedule

4.        On suitable situation

An important instruction and therefore needed

Planned schedule


1.        Compulsory

2.        Compulsory but with a flexible schedule

3.        Optional

An rather important instruction and therefore needed



1.        Code

2.        Specifications

3.        Procedures

4.        Instructions

5.        Descriptions

An important instruction and therefore needed



1.        Very small, not registered as a separate request

2.        Small registered as a separate request

3.        Moderate

4.        Large


At least moderate

Influence on software

1.        Corrective

2.        Changing functionality

3.        Changing capacity or performance

4.        Improving quality

5.        Applying to new environment

Interesting as additional

Information from a reporting point of view



Reason for the change

1.        Error: software doesn’t work according to the specifications

2.        Need to improve software cost/benefit analysis

3.        Changes in the business of the client

4.        Need to change technological environment

Interesting as additional

Information from a reporting point of view.


Anything except error

Initiator of the change

1.        User

2.        User liaison

3.        IT management

4.        Maintenance

5.        Other IT person

Interesting as additional

Information from a reporting point of view. Can be concluded

from other factors.

IT management or maintainer

Role of the maintainer

1.        Reactive

2.        Proactive

3.        Co-operative

Interesting as additional

Information from a reporting point of view. Can be concluded from other factors.


Planning Scope

1.        Immediate

2.        Short range (a few months)

3.        Year(s)

Not very interesting. Can be concluded from other factors.


Preparation before implementation

1.        Not needed

2.        On request level

3.        On application level

Not very interesting. Can be concluded from other factors.

On application level


1.        Minor

2.        Moderate

3.        Major

Useful as additional information

At least moderate risks

Importance for Business

1.        No special importance

2.        A moderate importance

3.        Critical

4.        Strategic

Useful as additional information

At least moderate importance

Table 6 – Preventative Software Maintenance – Classification Factors, extension to IEEE [4] Risto Vehvilainen.


[1] IEEE Std. 610.12, “Standard Glossary of Software Engineering Terminology”, IEEE

Computer Society Press, Los Alamitos, CA, 1990.


[2] Barry W. Boehm, Software Engineering Economics, Prentice Hall PTR, Upper Saddle River, NJ, 1981


[3] IEEE Std. 1219-1998, “Standard for Software Maintenance”, IEEE Computer

Society Press, Los Alamitos, CA, 1998.


[4] Risto Vehvilainen, "What Is Preventive Software Maintenance?," icsm, pp.18, 16th IEEE International Conference on Software Maintenance (ICSM'00), 2000)


[5] “Effort Estimation for Corrective Software Maintenance”, SEKE '02, July 15-19, 2002, Ischia, Italy. Copyright 2002 ACM 1-58113-556-4/02/0700


[6] “Software Maintenance Training: Transition Experiences”, Thomas M. Pigoski and C. Steve Looney. 1063-6773/93 1993 IEEE


[7] “Programming Understanding Needs During Corrective Maintenance of Large Scale Software”, Anneliese von Mayrhause and A. Marie Vans. 0730-3157/97 1997 IEEE