Opportunistic Software System Development: Making Systems from What’s Available
Background and the relevance of the theme
Users' demands for more functionality in software are increasing exponentially, and software development costs are continually spiraling out of control. This situation is driving the software development industry toward composition of systems from readily available software components, including other systems. Systematic software re-use started this trend in the early 1970’s, followed in the 1990’s by increased reliance on COTS (commercial off-the-shelf) products. The trend continues today with SOA (Service- Oriented Architecture), despite all the difficulties encountered in each incarnation. If you ask any audience of managers and engineers today whether they’ve experienced these difficulties, the vast majority will have stories to tell that confirm the challenges, including severe time, quality and user demands. Some practitioners have found ways to be successful using these and other paradigms. But we must continue to seek out new ways of successfully combining existing software into new systems.
This situation has been highlighted in such recent publications as Barry Boehm’s “A View of 20th and 21st Century Software Engineering” at ICSE 2006, Noble and Biddle’s Postmodern Programming publication, and the SEI’s Ultra-Large-Scale Systems study. Yet our universities, practitioner books and industrial training organizations continue to put too much emphasis on how to create masterpieces of code from blank sheets of paper and largely ignore the very different skills required to integrate and test software they did not create and do not control.
This special issue proposes to address this gap between modern demand and current approaches by articulating the current and future situation and describing paradigms and practices that point the way forward.
We live in an era in which there is pressure to make systems development cycle times shorter and shorter while providing more and more ambitious functionality. The Opportunistic Software System Development approach is a response to it. This approach, however, poses unique challenges to developers, development processes, requirements engineering, system architecture, maintenance and evolution, on how to deliver the right functionality with a minimal design and development. To cope with these challenges, creative, innovative, and imaginative engineering and management practices are required.
In the Opportunistic Software System Development approach, developers meld together unrelated software components and systems – components and systems that were not designed to work together. Typically there is less development of software functionality to and more development of “glue code” and performing minimal necessary alterations and adaptations of various software pieces to meet particular requirements. The Opportunistic Software System Development approach is akin to the recent television shows in the UK (Scrapheap Challenge) and US (Junkyard Wars) where competing teams were given a capability they must fulfill using only what is available at a junkyard/scrapheap, a set of tools for integrating the pieces, and their own wits and innovation. This is the New Junkyard Wars: making new capabilities out of useful software that is at hand, applying innovative engineering and management techniques and often using an item in a way that was never envisioned by its creator. It is advancing the state of the practice by building on what exists rather than recreating everything every time.
Although from a distance, this approach may look like the traditional software re-use paradigm, there are some key differences. Software or systems that are used in systematic re-use paradigms are designed to be reused in a specific context. The development processes are systematic and planned. There are clear methods and guidelines on how to perform and manage each development task, yielding a degree of predictability for the result. Even in some COTS-based system developments – those that are centered on a single COTS product or product suite – the majority of the work involves tailoring the system with mechanisms that have been built into the product for just that purpose.
In COTS-based development there is typically a formal selection process, based on fully crafted requirements; in Opportunistic Development, potential components – even if they are COTS products – are found through expeditions into what is available. The careful selection process for COTS-based development means that once a selection is made, the system depends on that product; in OSSD, if one selection doesn’t work out, you just try another. In OSSD, there is less interaction with the supplier, if you know who the vendor is. COTS-based development is used in substantial system developments; OSSD may be most applicable for smaller or prototype systems, which can later be “hardened” for maintainability over a long lifetime.
The key difference between these traditional reuse paradigms and the theme proposed here is that, in the opportunistic development paradigm, the focus is on combining systems that were never meant to work together or even to be reused. It is about taking existing software systems and performing minimal coding with the emphasis on gluing them together, often without interaction with the original developers or access to source code, in order to achieve desired functionality with a desired look. The selection of candidate software is more experimental and prototypical; new kinds of collaboration and engineering are required.
The opportunistic development is about taking existing software systems and performing minimal coding to them necessary to give the desired custom look with the emphasis on gluing them together without any interactions with the original developers. The coding that is done does not deliver functionality but enables it. It is about re-use at systems level rather than at sub-routine and at the same time its not COTS-based development since the identity of the COTS product is removed.
The opportunistic paradigm is a major shift in which the structural bias of current paradigms and development processes are challenged. It requires a major change of mindset from designing and writing original program code that meets a particular requirement to a world in which there are currently few rules for design, construction, or process management. The development process is ad hoc and opportunistic, with the team continuously scavenging and trying out candidate solutions while constantly paying attention to architectural mismatches, programming languages, uniform ways of describing data, etc. The major development emphasis is on smart engineering, creativity, innovation and the most imaginative ways of gluing together seemingly unrelated and incompatible software pieces to provide interoperable and maintainable systems that meet users’ needs. It’s about fabricating hybrid systems by scavenging for functionality wherever you can find it to ‘weld’ together the pieces into a composite system.
This paradigm will require a new set of integration and testing skills, strategies for discovery of software and systems for composition, and engineering and management practices; the definition of new frameworks and infrastructures to handle the complexities; codification of concepts, techniques, and tools; and flexible development methods and processes. Requirements engineers, developers, and system integrators are challenged to come up with new techniques for success in this environment. In opportunistic development, there are no theories or recipes to follow today nor are there any prescribed ways of doing things. The purpose of this special issue will be to gather together the most effective practices that are known today and to point the way to more work that needs to be done to make Opportunistic Software System Development more accessible to all practitioners. We will also solicit contributions from those who have failed in their attempts to use Opportunistic Software Development processes.
Call for Articles
Users’ demands for more functionality in software systems are increasing exponentially, and software development costs are continually spiraling out of control. This situation is driving the software development industry towards composition of systems from readily available software components, including other systems. Systematic software re-use started this trend in the early 1970’s, followed in the 1990’s by increased reliance on COTS (commercial off-the-shelf) products. The trend continues today with SOA (Service-Oriented Architecture), despite all the difficulties encountered in each incarnation. In order to make system development more cost-efficient and timely, we continue to seek out new ways of combining existing software into new systems, and many practitioners have found ways to be successful using these and other paradigms.
The Opportunistic Software System Development (OSSD) paradigm that is the subject of this special issue focuses on gluing together pieces of software that were never designed to work together: opportunistic systems are not composed of neatly designed homogeneous software pieces with clear boundaries (as in software product lines). In this approach, there are currently few rules for design, construction, or process management. Today this development process is ad hoc and opportunistic, with the developers continuously scavenging and trying out candidate solutions while constantly paying attention to architectural mismatches, programming languages, and uniform ways of describing data. The major development emphasis is on smart engineering, creativity, innovation, and the most imaginative ways of gluing together seemingly unrelated or incompatible software pieces to provide interoperable and maintainable systems that meet users’ needs. We want to make new capabilities out of useful software that is at hand, applying innovative engineering and management techniques and often using an item in a way that was never envisioned by its creator. OSSD is advancing the state of the practice by building on what exists rather than recreating everything every time. This is a major shift in which the structural bias of current paradigms and development processes is challenged.
In COTS-based development there is typically a formal selection process, based on fully crafted requirements; in Opportunistic Development, potential components – even if they are COTS products – are found through expeditions into what is available. The careful selection process for COTS-based development means that once a selection is made, the system depends on that product; in OSSD, if one selection doesn’t work out, you just try another. In OSSD, there is less interaction with the supplier, if you know who the vendor is. COTS-based development is used in substantial system developments; OSSD may be most applicable for smaller or prototype systems, which can later be “hardened” for maintainability over a long lifetime.
In the opportunistic development paradigm, creativity, ingenuity and innovation are the keys to success. It’s about creatively picking and mixing the right combination of software pieces. It’s an opportunistic outlook in which every piece of software is a potential solution to any requirement, the only limits being the creativity and imagination of the system developers; requirements emerge as the development goes along; the processes are adapted to the circumstances; it’s about achieving the ‘mission’ goals in whatever way one can using whatever software functionality can be found anywhere. The emergence of the opportunistic paradigm poses real challenges for today’s developers. It demands wits, ingenuity, smart engineering, and the most imaginative use of what is available. New sets of skills, tools, and techniques coupled with imagination and innovation need to be assembled and codified to help practitioners cope with these new demands.
This IEEE Software special issue therefore seeks to provide an account of the state-of-the-practice in Opportunistic Software System Development: how systems are currently developed in such chaotic environments, why it all works when it does, accounts of strategies used, case studies, lessons learned, experiences in successfully assembling functioning systems using existing software pieces. More specifically, the main goal of this special issue is to broaden the software development community’s knowledge of opportunistic development and to publicise the issues that must be addressed to advance the opportunistic software development vision and to highlight promising future directions.
We solicit contributions from practitioners and academic researchers in a wide range of topics concerned with opportunistic systems development. The contributions should describe new concepts, necessary development skills, development, maintenance and evolution techniques, case studies, suitable architecture styles, gluing strategies, design issues, requirements engineering practices, experiences, and tools to support the challenges associated with fabricating such systems. We welcome both supporting and opposing views.
The submissions should address one or more of the following research questions:
- What are the skills needed in such an opportunistic environment?
- What are the conditions under which composing systems in an opportunistic way are appropriate? What kinds of systems or applications are suitable for this type of development?
- How do we depict and evolve requirements for opportunistic systems?
- How and where can we find the pieces?
- What is the best way to "glue" together the parts? How can we deal with the unintended consequences of gluing together pieces that were never meant to work together?
- How does the opportunistic development paradigm affect how we perform test, evaluation, verification, and validation? How can we ensure the quality of opportunistic systems?
- How do we handle the evolution of the composite system and its parts?
- How is management of such development projects different? How do we address the need for collaborative approaches?
- How does this new paradigm affect our educational system? What are the skills we need to teach?
- What are the impediments? Can opportunistic development always – or ever – be successful?
The main goal of this special issue is to broaden the software community’s knowledge of the opportunistic style of systems development and to publicise core and critical issues that must be addressed. Topics of interest include but are not limited to:
- State-of-the-practice of opportunistic development
- Real-world case studies, examples, and experience reports
- Lessons learned – how-not-to as well as how-to
- Maintenance, evolution, and sustainability of opportunistic systems
- Limitations and pitfalls of the opportunistic development paradigm
- New lifecycle and processes for opportunistic development
- Architecture styles and dealing with mismatches
- Smart engineering discipline
- Opportunistic development approaches and life cycles
- Practical advice and proven techniques for opportunistic development
- Strategies for finding and skills for scavenging for software pieces
- Find-and-glue strategies
- Requirements engineering for opportunistic systems
- Composition and integration strategies
- Emergent behaviour and properties and dealing with unintended consequences
- Creative use of mixed systems
- Tools and machinery for fabricating systems
- Validation, verification, and testing strategies and scenarios
- Industrial experiences in opportunistic development
- Challenges and new ideas for the opportunistic style of systems development
- Required combination of team skills
- Innovation and creativity
- Overcoming technical and organizational barriers to opportunistic development
- Failure prediction, reliability and dependability of opportunistic systems
Manuscripts must not exceed 5,400 words including figures and tables, which account for 200 words each. Submissions exceeding these limits may be rejected without review. All contributions that are within the special issue theme and scope will be peer-reviewed and will be subject to editing for magazine style, clarity, organization and space. We also reserve the right to edit the title of all submissions.
For author guidelines and submission details, please visit Computer Society Author Center
or contact the publications coordinator.
Submit your article via the Computer Society's Electronic Submission System, and specify that you are submitting
it for the "Opportunistic Software Systems Development" special issue.
Submissions are due March 21, 2008
Contacts:
For further information please contact one of the Guest Editors:
Cornelius Ncube – Bournemouth University, UK
cncube@bournemouth.ac.uk
Patricia Oberndorf - Carnegie Mellon Software Engineering Institute
po@sei.cmu.edu
Anatol W. Kark - National Research Council, Canada
Anatol.Kark@nrc-cnrc.gc.ca