Law Technology Today From the ABA Legal Technology Resource Center
Related Articles
How can counsel effectively asses the ability of a document review operation to deliver a competent review that is cost effective, timely, and accurate?
Most attorneys would agree on what they expect when it comes to document review. There is, however, much less certainty around how to effectively assess the ability of the document review operation to meet these requirements. Of course, references, employee qualifications, reputation, and relevant experience are all critical to the evaluation process when using an external provider. And, even with a well-developed internal operation, can one be confident that the complex, high‑volume matters that have become more and more prevalent can be properly handled? How can one be sure that past successes can be replicated on future matters in the constantly evolving environment of eDiscovery? The Capability Maturity Model may provide the needed framework for doing so.
Document Review has become—particularly with the advent of electronic discovery—a system of diverse and complex functions that requires mastery of:
- Technology: the management and optimization of the review platform capabilities, the appropriate use of technology assisted review, and strong data (and physical) security controls.
- Process Discipline: the right time/right place deployment of various applications and processes, robust policies and procedures, and the internal controls necessary to execute those policies and procedures consistently and effectively.
- Intellectual Capital: effective knowledge transfer; knowledge retention; learning, training, technological, statistical, linguistic, and legal acumen. And, of course, document review expertise, which is to say: the ability to quickly and accurately identify not just relevance and privilege, but the myriad coding requirements associated with a review.
There are a lot of moving parts, and as scale and complexity rise, the need for a well-functioning system rises exponentially. In other words, the value of organizational competence as denoted by process maturity far exceeds the value of the component parts (people, process, technology). In short, the best technology (if one exists) and the “best” people may be utilized, but if the process maturity within which they operate is deficient, the result will be sub-optimal. It is process maturity that provides the framework in which the contribution of people, process, and technology can excel, or be inhibited. As Peter Senge notes in The Fifth Discipline, “Organizations break down, despite individual brilliance and innovative products, because they are unable to pull their diverse functions and talents into a productive whole.”
And while it may be difficult—if not impossible—to assess the performance levels of the individual components a priori (independent of actual experience), process maturity can function as a leading indicator that allows one to gauge the level of competence, the likelihood being that the service provider can indeed perform a document review at the standards necessary.
The Capability Maturity Model (“CMM”) was developed to do precisely this, albeit in a different context, but its core principles can be adapted to assess organizational competence in a variety of settings.
It may therefore be beneficial to adopt the CMM to assess the ability of a document review provider to deliver an effective and efficient review in a more meaningful way than is currently conceived by the majority of RFPs and other pre-engagement vetting practices.
The Capability Maturity Model was developed by the Carnegie Mellon Software Engineering Institute in the late 1980s. The research was funded by the Department of Defense, which wanted an objective standard to determine an organization’s ability to deliver to specification on a software development project. The “maturity level” refers to process maturity, and while originally developed to assess the ability of a contractor to deliver a software development project, the model has been adopted in other areas as well.
There are five defined levels of process maturity.
Level 1: Initial
This “initial” stage in organizational maturity is defined as having ad hoc processes. The organization is largely reactive, lacks systematic controls, and process outcomes are, in many ways, unpredictable. Ad hoc processes in document review result in a great deal of variation in outcomes from matter to matter (even within a single matter that spans long periods of time and is subject to the disbandment and reassembling of review teams). With ad hoc systems, past performance is never predictive of future success.
Consider that as data volumes grow, and as review and matter complexity increase, significant issues develop if processes remain ad hoc and reactive. The lack of established processes becomes a significantly limiting constraint. In the days of banker boxes stacked along conference room walls, systemic controls were perhaps less critical. Bright, capable associates could muscle through. But as electronically stored information (“ESI”) ballooned, the sheer scale alone demands something more.
Level 2: Repeatable
An organization functioning at a level-two maturity is still often reactive on an organizational level, but has developed some level of project-based processes. In terms of document review, operations are improved (increased efficiency comparative to ad hoc services) which may result in a better cost basis on individual matters. But there will be variation between groups within the document review organization, or from matter to matter, as processes are likely driven by the individual project managers and the systemic policies and procedures necessary to achieving consistent outcomes are absent. As a result, service levels across matters can still be unpredictable and overly dependent on the individuals involved.
Level 3: Defined
At level-three maturity, processes are now organizationally defined. Each project is executed within the organizationally-defined methodology. It is important to note that there is not one methodology that is right or best. Organizations will adopt differing strategies that solve these challenges in a manner consistent with their internal goals and philosophies.
The key is that with well-defined processes and the organizational structure in place to support them—that is to say, management of resources and how work is allocated, coordinated, and supervised to solve problems and meet challenges—variation from project to project and among different teams is greatly reduced. The efficiency of the document review will be further improved from that of an organization functioning at Level 2; however, the effectiveness of the service may still be unsatisfactory, absent of the establishment of key process indicators and other management measures that enable assessment of both inputs and outcomes. Needed improvements will be difficult to identify absent effective management controls.
It is important to recognize that processes can be efficient (fast and cost effective), but ineffective (poor results and perhaps ultimately more costly). Similarly, processes can be inefficient (untimely and costly), yet effective (end results are consistent and high quality). A competent document review must be both efficient and effective (this pursuit of both effectiveness and efficiency is the central tenant of Lean Six Sigma).
Level 4: Managed
At this stage, organization-wide processes are not only in place, but are measured and controlled.
Metrics such as KPIs (Key Process Indicators) and CTQCs (Critical to Quality Characteristics) have been identified, and are measured and tracked. The ability of the process to deliver the desired results consistently is established and continuously monitored. At this level of maturity, processes can be adjusted for individual review projects as needed and unique requirements met, without sacrificing quality or negatively impacting fundamental operational standards.
Clients will enjoy a high level of predictability in the delivery of services across matters. Organizations operating in a truly managed environment will evidence strong internal controls and likely have adopted practices such as Lean and Six Sigma throughout the firm. Mature policies and procedures will be in place and continuously monitored and reported, along with the policies and procedures (as well as the resident expertise) in place to implement corrective actions as necessary. In this regard, disciplines like Lean Six Sigma are not “overlays” or simply a discrete function within an organization (although there are often teams that oversee the work); rather, the processes, methodologies, and discipline are integral to the way all work is approached.
Level 5: Optimizing
An organization functioning at this level focuses on continuous process improvement. The opportunity for learning and growth, that is to say, the ability to leverage experience and expertise across projects, is greatly enhanced. Improvements to existing processes and innovations can be introduced, completing the virtuous cycle known in Six Sigma as “DMAIC” (Define, Measure, Analyze, Improve, and Control). This continuous process of improvement (and it is ongoing) typically results in driving operational costs lower over time.
Interestingly, the Capability Maturity Model not only captures much of what legal counsel has come to expect from services providers, but charts the evolved expectations of legal departments that seek to control litigation and compliance costs and enhance their efforts to better measure and manage spend.
However, while many legal departments or law firms have concrete expectations regarding the level of capability and the type of results they demand from service providers, they may not possess a reliable framework in which to correctly determine if a service provider is able to meet those expectations. This assessment model can be used to “score” a prospective provider holistically, as well as for discrete attributes. As a method for vetting prospective legal service providers, the CMM provides indicators of the ability to deliver, and provides counsel and procurement professionals a method for focusing on the truly meaningful and important characteristics in choosing a document review provider.
