Evaluating Technology Deployment at the State Level

Evaluating Technology Deployment at the State Level: Summary of Insights from the Evaluation of the Georgia Manufacturing Extension Alliance

Philip Shapira, School of Public Policy, Georgia Institute of Technology, Atlanta, GA 30332-0345, USA. Email: ps25@prism.gatech.edu. GaMEP Evaluation Working Paper 9802. Published in newsletter of the Institute of Technology and Regional Policy, Joanneum Research, 1998.

Introduction

In recent years, federal and state governments in the United States have increased their investments in public-private partnerships to promote the deployment of technology and improved business practices by industry, particularly among small and mid-sized manufacturers. In parallel, greater attention has been paid to the evaluation of these technology deployment initiatives, not only to assess the economic and business impacts of program interventions but also to promote more effective service delivery. This article summarizes the experience of one of these state partnerships – the Georgia Manufacturing Extension Alliance – in evaluating its technology deployment services.

Development of the Georgia Manufacturing Extension Alliance

For more than 35 years, the Georgia Institute of Technology (Georgia Tech) has provided industrial extension field services to manufacturers in the southeastern U.S. state of Georgia. These services include assistance in deploying new technologies, solving manufacturing problems, plant expansions, and training employees. Services are focused on the small and medium-sized manufacturing establishments with fewer than 500 employees that comprise 98 percent of the state’s 10,000 manufacturers. In 1994, Georgia Tech became part of the U.S. Manufacturing Extension Partnership (MEP) – a national network sponsored by the federal government’s National Institute of Standards and Technology (NIST). The MEP includes more than 70 manufacturing extension centers in all 50 states that work with over 2,000 affiliated U.S. public and private organizations to deliver industrial services to small and mid-sized firms.

With its affiliation with the national MEP network and aided by additional federal and state funding, Georgia Tech developed a new service structure, known as the Georgia Manufacturing Extension Alliance (GMEA). The lead organization in GMEA is the Georgia Tech Economic Development Institute (EDI). Under GMEA, existing services in manufacturing technology, operations planning and control, and facility planning were augmented by additional services in management and marketing systems, quality management, information technologies, energy management, environmental and occupational safety and health, and technology linkage. New regional field offices were opened, and a partnership was formed with other organizations including small business development centers, technical colleges, and utilities to offer a comprehensive array of technology and business support services to firms.

GMEA now operates a network of 18 regional offices, staffed with industrially experienced engineers and business professionals. Field office services are supported by program skill centers in areas such as quality, manufacturing information technology, human resource development, strategic management assistance, energy, and environmental services. From February 1994 to December 1996, GMEA served over 2,100 companies, equivalent to 21 percent of all manufacturers in the state. Included here were 39 percent of Georgia manufacturers with 20 to 499 employees. GMEA customers were served through 2,647 informal engagements, technical projects and assessments; 11 network group service projects (usually involving quality or labor force development); and 240 workshops and seminars. Roughly 36 percent of closed projects involved referrals to other organizations, private-sector consultants or vendors.

GMEA’s Evaluation Plan

Georgia Tech did not formally evaluate its predecessor industrial extension services, but – with the development of GMEA in 1994 – an explicit evaluation element was added. The evaluation element has three aims. First, to provide consistent feedback about the effectiveness, targeting and impacts of GMEA's services. Second, to support systematic learning about how services are being delivered and what services and approaches work best and why, so as to assist the ongoing improvement and management of program services. Third, to furnish evaluative information to GMEA's major stakeholders and sponsors, including the state of Georgia and NIST.

GMEA’s evaluation element is under the direction of the author of this article - an "external" faculty member from a separate academic unit who is not employed or supervised by the program. He is aided by an "internal" EDI senior researcher, who does not provide direct services to firms but who has access to direct service data. To date, approximately three percent of the program’s federal funding has been annually allocated to evaluation.

To develop evaluative procedures, we modeled the program’s operation (Figure 1). The essence of this program logic model is as follows. GMEA has a series of resources available to it (program inputs), which include staff time, expertise, funding, office locations, information systems, and access to other technology sources. It seeks to apply these resources to the needs of customers through the form of services and other kinds of assistance (program intervention). At or soon after the point of service delivery, customers are able to form a view of the effectiveness of, and satisfaction with, service delivery (customer valuation). We can further inquire as to whether the customer will pursue any implementation steps as a result of the services provided, such as making changes in equipment and facilities or initiating a new training program (customer intermediate actions). When customers act, we seek to ascertain any effects on the firm, for instance changes in sales, quality, investment or technology levels (business outcomes). We also aim to explore effects on jobs, taxes, and other economic development factors (development outcomes).

In developing information on these various elements, we recognized the need to understand the type, size, and industry of the customer being assisted (customer profile) and the resources committed by the customer, including customer staff time and investments (customer inputs). Additionally, we identified the need to be able to make comparisons of customers and non-customers in terms of participation and outcomes and consider the broader influence of changes in business factors (non-customer controls and measures of industry and business conditions).

A series of tools and procedures are used to obtain information and measurements on the various components of the program logic model that, in turn, can provide the foundation for subsequent evaluation analyses. Logistical, industry, personnel, and geographical information are tracked for each customer. Program staff activities, services, projects, expenditures, and fees are also tracked. A client feedback survey is administered to each customer upon completion of a major engagement. Additionally, a long-term benchmark survey is conducted every two years, with all manufacturers in the state, to track customer performance against non-customer controls. We also conduct structured case studies and special studies to examine the detailed linkages between GMEA services and impact on firm operations and profitability and analyze such topics as defense dependency and diffusion of ISO 9000 practices. Finally, the evaluation team, along with GMEA management, participates in organizational and external reviews of the program.

Findings from the GMEA Evaluation

The findings and results from these evaluation procedures have been used to produce a series of analytical and evaluative studies, which are distributed or used in briefings to program management, field staff, program sponsors, industry advisors, and customers. A World Wide Web site is maintained that allows open access to GMEA evaluation studies. Here are some of the major findings from the evaluation:

Customers generally report a high level of satisfaction with GMEA services (mean score of 4.5 out of a 5.0 maximum). This is a finding from the feedback survey administered by mail to customers about 30-45 days after project completion. A 70 percent response rate is obtained. Program managers receive copies of the completed surveys very soon after they are received. In a few cases, customers report problems or dissatisfaction. In such cases, program managers are able to respond to these problems, to correct them if possible or at least take measures to avoid difficulties happening again.

In the post-project survey, more than 80 percent of GMEA customers report that they have taken or expected to take action as a result of the assistance and services received. Companies that have taken or anticipate taking action tend to have higher satisfaction ratings than those not anticipating taking action. They also tend to have received more hours of service from GMEA staff.

However, customers do not act to the extent they thought 30-45 days after project closure. This was evident in a special study, using survey data for the same firms collected at two points in time – immediately after program participation and one-year later. This study found that, one-year after project completion, 68 percent of firms had actually acted on the program’s recommendations – still a favorable result. We also found that a further 17 percent of the projects were on hold (i.e., the firm was considering whether to implement project recommendations).

Nearly two-fifths of GMEA customers say that assistance from the program helped to improve profitability, with about one-sixth indicating increased sales. These results were reported in the long-term manufacturing survey, for customers who had completed projects 12 or more months prior to the survey point. An even greater proportion of these customers reported impacts in areas that are hard to quantify in dollar terms, although still significant for business performance, including existing process improvement (60 percent), improved management skills (over 55 percent), and greater attention to quality (about 45 percent).

Different types of projects have different impacts on firms, according to a further analysis of customer evaluation surveys. Product development and marketing projects are 60 percent more likely to increase sales than is the average project. Energy projects are most likely to lead to cost savings, and plant layout and environmental projects tend to help companies avoid capital spending. Marketing projects have the strongest link to job creation, and management and human resources projects have the strongest link to job retention. Quality projects do not rate highly in any impact area, although they do require the largest manufacturing customer staff time commitment.

In addition to project assistance, GMEA provides training courses and workshops – in 1997, to more than 2,000 participants from over 800 companies. A pilot study indicates that most training participants found these courses useful to their job (55 percent) and used what they learned (67 percent). Most participants believed that the training resulted in improvement in certain areas of operation and business performance, including improved quality (59 percent), improved shop floor processes (51 percent), changes in business or management strategy (49 percent), and improved employee skills (46 percent).

Results from a cost-benefit analysis indicate that the GMEA leverages substantial private investment that, in turn, is likely to lead to positive public and private returns over time. Companies invested from $3 to $13.3 for every dollar of public expenditures. For a typical company, the estimated private payback period for this private investment ranged from six to 22 months. The estimated net public and private sector benefits from GMEA’s first year services (provided at a state and federal cost of $3.8 million) ranged between $10 million and $26 million. The ratio of private and public returns to private and public investment ranges was between 1.2 and 2.7.
GMEA clients experienced a 0.3 percent growth rate in value-added per employee over non-clients for the period 1994-1996 – a finding drawn from the long-term benchmark survey of all manufacturers in the state. This survey allows customers to be compared with non-customers controlling for a variety of factors, including industry, size, urban access, inter-firm relationships, and assistance from other service providers. The productivity advantage of firms assisted by GMEA is significant, and is equivalent to a value-added increase of $366,000 to $440,000 for the average client plant over the two-year study period.
While many GMEA projects have only modest impacts, case studies confirm that some projects do lead to major effects. For example, a product development project yielded $2 million in bookings over two years and 10 new jobs. A plant layout project generated an $8 million sales increase, in which the CAD layout was used as a sales tool. This project also led to $50,000 in operating savings, $750,000 in inventory savings, a 40 percent increase in direct labor productivity, and 16 new jobs. These case studies use a logic-based model that allows for counter-factual explanations.

Issues in the Use of Evaluative Information and Analysis

While several different evaluation approaches indicate that the GMEA program appears to have favorable impacts, there are significant contrasts in terms of detailed findings, the reliability of estimates, the availability of controls, and time horizons. In the GMEA evaluation, a mix of quantitative and qualitative methods is used, but there is no clear superiority on this dimension. While it is important to quantify program impacts and we take care to qualify and verify numerical estimates, it is apparent that companies usually find it rather difficult to estimate the dollar value of program services. Some technology deployment and industrial extension services (such as reducing energy use or materials wastage) have immediate and quantifiable benefits. But other services, including inter-firm networking, quality assistance, and labor force training, have impacts that accrue over the longer term upon which it is hard to place a dollar value. As our one-year follow-up demonstrated, the elapsed time since project completion also affects how companies report benefits and costs and, where estimates can be made, there is frequently a wide margin of error. When aggregated together, "bottom-line" numbers can be derived, but great care needs to be taken in associating these numbers with a higher degree of accuracy than the underlying data collection realties allow.

There are also differences in the usefulness of different evaluation approaches to program managers, federal and state sponsors, and other interested parties. Among professional evaluators, the most favored method is usually the sophisticated controlled study (preferably with random assignment, although that is often hard to achieve). However, for other audiences, we have observed no direct correlation between the usefulness of an evaluation method with that method’s degree of sophistication or even use of controls. Whether as professional evaluators we like it or not, simple methods are often influential. This is evident at the state policymaking and funding level, where the demand for complex evaluation techniques is relatively weak. It is also true at the federal level, where business testimonials and case examples (coupled with targeted lobbying) can go a long way in securing continued funding. Business testimonials are more easily understood, of course – although, arguably to their credit, there is at least some "street wisdom" among decision makers that recognizes the difficulties of quantifying the impacts of technology deployment programs.

Thus, although some methods are clearly better than others for particular purposes, there is no one single method that by itself is adequate to the task of evaluating all aspects of a large and complex program like GMEA. Particularly in an environment where reliable econometric data is hard to come by, our approach has been to use a variety of sources to understand what the program is doing, what its impacts are, and where there may be opportunities for defining good practice and improving program performance. We have sought to implement evaluation methods that address questions of program justification. The early evidence from our surveys, case studies, and control group comparisons suggests that the program is leading to positive results. However, we would be among the first to recognize that further rigorous long-term studies are needed to conclusively demonstrate this. At the same time, we have also tried to implement evaluation approaches that promote program learning and dialogue about how program performance can be enhanced. We are beginning to see that some services and strategies are more likely to generate different, if not greater, results than others. This is information that program managers, sponsors, and customers find relevant. The decentralized nature of the MEP allows individual programs to alter their service mix and try innovative approaches. Complementary opportunities for comparison, through forums, workshops, personnel exchanges, reviews, best practice case studies, and (hopefully) econometric studies (including controlled ones) can allow successful program innovations to become verified, widely known, and adopted throughout the system. It is probably in this way that evaluation studies to aid program improvement can become most widely utilized.

Note

Philip Shapira is an Associate Professor with the School of Public Policy, Georgia Institute of Technology, Atlanta, GA 30332-0345, USA; and a Visiting Researcher at the Fraunhofer Institute for Systems and Innovation Research, Karlsruhe, Germany. Email: <ps25@prism.gatech.edu>.

For further information on the evaluation of federal and state industrial extension programs in the U.S., including the Georgia Manufacturing Extension Alliance, see the publications listing of the Georgia Tech Policy Project on Industrial Modernization at <http://www.cherry.gatech.edu/mod> and the web site of the US Manufacturing Extension Partnership at <http://www.mep.nist.gov>.

Figure 1. Program Logic Model, Evaluation of the Georgia Manufacturing Extension Alliance

TOP | HOME PAGE