Economic Development Institute, Georgia Institute of Technology

Atlanta, GA 30332

School of Public Policy, Georgia Institute of Technology

Atlanta, GA 30332

Prepared for symposium to appear in the Journal of Technology Transfer.

Federal and state governments have made extensive investments in policies to promote technology development and deployment by the business sector. Programs have been established in a variety of technology promotion areas, including support for start-up technology ventures, collaborative research and development between firms and public technology institutions, the transfer and commercialization of technologies developed by universities and federal laboratories, and the deployment of new manufacturing technologies in industry. One recent study estimates that combined federal and state support for such cooperative technology programs now exceeded $3 billion annually (Coburn and Berglund 1995). At the same time, demands on these programs to demonstrate economic and business results have also grown, in parallel not only with increased budgets but also with the renewed interest throughout the public sector in the last few years in performance measurement and the more effective delivery of public services (Carlisle 1997; Gore, 1993).

However, technology promotion programs have specific characteristics that often make it difficult to present hard evidence which can attribute economic and business outcomes to publicly-supported inputs. For example, technology promotion programs frequently provide assistance or resources that require additional downstream private actions and investments for results to materialize. Public policy can usually only encourage these further private steps, but cannot control them. Moreover, it generally takes a length of time before program assistance is translated into action by assisted businesses and, in turn, into realized effects on production, sales, or jobs (Bozeman, Youtie, and Shapira 1997). Policy-makers recognize this when they affirm that technology promotion programs are long-term initiatives that cannot be expected to show significant results over short time frames. Yet, these same policy-makers also desire evidence of rapid impacts as they make annual budget decisions (Shapira, Youtie, and Roessner, 1996). This is a wish that almost all program managers aim to fulfill as they try to justify the economic value they believe their program has generated.

One strategy that technology programs use to present at least some immediate information about long-run effects is to estimate or project program benefits and outcomes very soon after assistance has been completed. Companies receiving assistance from technology promotion programs may be asked to report anticipated figures for sales increases or new jobs created as a consequence of program assistance. Program staff or outside evaluators may then report the anticipated results directly to sponsors or may construct some type of model which corrects for or extrapolates based on the fact that the results are "anticipated" not "actual" results (Pressman 1996).

The problem here is that anticipated results may not be valid indicators of actual program impacts. Even with hindsight, it is difficult for companies to put precise dollar values on the impacts of program assistance. Detailed records are rarely kept, while in many cases technology programs influence "soft" factors such as know-how, skill, or knowledge flows which, if not entirely intangible, are complex to monetarize. The problems of estimating the economic value of assistance are even greater when companies are asked to project forward. Survey respondents have special difficulties in answering hypothetical questions, particularly about future effects, and tend to provide speculative answers (Converse and Presser 1990; Smith 1981). Yet, if too much time elapses between program participation and surveying, response rates and the ability of customers to provide data about the impacts of program participation may decline, as personnel change or other business events occur. As with most evaluation designs, given limited resources (and the limited patience of customers in responding to repeated requests for information), trade-offs are involved in selecting not only what questions should be asked, but at what point in time those questions should be administered. Other evaluation strategies (such as using control groups of non-customers) can provide additional reference points to interpret data from customers, but even with more intricate evaluation designs, the issue of survey timing remains a critical element.

This article analyzes customer reports of impact from a particular technology program - the Georgia Manufacturing Extension Alliance. Using survey data for the same firms collected at two points in time - immediately after program participation and one-year later - we are able to examine the reported economic effects of the program and explore the relationships between customer reports of impact and the timing of data collection. We discern that program participation has significant economic effects; but we also find that that close to the point of service delivery, customers receiving assistance tend to over-estimate the benefits of program participation and under-estimate the commitment and resources necessary to achieve the benefits. Subsequent measurement, at about a year after program participation, indicates that customers can provide a more realistic assessment of benefits and costs, although with some drop-off in response rates. The one-year survey shows that program participants still receive net benefits, although at a lower level than first anticipated immediately after the close of the project. Importantly, for a relatively small number of cases where program participation results in very large positive impacts, we find some evidence that immediate post-project measurements under-estimate the scale of the ensuing outcomes. In following sections, these findings, and the background to them, are discussed in more detail.

The Georgia Manufacturing Extension Alliance (GMEA) provides industrial extension and technology deployment services to the state’s manufacturers. GMEA’s services are focused particularly towards the small and medium-sized companies with fewer than 500 employees that comprise the vast majority of Georgia’s 10,000 manufacturers. The lead organization in GMEA is Georgia Institute of Technology’s Economic Development Institute, which builds on a 30-year history of industrial extension service provision. The Economic Development Institute operates a network of 18 regional fields offices, staffed with industrially-experienced engineers and business professionals. Field office services are supported by staff in several program skill centers in such areas as quality, manufacturing information technology, human resource development, strategic management assistance, and energy and environmental services. As necessary, GMEA links companies with specialized expertise at Georgia Tech, federal laboratories, industry technology centers, and private consultants. GMEA also works in partnership with organizations including small business development centers, technical colleges, and utilities to offer a comprehensive array of technology and business support services to firms.

In 1994, GMEA was formed from what was then the state-sponsored Georgia Tech Industrial Extension Service, becoming part of the national Manufacturing Extension Partnership (MEP). Coordinated by the U.S. Department of Commerce’s National Institute of Standards and Technology, the MEP is comprised of more than 70 industrial extension and manufacturing technology deployment programs operating in all 50 states. The MEP is itself a partnership involving federal, state, and industry resources. Additional federal resources provided through the MEP have allowed GMEA to increase the scale of its operations and forge new linkages with state, federal, and industry groups. The MEP has encouraged the development of systematic evaluation procedures for its manufacturing extension affiliates, including the implementation of standardized performance measures, periodic reporting, and customer surveying (National Institute of Standards and Technology 1994; National Institute of Standards and Technology, 1996).

Through company assessments, formal projects, informal assistance engagements, training workshops, technology demonstrations, and other activities, GMEA now serves about 1,000 Georgia manufacturers annually. To understand and assess the impacts and outcomes of these services to manufacturers, GMEA has established an explicit evaluation protocol (Shapira and Youtie 1994). This evaluation protocol assesses program and customer impacts through several complementary methods. The first is a post-project survey of customers 30 to 45 days after a project has been closed. This survey asks for customer satisfaction information and reports of both received and anticipated quantitative and qualitative outcomes. The second is a one-year follow-up survey conducted by telephone after the first year to further estimate actual (not anticipated) outcomes. In addition to the two customer surveys, the GMEA evaluation strategy also includes a controlled survey sent every two years to all Georgia manufacturers with 10 or more employees designed to assess longer-term impacts of the program, as well as cost-benefit analyses of the program and case studies of successful projects.

In this article, we focus on the results of the first two methods: the post-project survey of customers and the one-year follow-up survey. The post-project survey procedure was instituted by GMEA in 1994. Each month, information about closed projects with manufacturers is drawn from the program’s management information system. For projects with significant program intervention (defined as eight hours or more of program staff assistance), a standardized satisfaction and impact questionnaire is sent centrally by mail to the principal company contact for the project. Shorter program interactions with companies, such as initial visits or informal consultations, are not formally evaluated through this procedure. In 1994, about 55 percent of the program’s interactions with customers were for 8 hours or more (by 1997, these more lengthy interactions had grown to represent two-thirds of program interventions). The time required for information reporting and mailing means that customers usually receive the post-project questionnaire about 30-45 days from the completion of the project. As necessary, the first questionnaire is followed by a second mailing and telephone contact. The response rate to the post-project survey procedure is relatively high - about 70 percent.

In July and August of 1995, we then conducted a telephone follow-up survey of the first wave of completed 1994 GMEA projects. Initially, there were 129 projects for which post-service mail questionnaires were available. Sixteen of these projects were excluded from further analysis because of various reasons (for example, the projects were duplicates or it turned out that the projects were still ongoing). This left a database of 113 projects. Eighty percent of these projects were at least a year old. (The remaining projects were generally at least nine months old, although one was less than nine months old.) Since most of the projects surveyed were at least a year old, we refer to this follow-up survey as the one-year follow-up survey. Customer contacts for 75 of these 113 projects were reached during the one-year follow-up survey administration period.

The overall response rate to the follow-up survey was 66 percent. The primary reason for non-response (for 28 of 38 non-respondents) was that the company did not return telephone calls or faxes, with no further information available. In other cases, the company was reached but the principal project contact had left, the company declined to respond to the questionnaire, or discrepancies were discovered in the initial database. We further explored the characteristics associated with non-response by examining differences between respondent and non-respondent answers in the post-project mail questionnaire (Table 1). No clear direction of bias emerged. Respondents were somewhat more likely to anticipate taking action as a result of the assistance and services they received than were non-respondents. At the post-project stage, 85 percent of subsequent follow-up survey respondents anticipated taking action, while 75 percent of the follow-up survey non-respondents anticipated taking action (p=.151). On the other hand, at the post-project stage, non-respondents were somewhat more satisfied with the assistance and services they received than were respondents. The mean overall service rating on the post-service questionnaire was 4.2 for one year follow-up survey respondents compared to 4.4 for non-respondents (p=.131). These ratings were based on a five-point scale in which 1=poor, 3=adequate, and 5=excellent.

Additionally, the number of workers employed in respondent facilities was compared to that for non-respondents, as well as to the total GMEA customer base. GMEA customers that received surveys tended to be larger than the general GMEA customer pool (perhaps because very small firms are served through means other than formal projects, including informal assistance, workshops and seminars). One year follow-up survey respondents tended to employ fewer employees than non-respondents, although the differences were not significant.

Analysis of post-project surveys of 113 GMEA projects closed in 1994 and subsequent responses to 1995 one-year follow-up survey.

The business and economic impacts of a program like GMEA are determined by a sequence of events. First, does the customer take any action as a result of the assistance and services provided? Second, what are those actions and what impacts do they have in terms of sales, cost savings, capital investment, and jobs? We now turn to probe these questions, drawing on the information and results reported GMEA customers in the two surveys. Our aim is to determine the likelihood and scale of reported actions and impacts and to make comparisons between customer responses to the one year follow-up survey and those provided in the post-project mail questionnaire. (We leave for analysis in subsequent articles such questions as how the business and economic performance of assisted firms compares, over the long-run, with non-assisted controls.)

In the post-project mail questionnaire, 64 customers (85 percent) took or anticipated taking action as a result of the assistance and services they received. One year later, 51 of these firms (or 68 percent) actually took action (Table 2). Additionally, one year later, nine projects were reported to be on hold (i.e. the firm was still considering whether to implement the project recommendations). It appears that one year after, customers did not take action to the extent they thought 30-45 days after project closure. This is due mainly to decisions to put some projects on hold, rather than to definitely not take action on project recommendations. If some of the projects reported to be on hold on-year after project close-out are actually implemented, there will be a narrowing of the gap between the follow-up survey rate of action and the post-project survey expectation.

In the post-project mail questionnaire, 30 percent of the project contacts anticipated sales increases as a direct result of GMEA assistance and services. One year later, only 17 percent actually experienced sales increases. Actual median sales increases were lower than anticipated median sales increases. For a few outlying customers, however, actual sales increases were substantially higher than anticipated sales increases; the high impact outlying customer reports positively skewed the mean such that actual mean sales increases exceeded anticipated mean sales increases. It is possible that these very large sales increases are due to factors not entirely attributable to program intervention. To err on the side of conservatism in attributing program intervention to sales impacts, we thus also report an adjusted mean, which excludes outliers more than three standard deviations from the mean. The adjusted mean annualized sales increase for those companies reporting this outcome was $171,000 in the one-year follow-up, compared with the initial prediction of $207,000 in the immediate post-project survey. (See Table 2.)

Technology deployment projects can result in changes in operating costs, as program staff help firms to better use labor or make savings in factors such as materials or energy. Based on the one-year follow-up survey, Table 3 shows the proportion of GMEA projects which led to changes in labor, material, waste minimization, energy, and other areas. Over one-fifth of projects, companies reported labor operating cost changes resulting from GMEA project participation. Five projects actually produced higher operating costs following the creation of new jobs, resulting in mean added annualized costs of $25,000 across all respondents. In just under one-fifth of projects, waste minimization savings were reported, with mean annualized savings of $27,000 for reporting companies.

There were two main differences between the post-service survey and one-year follow-up survey in terms of how questions about operating costs were asked. The post-service survey had one general question about operating costs, but the one year follow-up survey had several questions asking about each operating cost component individually. Furthermore, the post-service survey only asked about savings, whereas the one-year follow-up survey asked about additional costs as well as additional savings.

Despite these differences, some comparison of operating cost changes between the post-project and one-year surveys can be made. Overall,

in the one-year survey, 40 percent of customers said that GMEA projects had led to at least one of the operating cost items. This is slightly less than the 45 percent of customers anticipating operating cost impacts in the immediate post-project survey (See Table 2.) Actual annualized median cost savings ($20,000) were lower than anticipated annualized median cost savings ($50,000). Again, actual mean cost savings exceeded anticipated mean cost savings, because a few high impact projects had an upward influence on the one-year follow-up survey mean.

Capital expenditures include investments in plant, equipment, or other capital items. Project respondents generally viewed capital expenditures as being for equipment, although one respondent referred to construction of a new facility as a capital expenditure. Two aspects of capital expenditures were addressed: capital investments made and capital expenditures avoided.

Substantially more projects resulted in new capital investments than in capital expenditures avoided. Twenty-eight percent of the projects in the one year follow-up survey effort led to increases in capital expenditures, a rate fairly close to the 32 percent in the post-service mail questionnaire anticipating capital expenditures. Only nine percent of projects actually helped companies avoid capital expenditures according to the one-year follow-up survey, although 17 percent of the projects were anticipated to help avoid such expenditures in the immediate post-project survey.

Where capital investments were made, they were significant in monetary terms. In the on-year survey, customers reported that their capital investment-related projects resulted in mean expenditures of over $400,000. This mean was bolstered by a few very large capital investments. It may not be reasonable to attribute these unusually large investments entirely to GMEA actions, although it may be fair to attribute a fraction (Shapira and Youtie 1995). The average dropped to $165,600 when one very large outlying project was excluded, and expenditures related to another project capped at $1 million. Still, even this capped adjusted mean represents a significant level of investment.

Capital expenditures avoided had smaller dollar impacts than capital investments made. In the one-year follow-up, companies that reported avoiding capital expenditures due to project assistance reported mean savings of $74,000.

When the results of the post-project and one-year surveys are compared, we find what is by now the consistent pattern. When they are made, the capital investments required to implement project recommendations turn out to be higher at the one-year mark than anticipated 30-45 days after project completion. Equally, when they occur, avoided capital expenditures reported in the one-year follow-up were, on average, lower than originally anticipated in the post-service mail questionnaire.

Technology deployment projects may help manufacturers create new jobs or save existing jobs. However, through changes in technology or manufacturing operations, these projects might lead to fewer jobs in some cases. The post-project mail questionnaire asked about new jobs created or current jobs saved, but did not ask about jobs lost. This omission was rectified in the one-year follow-up survey.

In the one-year survey, we found that more companies had added jobs than they anticipated at the point of service (Table 4). Fifteen projects in the one-year follow-up survey had added jobs, whereas new jobs were anticipated for only 10 projects in the post-project survey. The mean number of jobs added was also higher in the one year follow-up survey than the post-service mail survey, though the median number of new jobs created was only one more (indicating that most of the extra jobs were clustered in a few cases). In the one-year follow-up, two companies reported that jobs had been lost, with a mean of 5 jobs lost. Overall, the cases and number of jobs added far outweighed the instances of job loss. In the one year survey, fourteen companies reported that jobs had been saved, representing a somewhat higher number of cases than originally anticipated in the post-project survey.

In addition to capital expenditures, companies incur other costs in participating in projects, including the value of committed staff time. Both surveys asked companies to report the total days of staff time committed to the GMEA project. Forty percent of the companies participating in the one-year follow-up survey provided this information. On average, these companies actually committed more days to a project than anticipated (Table 5). In the one-year follow-up, the mean project required more than 266 days, compared with 132 days anticipated in the post-service mail questionnaire. In the one-year survey, the median number of days was 95, in contrast to the 25 days expected at project completion. In many cases, more than one company employee worked on the GMEA project, which helps to account for these rather large reported company time commitments. But - as the differences between the means and medians suggest - there is also a wide upward tilt in the distribution of survey responses. Indeed, in the one-year survey, 15 companies reported that they committed 100 or more days to projects. Complementary data gathered from on-site visits to several customer facilities suggested that GMEA customers were attributing all the days spent on the project from the company’s perspective, including personnel commitments before GMEA was even contacted. To adjust (albeit arbitrarily) for the effect of these few large outliers, we also present a capped mean company staff time commitment, capping company staff time attributable to GMEA participation to 100 days. The capped mean company staff commitment in the one-year follow-up remains significantly larger than the company time commitment initially expected in the post-project survey.

In several of the impact areas associated with GMEA project participation (e.g., sales impacts and cost savings), the mean one-year impacts exceeded mean end-of-project impacts, but the reverse was true of the medians. This trend suggests that reports of actual impacts are likely to include outlying, high performing projects. We investigated this trend by conducting on-site case studies of two high performing projects. These case studies further illuminated the extent of differences between anticipated and actual impacts (Table 6). While we generally find that companies over-estimate benefits and under-estimate costs at project completion compared with the results the subsequently report at the one-year mark, the reverse is true for the small number of high performing cases. We found that that in two high performing cases, actual sales and jobs impacts one-year out were higher than initially anticipated. A product development project actually yielded $2 million in extra sales bookings and 10 new jobs, substantially more than the $50,000 and 6 new jobs anticipated in the post-project mail survey. A plant layout project (in which a computer-generated layout was used as a sales tool) generated an $8 million sales increase and 16 new jobs, rather than the $2 million sales increase and 10 new jobs that the company had first anticipated. In addition, the plant layout project yielded substantial operating cost savings (resulting from reduced overtime pay, energy consumption, and scrap rate) that had neither materialized nor had been anticipated at the time of the post-project customer survey.

The one year follow-up survey suggests that for the average project, GMEA customers tended to overestimate their benefits and underestimate their costs in their anticipated responses soon after the point of service. One mitigating factor is that the one-year time frame may still not be sufficient to allow all benefits to materialize (nor perhaps to allow all costs to be recognized). As part of our longer-term controlled evaluation design, a statewide manufacturing technology survey conducted towards the end of 1996 may provide sufficient records to allow us to track assisted companies over a two-year time frame.

It could be argued that while all companies find it hard to accurately predict project impacts ahead of time, smaller companies with less developed accounting systems may face particular problems. We explored this idea by examining differences between anticipated and actual impacts among large manufacturers (100 or more employees) and small manufacturers (less than 100 employees). No relationship between company size and nearness of anticipated to actual impacts emerged.

Should technology programs eschew immediate post-project impact data entirely in favor of longer term analyses? To be considered here is the fact that many technology assistance programs now survey customers at the point of project completion as part of their quality control procedures. These programs want to assess company satisfaction with services received and obtain rapid feedback on any problems or further needs so as to respond rapidly. To this useful procedure, questions about received and anticipated impacts are often added with little extra marginal cost (since a post-project customer survey is being undertaken anyway). The question thus becomes: should programs add a further survey point, some distance away from the completion of a project to more accurately capture benefits and costs? An additional survey point adds cost, of course, in an environment where financial resources for systematic evaluation efforts are scarce. Additionally, programs are limited in the number of times they can go back to their customer to request estimates of program impacts. At some point, too many data requests become a burden to the customer that may discourage further program participation (Shapira, Youtie, Roessner 1996). And, not to be forgotten is the fact that while subsequent measurements may be more valid, they also may show a smaller level of net program impact for typical customers. In the topsy-turvy world of budgetary politics, at least some program managers may gamble that better information is not worthwhile.

This said, we would argue that it is critically important to conduct additional follow-up studies of program impacts over time, beyond the immediate close-out of a project. The results from post-project surveys are not radically out-of-line with the results reported one-year later (and, in fact, for capital expenditures, the match is quite close). If no other survey can be undertaken, a well-managed post-project survey can provide some useful insights. However, at least in the case of technology deployment, a one-year perspective on project impacts allows a rather more valid analysis of realized results than possible through short-term post-project surveys. Of course, within the wise use of resources and the constraints of company forbearance and record-keeping, we would argue that even longer-term customer tracking is desirable (and for other types of technology promotion programs, particularly those which are more research-intensive, long-run follow-ups over multiple years would seem to be essential). Yet, even limited to a one-year follow-up, an added validity is provided which gives policy-makers data and analysis in which they can have a higher level of confidence. While we find, in the case of GMEA at the one-year mark, that program participation benefits are smaller than initially expected while costs are larger, overall the net economic impacts are significant and positive. One-year employment impacts also appear to be higher than initially reported. Finally, we note that subsequent follow-up is needed to properly identify high performing projects. While often not recognized until a period of time has elapsed, the actual economic impacts from these projects can far exceed what was originally anticipated. Moreover, the benefits from high performing projects are such that they may justify the program by themselves. Without longer-term customer tracking, these particular impacts might be missed.

The authors would like to acknowledge the assistance of Alan Olisko and Mike Lane for their assistance in conducting the surveys. The comments of Gretchen Jordan and John Mortensen were also most helpful in improving the paper.

1. Berglund, D., and C. Coburn. Partnerships: A Compendium of State and Federal Cooperative Technology Programs. Columbus, OH: Batelle Press, 1995.

 

1. Bozeman, B., J. Youtie, and P. Shapira. A Design for the Evaluation of the Impacts of the Georgia Research Alliance. Atlanta, GA: School of Public Policy and Economic Development Institute, Georgia Institute of Technology, 1997.

 

1. Carlisle, R. "The Influence of Evaluation on Technology Policy-making and Program Justification: A State-Level Perspective." In Manufacturing Modernization: Learning from Evaluation Practices and Results, edited by P. Shapira and J. Youtie. Atlanta, GA: School of Public Policy and Economic Development Institute, Georgia Institute of Technology (1997): 119-126.

 

1. Converse, J. and S. Presser. Survey Questions: Handcrafting the Standardized Questionnaire. Newbury Park, CA: Sage Publications, 1990.

 

1. Gore, A. Creating a Government that Works Better and Costs Less: A Report of the National Performance Review. New York: Plume, 1993.

 

1. National Institute of Standards and Technology. Setting Priorities and Measuring Results. Gaithersburg, MD: U.S. Department of Commerce, 1994.

 

1. National Institute of Standards and Technology. NIST MEP Reporting Guidelines, Version 2.1, Gaithersburg, MD: U.S. Department of Commerce, 1996. Electronic copy available through <<http://www.mep.nist.gov/centers/programs/evaluation/guidelines/index.html>>.

 

1. Pressman L. "Economic Impact of Technology Licensing: Measuring Induced Investment." Presentation at Workshop Technology Transfer: How Do We Know What Works? Santa Fe, New Mexico, April 28-May 2, 1996.

 

1. Shapira, P., and J. Youtie, Georgia Manufacturing Extension Alliance: Overview of the Evaluation Plan. GMEA Evaluation Working Paper 9401. Atlanta, GA: Georgia Institute of Technology, School of Public Policy and Economic Development Institute, 1994. Electronic copy available through <<http://www.cherry.gatech.edu/mod>>.

 

1. Shapira, P., J. Youtie, and J.D. Roessner. "Current Practices in the Evaluation of Industrial Extension Programs." Research Policy 25 (1996): 185-214.

 

1. Shapira, P. and J. Youtie. Assessing GMEA’s Economic Impacts: Towards a Benefit-Cost Methodology. GMEA Evaluation Working Paper. Atlanta, GA: Georgia Institute of Technology, School of Public Policy and Economic Development Institute, 1995.

 

1. Smith, T., 1981. "Qualifications to generalized absolutes: ‘approval of hitting’ questions on the GSS." Public Opinion Quarterly 45 (1981): 224-230.
2. Author Biographies

Jan Youtie is a senior research associate in the Economic Development Institute, Georgia Institute of Technology. Her research interests include economic development, technology policy, evaluation, and market analysis.

Philip Shapira is an associate professor in the School of Public Policy, Georgia Institute of Technology. He teaches and conducts research in the areas of technology policy, regional economic development, and industrial competitiveness.