The idea for Workforce Engage™ (WE) first came to Community Initiatives leadership in late 2000 when CI began to refocus some of its talent for building healthy and engaged communities on the workplace-and then on hospitals in particular. Driving the development of their new theory for health (as fundamental to economic value) in the workplace was CI's expertise in teams, networks, and community and leadership development, augmented by the emergent theory of social capital.

Stemming from experiences working with over 500 communities and organizations and extensive research on the underlying determinants of employee satisfaction and commitment, by September of 2002, CI had identified seven practices that bring about an engaged workplace.

During this research period, Workforce Engage supplemented its own knowledge and efforts by working with subject matter experts such as the American Hospital Association (AHA) and University of California San Francisco (UCSF) to help identify key factors driving employee commitment; the Health Research and Educational Trust (HRET) to gain wisdom about best practice indicators in healthcare; and the Centre for Innovation in Management (CIM, Simon Fraser University) to learn about the latest studies on social capital, how to measure it, and what outcomes it effects. By late 2002, Workforce Engage had drafted a solid list of research-supported questions.

Workforce Engage consulted with researchers and statisticians to determine the best format for its questionnaire (the basis of the WE™ system. These experts included people such as Carl Larson, Bob Boutilier, Barry Gruenberg, and Chris Nelson. Workforce Engage wanted to build a thorough (in that it really investigated issues), context-specific, and innovative tool. By January of 2002, CI had determined that a Likert scale would provide the most reliable measurement for the basic tool-with yes/no follow-up questions and a short series of open-ended questions to allow for real user input. This format was supported by our experts in research, statistics, and technology.

Similar diligence to that employed to make decisions about the questions was employed for technology. With the help of several CTOs, CI looked at tools hospital systems and providers currently use. CI's focus was on implementing an architecture that would be open to expansion and offer a structured dependable way to describe and transfer data.

Outcome: At the end of this phase, CI had thoroughly researched its topic area, defined a construct (seven practices), stated the purpose for what we wanted to measure, and selected an item format. In addition, WE had determined the best technology to use to build the tool and outlined a timeline for development.

In January 2002, Workforce Engage met with Carl Larson for guidance on the best process to developing Workforce EngageÔ as a reliable tool. Carl Larson helped define a process to construct, evaluate, and validate the WEÔ scorecard.

Outcome: At the end of this meeting, CI had identified the process reflected in the table below.

In early March 2002, CI conducted an item pool session with Northern Colorado Medical Center (Banner Health), following a process outlined by Carl Larson. Participants came from throughout the hospital. The resulting questions were added to WE's preliminary list of questions (approximately 70 in all) and sent to subject matter experts, including hospital executives, psychometricians, and researchers.

The objective of this phase was to identify the best 30-35 questions for Workforce Engage's core tool, the WE scorecard. After compiling results from all participants, Chris Nelson analyzed the data from this expert evaluation. The results of his analysis were a series of instruments that grouped questions based on user input to single categories and then across the entire evaluation.

Outcome: At the end of this phase, WE had identified approximately 40 questions to use in its pilot test. Note: these questions were given to a psychometrician to fine tune before we initiated our pilot test.

In April 2002, CI conducted a pilot test of the questions in the Engage Scorecard. Using an Access database, CI opened its questions to three campuses of the Iowa Health System: Blank Children's Hospital, Lutheran, and Methodist Hospitals. Following Chris Nelson's recommendation for the number of respondents needed in relation to the number of variables (items) to be factored 300 users respond to the questionnaire over the course of a week.

"Instruments F or M should be considered as the final instrument. I would also recommend that you include an additional 5-10 items in the final questionnaire; this will "build-in" some additional flexibility down the road. In summary, I think that your final questionnaire should be a very good instrument for use in your future efforts. It is obvious that your team has done a lot of good research and good work for developing such a reliable instrument."

  Validation Phase

Between August and December of 2002, Workforce Engage conducted a beta test of the Workforce Engage system. The purpose of the beta test was to validate the survey questions individually and collectively, to identify opportunities to refine and add questions, and to test the conceptual framework with hospital leadership and employees at all levels. In addition, the beta process was an opportunity to test the effectiveness of the web-based tool and the reporting and dialogue process. More than 25 sites, including 11 hospitals, two hospices, over a dozen long-term care settings and clinics, and a state hospital association, participated in the Workforce Engage beta test.

Outcomes: When the beta test was complete, CI conducted a statistical analysis of the 5600 survey responses. The analysis validated the core concepts of the tool and provided direction for further strengthening. The first test, Cronbach's alpha, measured the internal consistency of the survey. The alpha coefficient came in at .95, a very high number.

Next, a factor analysis with principal components analysis and varimax rotation was conducted in order to determine which questions covaried, or moved together. That is, if a respondent gave a high score to one question, what other questions would be likely to receive a high score? If a number of questions covary, it suggests they relate to a single issue. The questions within each of the Seven Practices, therefore, should move together if they measure a single dimension. The factor analysis confirmed the existence of multiple dimensions within the survey, and enabled CI to refine the Seven Practices. The Communication practice, for example, turned out to be three separate dimensions statistically: communication within teams, communication among teams, and communication between management and employees. Several other practices were adjusted as well based on the learning.