Social Network Analysis: A Tool for Examining Regional - - PDF document

IMPAQ International, LLC Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Social Network Analysis: A Tool for Examining Regional Collaboration on Workforce Development

Submitted to:

Association for Public Policy Analysis & Management 2016 Spring Research Conference

Submitted by:

1333 Broadway Suite 300 Oakland, CA 94612 Phone: 510.597.2400 / Fax: 510.465.7885 www.impaqint.com

Authors

• Ms. Kelley Akiya
• Dr. Raquel C. Sanchez
• Dr. Nada Rayyes

IMPAQ International, LLC Page ii Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

DISCLAIMER

This product was funded by a grant awarded by the U.S. Department of Labor’s Employment and Training

• Administration. The product was created by the grantee and does not necessarily reflect the official

position of the U. S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, expressed or implied, with respect to such information, including any information

• n linked sites and including, but not limited to, accuracy of the information or its completeness,

timeliness, usefulness, adequacy, continued availability or ownership. Any opinions, findings, conclusions

• r recommendation expressed herein are those of the author(s) and do not necessarily reflect the views
• f The U.S. Department of Labor, Lead Agency or the ABC Consortium.

IMPAQ International, LLC Page 1 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Introduction

The ABC1 Consortium is funded by a U.S. Department of Labor Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant2 to 11 Community Colleges in a region in California to develop career pathway programs in high-wage job sectors. The 11 ABC colleges offer various training programs, but all target at least one of three priority career clusters: advanced manufacturing, biotechnology, or transportation and logistics. The ABC programs are distinguished from other college programs by their focus on these career clusters, their provision of specific grant-funded student services (e.g., counseling and job placement), state-of-the-art equipment purchased with grant funding, dynamically dated or accelerated courses, and their emphasis on career paths that lead to advanced skills and employment opportunities. The research team was contracted by the grant lead to conduct an independent evaluation of the ABC

• initiative. The evaluation consists of an outcomes study and an implementation study, which also includes

a social network analysis (SNA). The primary research questions of the evaluation study are:

• 1. What are the effects of ABC-supported training programs on the participation in and completion
• f training programs in advanced manufacturing, transportation and logistics, and biotechnology?
• 2. What are the effects of the ABC-supported training programs on participants’ outcomes?
• 3. What are the key elements of the ABC initiative and how have they developed over the grant

period?

• 4. How has the ABC initiative brought partnering agencies together to collaborate around common

goals?

• 5. What are the key features of the network of ABC stakeholders and how does the ABC network

evolve over time?

• 6. How will ABC efforts be sustained beyond the life of this particular grant?

The current paper focuses on the SNA component of the evaluation, which we used to address research question 5 (What are the key features of the network of ABC stakeholders and how does the ABC network evolve over time?). The paper presents a preliminary descriptive analysis of social network data collected at two points in time: once during early implementation of the ABC initiative, and a second time near the end of the grant.

Background

Goals of the ABC Consortium

The purpose of the ABC consortium is to support regional alignment of college-based training both across the college campuses and with industry needs. According to the grant narrative, ABC “represents an unprecedented opportunity for the [region] to build accelerated, intensive, and regionally articulated

1 ABC is a pseudonym 2 Information about the TAACCCT Grant program is available at https://doleta.gov/taaccct/

IMPAQ International, LLC Page 2 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

programs of study so that TAA eligible3, dislocated workers or unemployed adults can earn degrees or credentials of value and enable them to enter the workforce in industries with growing occupational demand and opportunities for career and wage advancement.” The ABC initiative seeks to address the following challenges that influence the prospects of training participants and the economic health of the region: (1) Lack of “middle skill” employees; (2) Gaps in basic math and English skills, including English as a second language, among those seeking employment and furthering their education; (3) Lack of necessary technological skills; (4) substantial drop in wages due to jobs/positions, companies, and industries moving overseas and employees needing to take on jobs at much lower pay; (5) Difficulty matching hard to employ individuals with the region’s industrial jobs/careers in the three target sectors; and (6) lack of formal alignment between CTE programs and industry, or among community colleges. As the grant narrative described, prior to inception of the initiative, each college worked with industry, but did so separately, “developing disconnected certificates that are not regionally validated by industry

• r accepted by other educational institutions within the system.” The purpose of the ABC Consortium is

to address this fragmented system of training programs and establish a coherent regional network of education partners providing industry recognized credentials.

ABC Consortium Members

A community college district (the Lead Agency) leads the ABC Consortium in partnership with 11 community colleges spanning five separate community college districts. Other partners in the consortium include local Workforce Investment Boards (WIBs), American Job Centers (AJCs), universities, economic development agencies, employers, technical assistance (TA) providers and other community partners. After receiving the grant, the Lead Agency also hired two consultants, an industry cluster coordinator and a regional workforce coordinator to support industry engagement in the three clusters. In addition to these partnerships, the ABC Consortium engaged key individuals from other federal and state-funded initiatives operating in the region. These individuals include Deputy Sector Navigators (DSNs)4, positions funded by the California Community Colleges Chancellor’s Office (CCCO) Doing What Matters initiative, who are working with community colleges and employers in their region to align workforce training and career pathways in targeted industries. The ABC Consortium also partnered with the director of a Jobs and Innovation Accelerator Challenge grant5, a multi-agency federal grant program designed to support the advancement of high-growth, regional industry clusters.

Implementation of the ABC Initiative

Implementation of the grant is occurring over a four-year period. Year 1 was primarily a planning year, during which the ABC Consortium focused on developing training programs at the colleges, hiring staff, and solidifying partnerships. Colleges began serving students in the last few months of Year 1 and

3 Trade Adjustment Assistance (TAA) eligible workers are those adversely affected by foreign trade. 4 Additional Information about the Deputy Sector Navigator is available at: :http://doingwhatmatters.cccco.edu/

WEDDGrants/GranteeRoles.aspx

5 Additional information about the federal Jobs and Innovation Accelerator grant program is available at:

https://www.eda.gov/challenges/jobsaccelerator/

IMPAQ International, LLC Page 3 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

continued to serve students through Year 2 and Year 3 and during the first part of Year 4. In Year 2, one

• f the TA providers assisted colleges with career pathway mapping and small planning groups formed to

begin developing industry-led cluster partnerships. By year 3, the cluster partnerships were fully

• implemented. Year 4, the final year of the grant, was intended to be an evaluation and reporting year,

with the cluster partnerships transitioning into self-sustaining groups. The current paper was drafted approximately half way through Year 4.

A Social Network Analysis Approach to Evaluation

Given that building a coherent network of partners to improve alignment between fragmented systems was a stated goal of the grant, examining the formation and maintenance of the network was critical to evaluating grant implementation. While collaborative approaches are increasingly being used in K-12 education (Leana & Pil, 2006; Pounder, 1998), school-university partnerships (Gajda & Koliba, 2007; Hughes & Weiss, 2007), public-private partnerships (Austin, 2000), and evaluation (O’Sullivan & D’Agostino, 2002; Preskill & Torres, 1999; Rodriguez-Campos, 2012), collaboration rarely appears in the literature as a measureable program goal. When it does, the researcher often plays a facilitative and participatory role in the collaboration itself. As the third-party evaluator on this TAACCCT grant, we are in the unique position to empirically examine the development of a regional collaboration toward workforce development. However, few tools exist for operationalizing and measuring the development of a collaborative effort as a proximal outcome within a larger outcome study. Therefore, we took this opportunity to explore Social Network Analysis (SNA) as such an approach. While there are multiple approaches to incorporating analysis of networks in program evaluations, Taylor, Watley & Coffman (2015) argue that assessment of network connectivity is an important area of focus (p. 23). These authors further suggest that network connectivity has two dimensions that can be examined through evaluation: network membership, or the people or organizations who participate in the network, and how connections between members are structured. An SNA addresses these two dimensions by helping to describe network membership and the collaborative relationships within the network. SNA is also particularly useful in evaluating inter-agency collaboration on community-wide initiatives. For example, evaluators have used SNA to investigate inter- agency collaboration on initiatives related to child welfare (e.g. Friedman et al., 2007), education (e.g. Bartholomay, Chazdon, Marczak & Walker, 2011; Noonan, Erickson, McCall, Frey, & Zheng, 2014), and public health (e.g. Leischow et al., 2010; Gregson, Sowa, & Flynn, 2011). Because community college- based training programs and career pathway development require input and support from agencies across systems (e.g. higher education, workforce development, economic development) and coordinated engagement with employers and industry, we included an SNA of interagency collaboration in our evaluation. As described by Proven, Veazie, Staten & Teufel-Shone (2005), SNA in evaluations of inter-agency collaboration collects and analyzes data from individuals or organizations that may be interacting with

• ne another and is useful for describing connections among agencies within a network (p. 605).

IMPAQ International, LLC Page 4 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Additionally, because an SNA focuses on relationships between network members and displays and analyzes data using a matrix, researchers can examine the number of other organizations with whom a given organization has a relationship, the total number of relationships in a network, the types of interactions between organizations, and the extent or strength of each relationship (Proven et al., 2005). In adopting a social network paradigm (Daly, 2010), this study conceptualizes the role of the ABC Consortium as the intermediary in the effort to transform higher education and workforce development systems in the region. According Borgatti and Ofem’s (2010) typology, this is a Type 5 social network study, which focuses on the antecedents of network structuring (i.e., how and why different collaborative structures form).

Methods

We used a questionnaire to collect social network data from consortium members and administered the questionnaire twice during grant implementation. Collecting social network data at multiple points allows evaluators to examine the evolution of networks and track progress in building and sustaining collaboration (Proven et al., 2005). Therefore, we administered the questionnaire towards the end of Year 2, to capture information about the network during early stages of grant implementation and again at the beginning of Year 4, to capture information about the network as the grant was nearing its end date.

Questionnaire Development

To develop the questionnaire, we conducted a review of the literature to identify peer-reviewed studies across disciplines that used SNA or surveys to examine collaboration between organizations. The review prioritized studies related to collaboration on new initiatives, collaboration among workforce development partners, and systems integration. The initial draft questionnaire included items adapted from published surveys and interview protocols as well as original items created specifically for the study. We piloted the questionnaire with eight individuals who had been involved in grant-funded activities as

• f the first part of Year 2. We also invited pilot respondents to participate in a 20-minute telephone debrief

to elicit feedback on the length, format, comprehensiveness and relevance of the questionnaire as well as the clarity of the directions, terminology and response options. Four respondents participated in the debrief calls and one provided feedback via email. Based on the feedback from the pilot respondents, we revised the questionnaire directions, changed some terminology and added new questions. The revised questionnaire, administered in Year 2, asked respondents to name the organization(s) they work for, their job title(s), which industry cluster was most relevant to their work and to name up to 10 individuals with whom they collaborate on the ABC initiative. For each partner named, respondents were asked to provide the name of the partners’ organization, their role/position at the organization, their frequency of communication, topics on which collaborate, whether they collaborated with this partner prior to the ABC initiative, and how important the collaboration with this person is to achieving their

• rganization’s objectives for the ABC initiative.

During preliminary analysis of social network data collected in Year 2, partnerships between organizations (as opposed to individuals) and industry clusters (i.e., Advanced Manufacturing, Biotechnology and Transportation and Logistics) emerged as prominent themes. These findings were echoed in the analysis

IMPAQ International, LLC Page 5 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

• f qualitative data from site-visits, interviews and consortium meeting observations conducted in Year 2

and 3. Therefore, we revised the questionnaire prior to re-administering it in Year 4 to include more questions related to industry cluster and reduce the overall number of questions. The final instrument also included an item asking respondents whether they expected to continue working with their named partners after the grant ended.

Administration of the Questionnaire

We used information collected from site visits to the community colleges, interviews with partners, as well as observations of meetings, training and events to identify individuals in the region involved with the ABC initiative. We also consulted with the Lead Agency and reviewed attendance and sign-in sheets from ABC sponsored meetings and events. Initially, we invited 157 individuals to take an online questionnaire in Year 2, including community college staff implementing the ABC initiative at the 10 ABC colleges, members of the three cluster leadership teams, staff at the Lead Agency, consultants paid through the grant, and individuals who attended ABC-sponsored events. When contact information was available, partners named by respondents who were not included in the original sample were also invited to answer the questionnaire. Ultimately, 191 people were invited to take the questionnaire and 162 responded (85% percent). Prior to administering the questionnaire in Year 4, we reviewed site-visit, interview, and observation data collected since the questionnaire was first administered in Year 2. We also solicited feedback from the Lead Agency and their consultants about who was still involved in the ABC Consortium and who may have since joined the initiative. Individuals who had been invited to complete the questionnaire in Year 2, and were still involved in the ABC initiative, were invited to take the final questionnaire in Year 4. Individuals who had become involved in the ABC initiative since Year 2 were also invited. In total, 160 individuals were invited to complete the Year 4 questionnaire and 115 responded (72%).

Analysis

As a preliminary step in the analysis, we tabulated questionnaire responses to identify participants’ (respondents and partners) organizations, industry cluster and whether or not the relationships with named partners had existed prior to the ABC initiative. We then imported the network data into UCINET 6 (Borgatti, Everett & Freeman 2002), a software package designed for social network analysis. UCINET allows researchers to analyze a network of interacting individuals and relationships among them, and to display those relationships graphically in a map. In social network maps, nodes depict actors in a network and lines between them depict relationships, called ties, between nodes. For this study, the actors are the

• rganizations participating in the ABC consortium and ties represent the existence of relationships

between one or more individuals from the two organizations. We collapsed individual-level data by organization and examined the three industry clusters within the consortium (Advanced Manufacturing, Biotechnology and Transportation and Logistics) in Year 2 and Year

• 4. We considered each cluster in each year its own network, for a total of six networks (three in Year 2

and three in Year 4). The industry clusters were not mutually exclusive, so organizations could be included in more than one network in the same year.

IMPAQ International, LLC Page 6 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

For each network, we calculated degree centrality and betweenness centrality scores for each actor or

• rganization in order to investigate the role of various partners in the consortium. SNA assumes that an

actor’s position in a network partially shapes their opportunities and constraints within the network (Borgatti, Mehra, Brass, & Labianca, 2009) and therefore measures of position, such as degree centrality and betweenness centrality, help evaluators identify if and why an actor is considered important within the network. Degree centrality measures the extent to which an actor is directly connected to other nodes in the network, with high centrality indicating that an actor has many ties and is relatively visible or prominent within the network (Wasserman & Faust, 1994). Betweenness centrality measures the extent to which an actor is directly connected to nodes that are not directly connected to each other. Actors with high betweenness centrality potentially control information and can serve as a gatekeeper or broker within the network (Freeman, 1979). In addition, we calculated the betweenness centralization of each of the networks. Centralization is a measure of the entire network, as opposed to an individual actors, and indicates the extent to which betweenness varies across actors or how heterogeneous betweenness is within the network (Knoke & Young, 2008; Wasserman & Faust, 1994). In highly centralized networks, centrality is concentrated with a few actors and in decentralized networks, centrality is more dispersed. Lastly, we used the number of individual relationships between two organizations as a measure of tie

• strength. For example, if two individuals at Organization X had a relationship with one individual at

Organization Y, the strength of the tie between X and Y would be two. If only one individual at Organization X had a relationship with an individual at Organization Y, the tie would be one.

Results

Consortium Participants

Responses from the social network questionnaire suggest that a broad range of actors have participated in the ABC consortium. Exhibit 1 presents the number of individual actors identified as participating in the consortium in Year 2 and Year 4 organized by the type of organization to which they belong. The vast majority individual actors worked for organizations other than one of the participating ABC colleges or Lead Agency, such as employers, workforce development agencies, and other community partners such as unions and community-based organizations. Similarly, at the organizational level, employer partners, workforce development partners, other community partners and education partners were the most common types of organizational actors participating in the consortium (see exhibit 2). Overall, the number of individual actors decreased between Year 2 and Year 4 while the number of

• rganizational actors increased6. The number of individual actors from the ABC colleges decreased from

82 to 51, though one additional community college joined the consortium after Year 2. The number of

6 We adopted what Hanneman & Riddle (2005) refer to as a “mathematical approach” to network analysis, which

assumes that the observations or actors in a given network are not a "sample" of some larger population of possible observations, but are the population of interest. Therefore, we did conduct significance testing of differences in means.

IMPAQ International, LLC Page 7 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

individual and organizational actors belonging to the workforce development system also decreased between Year 2 and Year 4. However, more employer and economic development partners participated in the consortium in Year 4 than in Year 2.

Exhibit 1. Individual Actors Participating in the Consortium

Type of Organization Year 2 Year 4

N % N % ABC college 82 35.50 51 23.29 Employers 56 24.24 81 36.99 Workforce Development Partners 34 14.72 23 10.50 Other Community Partners 18 7.29 18 8.22 Education Partners 14 6.06 17 7.76 ABC Consultants 11 4.76 9 4.11 Economic Development 6 2.60 11 5.02 Deputy Sector Navigators 5 2.16 3 1.37 Lead Agency 5 2.16 6 2.74 Total 231 100.00 219 100.00

Exhibit 2. Organizational Actors Participating in the Consortium

Type of Organization Year 2 Year 4

N % N % Employers

37 34.91 59 47.20

Workforce Development Partners

19 17.92 11 8.80

Other Community Partners

15 14.15 16 12.80

Education Partners

13 12.26 11 8.80

DBS College

10 9.43 11 8.80

DBS Consultants

6 5.66 6 4.80

Economic Development

4 3.77 9 7.20

Deputy Sector Navigators

1 0.94 1 0.80

Lead Agency

1 0.94 1 0.80

Total

106

100

125 100 The decrease in community college and workforce development staff and increase in employers may reflect a shift in focus of the ABC Consortium. During the early stages of the grant, the ABC initiative included activities related to developing a workforce intermediary and common salesforce of business service representatives across the colleges, workforce development and economic development. In subsequent years, the ABC consortium turned its efforts to launching the industry-led cluster

• partnerships. Thus the work of the consortium became more relevant to employer and economic

development stakeholders. Turnover at the ABC colleges and at one of the TA providers (an ABC Consultant) may have also contributed to changes in individual membership. Several faculty and TA staff

IMPAQ International, LLC Page 8 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

who worked with the colleges left their organizations between Year 2 and Year 4, so while the

• rganizational partnerships remained, the number of individuals involved in these partnerships declined.

When we disaggregated organizational actors by industry cluster, differences in the number and type of

• rganizations emerged. Exhibits 3 and 4 present the number of each type of organizational actor within

each cluster in Year 2 and Year 4 respectively. In both Year 2 and Year 4, more organizational actors participated in Cluster 17 than the other two clusters. However, between Year 2 and Year 4, the number

• f organizational actors within Cluster 1 decreased from 83 to 66, while the number of organizational

actors within Cluster 2 decreased by a smaller amount (60 to 54) and the number of organizational actors within Cluster 3 increased modestly (49 to 54). In Year 2, employer partners made up approximately one- third of all organizational actors within Clusters 1 and 2, compared to just over ten percent of

• rganizational actors within the Cluster 3. By Year 4, the proportion of employer partners increased in all

three clusters. The most notable difference occurred within Cluster 3, which nearly tripled the number of employer partners by Year 4.

Exhibit 3. Organizational Actors Participating in the Consortium in Year 2, by Cluster

Type of Organization Cluster 1 Cluster 2 Cluster 3

N % N % ABC College

7 8.43 9 15.00 5 12.24

ABC Consultants

6 7.23 5 8.33 6 10.20

Lead Agency

1 1.2 1 1.67 1 2.04

DSN

1 1.2 1 1.67 1 2.04

Economic Development

4 4.82 4 6.67 3 6.12

Education

9 10.84 8 13.33 6 12.24

Employers

28 33.73 18 30.00 5 10.2

Workforce

15 18.07 12 20.00 14 28.57

Other

12 14.46 2 3.33 8 16.33

Total

83 100 60 100 49 100

7 Cluster 1, Cluster 2 and Cluster 3 are pseudonyms for the three clusters targeted in the ABC initiative.

IMPAQ International, LLC Page 9 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Exhibit 4. Organizations Participating in the Consortium in Year 4, by Cluster

Type of Organization Cluster 1 Cluster 2 Cluster 3

N % N % ABC College 7 10.61 8 14.81 7 12.96 ABC Consultants 4 6.06 4 7.41 2 3.70 Lead Agency 1 1.52 1 1.85 1 1.85 DSN 1 1.52 1 1.85 1 1.85 Economic Development 7 10.61 4 7.41 6 11.11 Education 7 10.61 8 14.81 3 5.56 Employers 25 37.88 21 38.89 18 33.33 Workforce 6 9.09 6 11.11 9 16.67 Other 8 12.12 1 1.85 7 12.96 Total 66 100 54 100 54 100

The three cluster networks were in different stages of development when the ABC Consortium formed which influenced their further growth and development over the course of the study. For example, Cluster 1 was a more mature network, with a large number of employers who already knew each other through a regional professional association. Cluster 3 started with a smaller group of actors, but benefited from the investment from the grant and grew its membership, especially once the consortium launched the cluster partnership.

Ties within the Consortium

Questionnaire responses concerning the collaborative ties between actors show that many of the participating individuals had not worked together before the ABC initiative. As shown in Exhibit 5, in Year 2, 41.7% of all reported ties were new, or between individuals who had not collaborated with each other before working on the ABC initiative. In Year 4, 50.32% of ties were new. Cluster 3 had the highest percentage of new ties in both Year 2 and Year 4.

Exhibit 5. Number of New Ties, by Cluster

Cluster Year 2 Year 4 N % N % Advanced Manufacturing 150 40.43 118 47.20 Bioscience 112 41.33 67 48.91 Transportation and Logistics 85 48.57 77 58.78 All Relationships 241 41.17 231 50.32

The questionnaire responses also suggest that individual actors expect to continue collaborating after the grant ends. As presented in Exhibit 6, of all reported ties within the consortium network, 85% were considered by the respondent to be very likely or likely to continue. When examined by cluster, we found a similar trend in each cluster network, with 90% of ties related to Cluster 1, 87% of those related to

IMPAQ International, LLC Page 10 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Cluster 2 and 86% related to Cluster 3 considered by questionnaire respondents likely or very likely to continue.

Exhibit 6. Number of Ties Reported as Likely to Continue

Consortium-wide Cluster 1 Cluster 2 Cluster 3 Rating N % N % N % N % Very Likely 283 61.66 166 66.40 100 72.46 66 50.00 Likely 106 23.09 61 24.40 20 14.49 48 36.36 Neutral 37 8.06 13 5.20 12 8.70 8 6.06 Unlikely 17 3.70 5 2.00 4 2.90 3 2.27 Extremely Unlikely 1 0.22 0.00 0.00 Missing 15 3.27 5 2.00 2 1.45 7 5.30 Total 459 100 250 100 138 100 132 100

Actor Centrality

We used UCINET 6 to calculate degree centrality scores for each organizational actor within the six

• networks. Exhibit 7 lists the organizational actors with the highest degree centrality scores in each
• network. The degree centrality scores suggest that the Lead Agency and ABC consultants were visible

within the networks, forming a relatively high number of ties with other actors. For example, the Lead Agency had the highest, second highest or third highest degree centrality score within each cluster in Year 2 and Year 4. Certain colleges were prominent in each of the cluster networks in both Year 2 and Year 4, though different colleges took on this role in the different clusters. This is because most colleges had programs in only one or two clusters. Also of note is the central role of the DSNs in Clusters 1 and 2 in Year 4. Though not the most central actor, they appear to be an important part of these networks, and invested time and energy into the ABC initiative despite not receiving any direct funding from the TAACCCT grant.

Exhibit 7. Organizations with the Highest Centrality Scores

Year 2 Year 4 Rank Cluster 1 Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3 One College 1 Lead Lead College 2 Lead College 5 Two College 2 College 3 Consultant 2 Lead College 4 College 8 Three Lead College 4 Consultant 1 WIB 1 College 3 Lead Four WIB 1 DSN College 5 College 1 College 7 Consultant 2 Five Consultant 1 Consultant 1 College 6 DSN DSN Consultant 3

We also used UCINET 6 to calculate betweenness centrality scores and to produce social network maps

• f the six networks. In these maps (see Exhibits 8-13), larger nodes represent actors with higher

betweenness scores and thicker lines represent stronger ties between the nodes or actors. Similar to the patterns in the degree centrality scores, the betweenness centrality scores suggest that the ABC colleges, the Lead Agency, ABC consultants and DSNs have played important roles within the networks, though some variation exists among the clusters.

IMPAQ International, LLC Page 11 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

As shown in Exhibits 8 and 9, within Cluster 1, two colleges, a WIB, the Lead Agency and the DSN had high betweenness in Year 2 and Year 4, though the betweenness of the Lead Agency and DSN decreased by Year 4. Within Cluster 2 (Exhibits 10 and 11), the Lead Agency and DSN had higher betweenness in Year 2 than in Year 4, whereas the two ABC colleges had higher betweenness in Year 4. Within Cluster 3 (Exhibits 12 and 13), the Lead Agency and multiple ABC consultants had high betweenness along with some of the ABC colleges in Year 2. The Lead Agency continued to have high betweenness in Year 4, but to a lesser extent, while the colleges had higher betweenness in Year 4. These results suggest that although the Lead Agency, ABC consultants and DSNs remained active within the networks throughout the grant period, over time other actors within the networks depended on them less to broker relationships. Also, in Clusters 2 and 3, the colleges emerged as brokers in Year 4, which is promising in terms of sustaining the network after the grant funding expires. Because the Lead Agency and ABC consultants are funded by the grant, relying too much on those actors to broker and facilitate collaboration at the end of grant implementation could jeopardize the continued success of the

• consortium. If other actors are positioned to facilitate collaboration, the consortium has more potential

to continue building and maintaining relationships after the grant funding ends. In general, employer partners, the majority of workforce development and education partners and other community organizations were in the periphery of the networks. Most were only connected to one or two

• ther actors in the network, though there are some exceptions. Within Cluster 1, the WIBs are located

near the center of the network in both Year 2 and Year 4, and had relatively high betweenness scores. This illustrates their ability to broker relationships or connect to different parts of the network. One factor that may contribute to this structure is that one of the local WIBs brought on a consultant to coordinate the industry-led cluster partnership. In a sense, the WIB has taken some ownership and accountability for growing and improving the cluster partnership, which necessitates facilitating collaboration between business and public sector partners in the region. In Clusters 1 and Cluster 3, there were are a small number of employers with multiple ties to other important actors in the network. This is reflective of differences in how the consortium engaged employer partners in the different clusters. The ABC Consortium engaged employer “champions” to lead industry- led cluster partnerships in Cluster 1 and Cluster 3, and so the employer champions have developed relationships with a variety of actors in the network. In Cluster 2, the college faculty and staff already had well developed relationships with employers before the ABC Consortium formed, so the focus of the colleges was more on sharing information with each other and other public sector partners. Thus, employers remained toward the periphery of the network.

Network Centralization

Exhibit 14 presents the betweenness centralization measure for each of the networks. The betweenness centralization was relatively low (0 indicates a completely decentralized network and 100 a completely centralized network), indicating that the networks were relatively decentralized and not particularly dependent on a small portion of actors. Even though the results of the actor centrality calculations indicate that some actors have high betweenness relative to other actors, the centralization measure suggests that the network as a whole is not overly dependent on these specific actors. In the case of the

IMPAQ International, LLC Page 12 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Exhibit 8. Cluster 1 Network, Year 2 Exhibit 9. Cluster 1 Network, Year 4

ABC Colleges DSN Economic Development Lead Agency Employers Education Partners Workforce Development Other ABC Consultants ABC Colleges DSN Economic Development Lead Agency Employers Education Partners Workforce Development Other ABC Consultants

IMPAQ International, LLC Page 13 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Exhibit 10. Cluster 2 Network, Year 2 Exhibit 11. Cluster 2 Network, Year 4

ABC Colleges DSN Economic Development Lead Agency Employers Education Partners Workforce Development Other ABC Consultants ABC Colleges DSN Economic Development Lead Agency Employers Education Partners Workforce Development Other ABC Consultants

IMPAQ International, LLC Page 14 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Exhibit 12. Cluster 3, Year 2 Exhibit 13. Cluster 3, Year 4

ABC Colleges DSN Economic Development Lead Agency Employers Education Partners Workforce Development Other ABC Consultants ABC Colleges DSN Economic Development Lead Agency Employers Education Partners Workforce Development Other ABC Consultants

IMPAQ International, LLC Page 15 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

ABC Consortium where the Lead Agency and ABC consultants receive temporary grant funds to support their work building the consortium, decentralization can be an advantage because these same actors may not have the resources to participate in the consortium to the same extent after the grant ends. If the networks were more centralized, with a small set of actors brokering most of the relationships, they could become fragmented or even collapse if those small set of actors leave the network or become less active.

Exhibit 14. Betweenness Centralization, by Cluster

Year Cluster 1 Cluster 2 Cluster 3 Year 2 10.36 11.43 16.13 Year 4 11.34 13.68 9.57

Discussion

The results of the SNA suggest that the ABC Consortium made substantial progress in developing a network of organizations invested in improving community college-based training programs in the region. Over 40% of the relationships in Year 2 and over 50% in Year 4 were new, providing evidence that the ABC Consortium supported increased collaboration in the region and brought people together who hadn’t previously worked with each other. The results also illustrate that the three clusters had different needs and advantages at the beginning of the grant, so the nature and structure of collaboration varied across

• clusters. For example, Cluster 3 had few organizational partners in Year 2, but also contained a higher

share of new relationships in Year 2 and Year 4, indicating that the ABC Consortium was both more necessary and more beneficial to Cluster 3 in terms of developing collaboration, compared to Clusters 1 and 2. The high degree centrality scores of the Lead Agency and ABC consultants suggests that the organizations tasked with coordinating efforts to build a cohesive regional network played an important role by being active in the ABC Consortium and visible to a range of stakeholders. Over the course of the grant, these actors formed a relatively high number of relationships with others, and observation and interview data collected as part of the evaluation suggests these organizations have been a constant and supportive

• presence. For example, the Lead Agency was in frequent contact with each of the participating colleges

about the administration of their training programs and grant requirements and were regular participants in the cluster partnership planning meetings. Similarly, the industry cluster coordinator and regional workforce coordinator (both ABC consultants) were well known and appreciated by employers, college staff and workforce development staff alike. An unresolved question about the ABC Consortium is if and how it can continue its work as a collaborative network in the future, without the financial support and infrastructure provided through the grant. By the time the questionnaire was administered in Year 4, several individuals with significant responsibilities in the ABC Consortium had left their positions. For example, some college staff, individuals working for one

• f the TA providers and the regional workforce coordinator were no longer working on the ABC initiative.

After Year 4, the Lead Agency and industry cluster coordinator will no longer receive funding from the grant and lead staff for the ABC initiative at the participating colleges will no longer have explicit responsibility for implementing the ABC initiative at their campus. With these changes, there’s a risk of

IMPAQ International, LLC Page 16 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

attrition in the networks and of weakening relationships between organizations that participated in the consortium. Although, the SNA methodology cannot predict whether collaboration among the consortium members will be sustained beyond the life of the TAACCCT grant, the results provide some promising signs. First, questionnaire respondents overwhelmingly reported that their relationships with their named partners would continue, which suggests that there is an expectation of and openness towards continuing

• collaboration. Second, other actors who were not reliant on funding from the grant had central roles in

the three networks in Year 4. For example, some of the local WIBs and participating colleges are central actors in these networks and have other resources and funding to continue to invest in training programs. The DSNs will also be in their roles for an additional two years after the grant ends, which illustrates the benefit of coordinating the two grants during the initial planning year. In addition, the decentralized structure of the networks in Year 4 lowers the risk that the network will collapse when the Lead Agency and ABC Consultants are no longer leading the implementation of the ABC initiative.

Policy Implications and Areas for Future Research

The Georgetown University Center on Education and the Workforce projects that by 2020, 30 percent of U.S. jobs will require a post-secondary credential or Associate’s Degree, and another 35 percent will require at least a Bachelor’s degree (Carnevale, Smith & Strohl, 2013). At the same time, the U.S. is expected to face shortages of workers with these qualifications. The TAACCCT program, which supported the ABC initiative, is one of several federal initiatives attempting to respond to this challenge through increasing low-income and low-skilled workers’ access careers providing middle-class incomes, while also meeting the needs of industry. However, TAACCCT and other federal, regional and local initiatives are implemented in a context of fragmented workforce development systems and gaps in information sharing or coordination with higher education, K-12 education, economic development, and social services (e.g. TANF, SNAP) systems. Fragmentation in the workforce system is due to multiple factors including the broad array of actors involved, disparate funding sources, and a lack of fully formed connections between workforce development, economic development and business in many communities (Anderson & Carpenter, 2015). Recognizing these challenges, stakeholders and funders are increasingly interested in fostering collaboration and integration of these disparate systems. A notable example is the Workforce Innovation and Opportunity Act, which replaced the Workforce Investment Act. WIOA calls for greater alignment between workforce, education and economic development systems and places greater responsibility on local WIBs to engage in collaboration, convening and partnership activities (Copus, Javier, Kavanagh, Painter, & Serrano, 2014). Moreover, it encourages engagement of business and industry to create career pathways and sector partnerships and encourages local workforce boards to serve as regional conveners (Copus et al., 2014). States are also placing greater value on collaboration in order to align system. In California, where the ABC initiative was implemented, the Career Pathways Trust is investing $250 million in grants to K-12 school districts and community college districts to broker regional efforts to implement career pathways and form regional collaborations with business, community organizations and postsecondary education (California

IMPAQ International, LLC Page 17 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Department of Education, 2015). The National Governor’s Association Center for Best Practices is also working with 25 states to support local sector partnerships, which bring employers within one industry together with government, education, training providers, economic development, labor, and community

• rganizations to focus on the needs of an industry within a regional labor market (Woosley & Groves,

2013) Despite the investment from a variety of funders to improve collaboration across systems as a means of improving services to job-seekers, guidance on how to convene and sustain partnerships is sparse. Rigorous evaluation studies of individual programs provide evidence that career pathway and sector- based strategies improve wages and earnings of workers (e.g. Maguire et al., 2010; Smith & King, 2011; Zeidenberg & Kienzl, 2009), but because development and evaluations of these models are in their infancy, researchers and stakeholders are still trying to understand how the specific components of these models improve outcomes and how they can be scaled-up or replicated in different contexts. Furthermore, there is little research documenting specific challenges for organizations in forming and sustaining collaboration or specific strategies for overcoming them (Melendez, Borges-Mendez, Visser, & Rosofsky, 2015). Recent scholarship by Melendez et al., (2015) in this area concludes that up-front costs, competition and fragmentation are prevalent barriers and that an “anchor” organization with programmatic and jurisdictional authority is needed to foster effective regional collaboration. Yet, further investigation into this approach is needed, as well as evaluation of whether such models of collaboration lead to better outcomes for workers and job seekers. SNA and other methodological approaches to measuring collaboration can potentially offer insight for implementers, job seekers or workers as well as governmental and non-governmental funders into the types of collaborative structures and partnership arrangements that ultimately lead to improved livelihoods for low-income and dislocated workers. Functional and efficient collaboration promotes information sharing among service providers as well as between service providers and job-seekers, which is an essential component to ensuring access to opportunity throughout a region or community. Therefore, identifying network characteristics that increase or decrease functionality and efficiency can support stakeholders in planning and implementing cross-system and inter-agency collaboration. Limitations Deciding which actors belong in a network (e.g. boundary specification) is a common challenge for researchers conducting an SNA (Marin & Wellman, 2011). When researchers exclude actors that in reality are part of the network, the dataset is incomplete, and both actors and network ties are missing. Laumann et al. (1983) suggests three non-mutually exclusive approaches to correctly identifying actors within a network: to include actors with formally defined positions, actors who participated in key events or activities, or to begin with a small group of seed actors and expand the boundary to include those with relationships to the seed actors (as cited in Marin & Wellman, 2011, p. 12). Although we employed all three approaches, there was ambiguity even among key leaders of the consortium as to who was involved with the initiative. Therefore, evaluators may have mis-specified the boundary of the network.

IMPAQ International, LLC Page 18 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Another limitation of the study is non-response. Although we achieved high response rates among those invited to take the questionnaire, we did not have contact information for all partners named by

• respondents. Therefore the SNA also includes actors that were not invited to take the questionnaire. In

particular, we were often not able to locate email addresses for employer contacts of community college staff or workforce system professionals. In the vast majority of cases, missing contact information was for individuals on the periphery who were only named by one other individual in the network. Moreover, even when we were able to contact similar individuals to invite them to complete a questionnaire, these individuals often had little if any knowledge of how their work related to the ABC initiative. Thus we assumed that in most cases the respondent was likely providing more accurate information about the relationship than the non-respondent would have if he or she completed a questionnaire. Together, the challenges with specifying the network boundary and non-response suggest that the results are best interpreted as a good approximation to the true collaborative relationships among network members, and not necessarily an exhaustive, precise documentation of them. Another limitation is the extent to which the study can report on the sustainability of the network. We collected social network data towards the end of the grant close to the time that funding would end, but did not have an opportunity to collect follow-up data. Therefore we cannot report on whether collaboration continued or changed in the months or years following the initial funding from the grant. We collected data on members’ perception about whether relationships would continue, but cannot offer insight into whether those relationships actually continued. We recommend that future research on consortiums initiated with temporary grant funds collect follow-up data well after the grant funding ends in order to determine whether collaborative relationships persist over time. Lastly, as the current paper is a preliminary descriptive analysis of social network data recently collected, it does not include statistical tests on whether certain attributes of an actor (e.g. the type of organization they work for) influence their position in the network or how different measures of position influence each other. Future analysis will examine these issues as well as the extent to which network-level measures (e.g. centralization) changed from Year 2 to Year 4.

References

Anderson, S., & Carpenter, A. (2015). Fragmentation in workforce development and efforts to coordinate regional workforce development systems: A case study of challenges in Atlanta and models for regional cooperation from across the country. Retrieved from https://www.frbatlanta.org/ commdev/publications/discussionpapers/2015/02.aspx Austin, J. E. (2000). Strategic collaboration between nonprofits and business. Nonprofit and voluntary sector quarterly, 29(suppl 1), 69-97. doi:10.1177/089976400773746346 Bartholomay, T., Chazdon, S., Marczak, M.S., & Walker, K.Z. (2011). Mapping extension's networks: Using Social Network Analysis to explore extension's outreach. Journal of Extension, 49 (6), Article 6FEA9. Berardo, R. (2009). Generalized trust in multi-organizational policy arenas. Political Research Quarterly,

IMPAQ International, LLC Page 19 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

62(1), 178-189. doi: 10.1177/1065912907312982 Borgatti, S.P., Everett, M.G. & Freeman, L.C. (2002). Ucinet for Windows: Software for Social Network

• Analysis. Harvard, MA: Analytic Technologies.

Borgatti, S.P., Mehra, A., Brass, D.J., & Labianca, G. (2009). Network analysis in the social sciences.

• Science. 323(5916), 892-8955. doi: 10.1126/science.1165821

Borgatti, S. & Ofem, B. (2010). Overview: Social Network Theory and Analysis. In A. J. Daly (Ed.) Social Network Theory and Educational Change, (pp. 17-29).Cambridge, MA: Harvard Education Press. California Department of Education (2015). California Career Pathways Trust (CCPT). Retrieved from http://www.cde.ca.gov/ci/ct/pt/ Carnevale, A.P., Smith, N., Strohl, J. (2013). Recovery: Job Growth and Education Requirements Through

• 2020. Retrieved from https://cew.georgetown.edu/wpcontent/uploads/2014/11/

Recovery2020.FR_.Web_.pdf Copus, J., Javier, R. Kavanagh, T., Painter, R. & Serrano, R. (2014). The Workforce Innovation and Opportunity Act (WIOA) –driving innovation, collaboration and performance. Retrieved from http://www.nawb.org/documents/Publications/WIOA_Overview.pdf

• Daly. A. (2010). Social Network Theory and Educational Change. Cambridge, MA: Harvard Education

Press. Freeman, L.C. (1979). Centrality in social networks: Conceptual Clarification. Social Networks, 1, 215-239. Friedman, S.F., Reynolds, J., Quan, M.A., Call, S., Crusto, C.A., & Kaufman, J.S. (2007). Measuring changes in interagency collaboration: An examination of the Bridgeport Safe Start Initiative. Evaluation and Program Planning, 30, 294-306. doi: 10.1016/j.evalprogplan.2007.04.001 Gajda, R. & Koliba, C. (2007). Evaluating the imperative of intraorganizational collaboration: A school improvement perspective. American Journal of Evaluation, 28, 26-44. doi: 10.1177/1098214006296198 Gregson, J., Sowa, M., & Flynn, H. K. (2011). Evaluating form and function of regional partnerships: Applying social network analysis to the Network for a Healthy California, 2001–2007. Journal of Nutrition Education and Behavior, 43, 67–74. doi: 10.1016/j.jneb.2011.02.018. Hanneman, R and Riddle, M. (2005). Introduction to social network methods. Riverside, CA: University of California, Riverside (published in digital form at http://faculty.ucr.edu/~hanneman/) Hughes, J., & Weiss, J. (2007). Simple rules for making alliances work.Harvard Business Review, 85(11), 122. Knoke, D., & Yang, S. (2008). Social network analysis (Vol. 154). Thousand Oaks, CA: Sage Publications. Leana, C., & Pil, F. K. 2006. Social capital and organizational performance: Evidence from urban public

IMPAQ International, LLC Page 20 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

• schools. Organization Science, 17: 353–366. doi.org/10.1287/orsc.1060.0191

Leischow, S.J., Luke, D.A., Mueller, N., Harris, J.K., Ponder, P., Marcus, S. & Clark, P.I. (2010). Mapping U.S. government tobacco control leadership: Networked for success? Nicotine & Tobacco Research, 12, 888–894. doi: 10.1093/ntr/ntq112. Maguire, S, Freely, J., Clymer, C., Conway, M. & Schwartz, D. (2010) Tuning in to Local Labor Markets: Findings from the Sectoral Employment Study. Retrieved from: http://ppv.issuelab.org/ resource/tuning_in_to_local_labor_markets_findings_from_the_sectoral_employment_impact_ study Marin, A., & Wellman, B. (2011). Social network analysis: An introduction. In J. Scott & P.J. Carrington (Eds.) The SAGE handbook of social network analysis, (pp. 11-25.). London: Sage Publications. Meléndez, E., Borges-Mendez, R., Visser, M. A., & Rosofsky, A. (2015). The Restructured Landscape of Economic Development Challenges and Opportunities for Regional Workforce Development

• Collaborations. Economic Development Quarterly, 29(2), 150-166. doi: 10.1177/0891242414

566 151 Noonan, P., Erickson, A.M., McCall, Z., Frey, B.B., & Zheng, C. (2014). Evaluating change in interagency collaboration of a state-level interagency education team: a social network approach within a utilization-focused framework. Educational Assessment, Evaluation and Accountability, 26, 301–

• 316. doi: 10.1007/s11092-014-9193-2

O'Sullivan, R. G., & D'Agostino, A. (2002). Promoting evaluation through collaboration findings from community-based programs for young children and their families. Evaluation, 8(3), 372-387. Pounder, D. G. (1998). Restructuring schools for collaboration: Promises and pitfalls. Albany, NY: State University of New York Press. doi: 10.1177/135638902401462466 Preskill, H., & Torres, R. T. (1999). Building capacity for organizational learning through evaluative

• inquiry. Evaluation, 5(1), 42-60. doi: 10.1177/135638909900500104

Provan, K.G., Veazie, M.A. , Staten, L.K., & Teufel-Shone, N.I. (2005). The use of network analysis to strengthen community partnerships. Public Administration Review, 65, 603-613. doi: 10.1111/j. 1540-6210.2005.00487.x Rodríguez-Campos, L. (2012). Advances in collaborative evaluation. Evaluation and program planning, 35(4), 523-528. doi: 10.1016/j.evalprogplan.2011.12.006 Smith, T. C., & King, C. T. (2011). Exploratory Return-on-Investment Analysis of Local Workforce

• Investments. Retrived from: https://www.utexas.edu/lbj/publications/4359

Taylor, M., Whatley, A. & Coffman, J. (2015). Network evaluation in practice: Approaches and

• applications. The Foundation Review, 7(2), 22-37. doi: doi: 10.9707/1944-5660.1247

Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. No. 8. In M.

IMPAQ International, LLC Page 21 Social Network Analysis: A Tool for Examining Collaboration Around Workforce Development

Granovetter (Ed.), Structural analysis in the social sciences [series]. New York City, NY: Cambridge University Press. Woosley, L. & Groves, G. (2013). State Sector Strategies Coming of Age: Implications for State Workforce

• Policymakers. Retrived from http://www.nga.org/files/live/sites/NGA/files/pdf/2013/

1301NGASSSReport.pdf Zeidenberg, J.D. & Kienzl, G. S. (2009). Educational Outcomes of I-BEST, Washington State Community and Technical College System's Integrated Basic Education and Skills Training Program: Findings from a Multivariate Analysis. New York, NY: Community College Research Center, Teachers College, Columbia University.