Increasing the Quantity and Quality of School Leadership Candidates Through Formation...

The standards movement, with its focus on measurable student outcomes and accountability, has resulted in societal demands for fundamental changes in the nature of schooling. Consequently, expectations for school leaders have changed significantly and school administrators, who previously were primarily expected to manage organizational processes and facilitate incremental change, are now being held accountable for student outcomes. They are expected to possess leadership skills to inspire, encourage, and empower individuals to perform at high levels of effectiveness and efficiency.

Educational leadership preparation programs are faced with the challenge of preparing school administrators who can lead in ways that will facilitate improved student achievement. At the same time, schools are facing high turnover in administrative positions and candidate pools for the positions are often inadequate (Adams, 2002; Glass, 2000; Keller, 1998; Lashway, 2001; Petzko & Scearcy, 2001; Volp, 2001).

The U.S. Department of Education Policy Forum on Educational Leadership was conducted in 1999 to address the shortage of qualified candidates for school leadership positions. In their policy brief, Forum participants recommended that schools begin "growing their own leaders by recognizing potential leaders and giving them structured opportunities to build their expertise" (U.S. Department of the Education Policy Brief, 1999). One way to do this is to provide shortterm "formation" activities to help individuals become better informed about what school leaders do and more aware their own interests and abilities in relation to leadership endeavors (Daresh & Playko, 1992). The purpose of this study was to examine the effectiveness of a regional initiative to provide formation activities as a means of improving the quality and quantity of candidates for school leadership positions.

Review of Literature

The literature review for this study focused on three topics: (a) the implications of current "reform" efforts for school leaders and school leadership candidates, (b) the identification of standards and competencies for school leaders, and (c) the need for innovative developmental processes for the education of prospective school leaders. The literature indicates that the knowledge and skills traditionally associated with the management of school operations are insufficient to address challenges inherent to the contemporary environment of schooling. The greatest challenge faced by school leaders may be that of implementing standards-based educational approaches with their emphasis on measurement and accountability without jeopardizing instructional practices that foster individual initiative and creativity. The literature describes numerous attempts that have been made to create a clear recitation of standards and related skills needed by school administrators to meet the challenges of their profession. A considerable body of work calls for new methods, such as the addition of formation activities to pre-service training, to help school leaders acquire needed skills.

Implications of Current "Reform" Efforts for School Leaders

Thomas Sergiovanni, in a keynote address at the January, 2002 New York State Conference of School Superintendents, stated that "standards and testing are like chlorine bleach; when used properly it works wonders but, when added to the wash alone, it can burn the life out of clothes." School leaders are faced with the responsibility to see that standards and tests are used in a positive manner. To be effective in promoting positive change, school leaders must understand the political nature of the debate over standards and its potential divisiveness. They are faced with the difficult task of ensuring that the debate results in positive outcomes for students and for other stakeholders of school districts and buildings. Because educational leaders serve multiple constituencies, they are expected to develop and implement practices to ensure the achievement of multiple purposes (Deal & Peterson, 1990). Without leadership, most people would agree that attempts at school reform are doomed (Bolman & Deal, 1994).

It is the complexity of educational enterprises and their importance to various constituencies that have brought questions about the role of leadership in school improvement to the forefront of the educational reform dialogue. In that dialogue, school leaders are subject to indictment as contributors to a myriad of problems including students who are deficient in basic skills, functional literacy, preparation for employment, higher level thinking skills, special subject area knowledge, citizenship, and responsibility (Murphy, 1992).

As the larger education community has become aware of and involved in discussions about the role of school leaders, it has become clear that role ambiguity and role overload of leaders are major issues that must be addressed by those who support standards-based school reform (Marsh, 1997). Bess and Goldman (2001) described educational leaders as being "caught in a zone of ambiguity." They observed that K-12 educational leaders are uncertain about the transition they are in, and they are unclear about what behaviors are appropriate in the transition stage or the next stage. To prepare school administrators to lead 21st Century schools, many efforts have been made to rethink and redefine educational leadership (Daresh, 1996; Glassman & Glassman, 1997; Leithwood, Jantzl, & Cattin, 1995; Marsh, 1997; McEwen, 1995; Murphy, 1992; Restine, 1997; Sharp, Walter & Sharp, 1998). Some initiatives have been focused on the identification of standards and competencies necessary for effective school leadership.

Standards and Competencies for School Leaders

A position statement approved by the National Association of Secondary School Principals Board of Directors on May 6, 2000 included the assertion that, "Leadership Development must be tied to meaningful assessment of leadership skills" (NASSP, 2000). Robert Millward (1998) supported this when he indicated that effective leadership programs have a clearly defined set of experiences that promote skill development across a wide range of tasks.

One of the first attempts to provide consistency in the content of school leadership programs was made in 1983 when the American Association of School Administrators (AASA) developed its Guidelines for the Preparation of School Administrators. It included seven "competencies and skills for goal accomplishment" applicable to all school leaders (Hoyle, 1987). In 1993, AASA established the Commission on Standards for the Superintendency, which developed eight standards and eighty-eight indicators of success for superintendents (Hoyle, 1993). The following year, the Council of Chief State School Officers formed the Interstate School Leaders Licensure Consortium (ISLLC) to develop standards for school leaders compatible with the National Council for the Accreditation of Teacher Education (NCATE) Curriculum Guidelines for school administration. The standards were released in 1996, and by 2002 over thirty states had adopted them as their blueprint for rethinking school leadership (Murphy, 2002).

Other states and several professional organizations drew upon the ISLLC standards and/or created their own initiatives to identify essential knowledge and skills required of school leaders. This led James Hoyle in 2001 to describe the various attempts to create standards as a "cacophonous crowd" with little harmony and no one listening. Others, such as Achilles and Price (2001), argued that the standards fail to address research and theory on education-specific knowledge needed by school leaders.

NCATE responded to this dissonance in 2001 by adopting eight generic standards and indicating that enabling skills, knowledge, and dispositions related to each standard should be tailored to specific positions of principals, central office staff, and superintendents. Thus, professional organizations, states, administrator preparation programs, leadership institutes, etc. are left to define specific indicators of competence for school administrators. They are also left to identify new and innovative processes to assist future leaders in developing competence.

Processes for Educating School Leaders

Traditional programs in educational administration have been criticized for their lack of adequate instruction on leadership theory and practice. The National Commission on Excellence in Educational Administration (1987) concluded that leadership preparation programs were marked by "lack of a definition of good educational leadership" (Milstein & Kruger, 1997, p. 100). Similarly, participants in a Policy Forum on Educational Leadership conducted by the United States Department of Education in 1999 criticized most university preparation programs for centering their curriculum on management, finance, legal issues, and other state-required content, with little emphasis being placed on leadership for instruction and school improvement issues (U.S. Department of Education Policy Brief, 1999).

One way to infuse leadership theory and practice into educational leadership preparation is through the inclusion of formation activities advocated by Daresh and Playko (1992). Their tridimensional model of administrator preparation includes attention to professional formation, strong academic preparation (i.e. university courses), and realistic guided practice (i.e. internships). Formation processes include leadership styles analysis, leadership development planning, platform development, mentoring, and reflection.

The concept of formation comes from the field of religious education, where it has been used to prepare individuals to assume roles as religious leaders. John Westerhoff (1987) of Duke University Divinity School described formation as a process that implies a kind of shaping whereby individuals are introduced to a broader understanding of the social realities of the world in which they will work and come to understand themselves more completely. Daresh and Playko saw formation as an applicable and essential factor to be considered in the preparation of educational leaders. This concept is often ignored in formal certification programs where it might most appropriately be considered a prerequisite of admissions to assist potential candidates to better understand school leadership endeavors and their own leadership interests and abilities.

A Formation Program

The formation program that is the subject of this study is the Southern Tier Leadership Academy, a collaborative effort of the New York State Education Department, Binghamton University, and local school superintendents in the greater Binghamton, New York area. It is a unique, innovative, regional approach to leadership development designed to identify teachers with high potential for leadership, stimulate their interest in educational administration, and provide them with experiences to enhance their understanding of leadership theory and its implications for practice. It is not designed as part of an administrative certification, or licensing program, but rather, as a catalyst to encourage potential leaders to consider pursuing leadership positions.

The process utilized in the Academy includes in-depth pre- and post-program professional assessment components through which each candidate receives an individual profile of strengths and weaknesses in relation to specific leadership competencies. In addition, participants are engaged in workshops on leadership theory and practice, mentoring by superintendents and building-level administrators, and opportunities for individual and group reflection. All of the Academy activities are designed to increase participants' leadership competencies to enable them to expand their role beyond management tasks. Participants are encouraged to become dynamic, inspirational leaders focused on continuous improvement. The components of the Academy and the approximate time frame of their implementation are illustrated in

The Academy was designed to increase participants' knowledge of educational leadership theory leadership theory and practice and their interest in school leadership positions. Furthermore, as a result of participation in the Academy experiences, the Academy designers assumed that participants would be able to demonstrate an increase in their use of leader behaviors in simulated educational leadership settings. The purpose of this study is to test these assumptions and to determine if a leadership formation experience, such as that offered by the Southern Tier Leadership Academy, can increase "high potential" teachers' leadership competencies and their interest in educational leadership.

Method

Sample

Table 2 provides basic information about Academy participants. The Academy was designed to serve a maximum of twenty-four participants in each eight-month program. This study addressed the experiences of the first three cohorts, or groups, of participants who completed the program.

Research Design

We used a mixed methods research design, employing both quantitative and qualitative measures. To ensure that the study was a comprehensive evaluation, we categorized our measures using Kirkpatrick's (1998) model of assessing training program effectiveness. Kirkpatrick's four-level framework is the best-known and most widely used model for the evaluation of training programs (Fisher, Schoenfeldt, & Shaw, 1999). We analyzed program outcomes at all four levels of Kirkpatrick's framework; reaction, learning, behavior, and results.

Measures

The quantitative measures used in this study included two reaction surveys, a measure for role-play assessments, a measure for the assessment of in-basket scenarios, the 14 Item Multifactor Leadership Questionnaire, and a questionnaire on educational and career plans. Each measure is described in Table 3 and the level of assessment from Kirkpatrick's framework is identified.

The role-plays and in-basket simulations were administered during the pre- and post-program assessments. The in-basket activities consisted of memos containing scenarios typically encountered by school administrators. The memos were provided to participants on computers using Microsoft Word and responses were typed into the same software program. The role-plays were one-on-one simulations performed with a member of the Academy staff. These role-plays were videotaped. Responses to the in-basket and role-play scenarios were assessed at a later date.

Qualitative data were collected using narrative response items on the reaction surveys. In addition, summaries of participants' leadership potential, which mentors and superintendents wrote at the conclusion of the Academy experiences, provided qualitative data. Participant journals were used to collect data specific to the mentoring program. Discussion of the journal data is not included in this article.

Findings

The major findings at each level of Kirkpatrick's framework are summarized below. In Kirkpatrick's framework, "learning" and "behavior" are usually assessed separately because the training occurs before there are opportunities for participants to demonstrate behavioral change. In the case of the Leadership Academy, however, the training continues over an eight-month period, and participants are given opportunities during that period to demonstrate learning and behavioral change. For this reason, learning and behavior are addressed simultaneously in this study. The study did not include measures of behavioral change beyond the eight-month training period.

Reaction

Post-program reaction data was collected using the Participants Reaction Survey and the Mentors and Superintendents Reaction Survey (see Table 3). Quantitative and qualitative reaction data was collected on each survey. Analyses of this data presented below demonstrate positive perceptions of participants, mentors and superintendents regarding the usefulness of the Academy activities.

Table 4 displays the mean reaction scores for participants in all three Academy cohorts for each activity. A one-sample t-test was used to compare the mean reaction scores to the positive reaction value of greater than 2.00. This comparison is also displayed on the table.

As shown in Table 2, the mean scores or all activities were positive and significant at the .001 or .05 level for all activities except the peer learning groups. The mean score for this activity was significantly below the neutral value.

Participants' narrative responses supported the quantitative response data. For example all of the fourteen comments submitted about the personal feedback session were positive, while eighteen of the twenty-two comments submitted about the peer learning groups were critical.

The largest number of narrative comments dealt with the mentoring portion of the Academy. Two-thirds of these comments mentioned that the mentors had been helpful. Specifically, participants said their mentors helped them:

* "translate theory into practice."

* "develop my leadership portfolio."

* "understand what it takes to be a leader."

* "by being my role model."

* "by providing support in my efforts to develop leadership skills and knowledge.

A recurring theme in the participant's narrative responses is their desire for more time and commitment from Academy personnel. This is evident in comments on:

* the value of one-on one attention in the personal feedback session,

* the need for additional assistance with leadership development plans,

* the need for additional or longer booster sessions, and

* the lack of commitment on the part of some mentors and superintendents.

The data collected from superintendents and mentors demonstrated a positive reaction to the Academy. On the ten items requiring a scale score response, all of the mean reaction scores of the mentors and superintendents were positive and significant at the .001 level.

Learning and Behavior

Learning and behavior change was assessed using pre- and post-program quantitative data collected in the role-play and in-basket activities of the Assessment Center. Additional quantitative data was collected using the 14-Item MLQ (Multifactor Leadership Questionnaire).

Multiple methods were used to measure learning and behavior change for three transformational competencies (idealized influence & inspirational motivation, intellectual stimulation, and individualized consideration) and one transactional competency (contingent reward) of Bass and Avoilo's (1993) leadership model. Participants' leadership behaviors were measured using Assessment Center pre- and post- program in-basket and role-play techniques. Each item in the Assessment Center was rated on a scale of 1 (Low) to 3 (High). Each role-play was assessed using a measure consisting of four items on each transformational and transactional competency. The number of items varied in the in-basket measures from a low of two items for inspirational motivation and idealized influence to twelve items for decisiveness. The scores reported were mean scores for each competency.

The 14-item MLQ was also used to measure transformational and transactional competencies. On this measure respondents indicate how often they have observed specific behaviors exhibited by the participants on a five-item scale from 0 (Not at All) to 4 (Frequently). This was utilized with Cohort 3 participants (N=19) as a pre- and post-program self-assessment measure and with Cohort 1 and 2 participants (N=18) as a post-program self-assessment measure. It was also utilized as a post program measure of participants' leadership behaviors, as observed by mentors and superintendents.

Table 5 displays the results obtained for one of the four leadership constructs, contingent reward. These results exemplify the findings associated with transformational and transactional competencies.

Most of the mean score results indicate positive growth in participants' ratings on the transformational and transactional competencies and the t-tests show significant (p < .02) results across multiple methods and multiple raters.

The results pertaining to the transformational leadership competencies of individualized consideration and intellectual stimulation differ according to the measure utilized. In both cases, the in-basket and MLQ evaluations demonstrated growth on the part of the participants and the role-play evaluations did not demonstrate this growth. This seems to support research findings of Sacket and Dreher (1982) who concluded that assessment center techniques have not proven to accurately measure cross-situational abilities. However, Neidig and Neidig (1984) pointed out that properly designed situational exercises purposely place assessees in a variety of job-related contexts, and therefore, stable performance across exercises is not necessarily expected. Accordingly, a leader may demonstrate high levels of a competence in one job-related context, but not in another.

With some exceptions, the alpha values cause concern regarding the reliability of the measures in measuring the identified constructs. To some extent, these results are not surprising or particularly alarming. The scales utilized in the role-play and in-basket activities were newly created and had not previously been tested empirically. The developers of the Academy experiences, like those of many other Assessment Centers (Neidig & Neidig, 1984), have relied upon content validity to support the validity of the constructs measured. While the significance levels of the results suggest that individual items were valid, it is clear that the Academy measures need further development to ensure that the constructs being measured are accurately identified.

The low reliability scores of the 14-Item MLQ measures are concerning. This is a shortened version of the Multifactor Leadership Questionnaire (Form 5X), which consists of thirty-six items to measure leadership components of Bass and Avolio's (1993) model of transformational and transactional leadership. The 14-item version was used due to time constraints faced by Academy participants, mentors, and superintendents. This version has previously had limited use as a research instrument.

One method was used to measure other basic leadership competencies addressed in the Academy. Participants' leadership behaviors were measured using Assessment Center pre- and post- program in-basket techniques. Each item in the in-basket evaluation was rated on a scale of 1 (Low) to 3 (High). The scores reported were mean scores for each competency. Table 6 displays the results obtained for each competency.

All of the mean score results indicate positive growth in participants' ratings on the competencies measured. The t-values indicate that the results are significant.

As was the case with the measurement of transformational and transactional leadership competencies, the in-basket measures of these competencies demonstrated low reliability. One possible reason for this is that too many ability-related behaviors are being judged in one exercise. Bycio, Alvares, and Hahn (1987, p.473) pointed out that, "even if exercises were developed so that a large number of ability-relevant behaviors were reliably elicited, we cannot assume that assessors could observe, record, and aggregate them all."

Results

Using the Educational and Career Plans Survey, we collected post-program data about the impact of the Academy on participants' educational and career plans. The following items of the survey were used to collect data regarding participants' interest in pursuing certification in educational administration:

Item 1 - Were you enrolled in an educational administration certification program prior to your participation in the Leadership Academy?

Item 3 - Did your experiences in the Leadership Academy influence you in such a way that it increased the likelihood that you would pursue, or continue to pursue certification as a school administrator?

A total of thirty-five participants responded to these items on the survey. Fifteen of them reported on Item 1 that they were not enrolled in a certification program prior to their Leadership Academy experiences. Of those fifteen, eleven indicated on Item 3 that the Academy increased the likelihood that they would pursue certification. In total, twenty-eight of the thirty-five respondents to Item 3 reported that the Leadership Academy experiences had increased their likelihood of pursuing, or continuing to pursue certification.

We calculated a Chi-Square to test these results against the null hypothesis that the results were random. Table 7 illustrates the results of this analysis.

As shown above, the null hypothesis is rejected and the results from the participants' responses are significant at the .001 level.

The following items were used to collect data on the willingness of participants' to take on additional leadership responsibilities:

Item 8: Have you taken on additional leadership tasks during or after your participation in the Leadership Academy?

Item 9: If you have taken on additional leadership tasks during and/or after your participation in the Leadership Academy, were your decisions to take on theses tasks in whole, or in part due to your participation in the Academy?

A total of thirty-four participants responded to Item 8, twenty-three reporting that they had taken on additional leadership responsibilities and eleven reporting that they had not. Of the twenty-three who reported that they had taken on additional responsibilities, nineteen of them indicated on Item 9 that their decisions to do so were due, at least in part, to their participation in the Academy.

We calculated a Chi-Square to test these results against the null hypothesis that the results were random. Table 8 illustrates the results of this analysis.

As shown above, the null hypothesis is rejected and the results from the participants' responses are significant at the .005 level.

Some qualitative data also provided information pertaining to the effectiveness of the Academy at Kirkpatrick's Level Four (Results). At the conclusion of the Academy experiences, mentors and superintendents were asked to write a short summary of their participants' leadership potential. We analyzed the summaries and found a total of thirty-eight comments regarding the effectiveness of the Academy.

Of these thirty-eight comments, sixteen specifically addressed the impact of the Academy on participants' decisions about pursuing certification as school administrators. Ten of the comments indicated that mentors and superintendents believed the Academy experiences had influenced some individuals to pursue certification. The following is an example of these comments.

* "_____ has taken the Academy experience very seriously and has become a student of leadership. He is now interested in beginning a certification course."

The other six comments indicated that mentors and superintendents believe the Academy experiences convinced some individuals that they should not pursue administrative certification or careers at this time. The following is an example of these comments.

* "At the beginning, she was somewhat nave when it came to understanding what administrators do. I think the Academy experience has led her to postpone entering the field until later in life."

These comments suggest that the Academy experience was useful in helping some individuals determine that they are not suited for leadership positions, or that they weren't ready to pursue leadership. This is a valuable effect of the Academy.

Discussion

There are several implications of the current study for the Southern Tier Leadership Academy and other leadership development efforts. Theoretically, the study provides support for the use of Daresh and Playko's (1992) tridimensional model of leadership development. In particular, the study demonstrated that professional formation activities can be useful in the pre-service education of school administrators. This was evident in significant positive results obtained on almost all measures. For example, the reaction data was highly significant and demonstrated strongly positive perceptions of participants, mentors and superintendents regarding the usefulness of the activities. The data associated with learning and behavior change indicated that participants had improved their performance of tasks related to the roles of school administrators. Further work is required to determine if formation activities are most useful as stand-alone opportunities for individuals who want to obtain a greater understanding of the field or as an essential component of school leadership preparation programs.

Methodologically, there are general implications associated with the multimethod research approach and implications that are specific to assessment center techniques. While it is important to note that the use of multiple methods makes it impossible to determine the effectiveness of one particular method, future inclusion of both quantitative and qualitative research measures in a single study shows great promise. Researchers can no longer rely solely on one method for all measurements (Dinger, 2001). Of particular interest in this study are the measures of participants' interest in pursuing administrative certification. Quantitative results indicated that the Academy experiences had increased the likelihood that many participants would pursue education leading to certifications. Qualitative results provided further information indicating that the Academy activities had also led some individuals to decide that they would not pursue administrative certification or careers at this time. Both results demonstrate positive impacts of the Academy on participants. Multimethod research techniques helped bring these results to light.

The value of multimethod research was also demonstrated by the results associated with the participants' reactions to specific Academy activities. The quantitative results demonstrated positive reactions to most of the activities and the qualitative results provided more specific information about why participants felt the activities were useful. In addition, the qualitative results identified specific aspects of the activities to which participants reacted negatively and specific suggestions for improvement.

Implications specific to assessment center techniques relate to the reliability of the measures. The convergence of positive results from multiple measures of leadership competencies seems to support the notion that the individuals had improved in their handling of job-related tasks. Accordingly, this supports the contention of Neidig and Neidig (1984) that the value of assessment centers lies in content validity resulting from thorough job analyses and careful exercise design. However, the low reliability of the measures supports the contention of Sackett and Dreher (1982) that, "no published research has shown that the trait ratings produced in managerial assessment centers accurately reflect the complex traits users purport to measure"(p.402).

From their study of three assessment centers, Bycio, Alvares, and Hahn (1987) concluded that the reliability of assessment centers could be improved by reducing the number of constructs that were typically measured and increasing the number of items used to measure each construct. The results of this study support this conclusion. The most reliable measure was the pre-program role-play measure. This measure was limited to four constructs and included items in two, one-hour simulations. The post-program measure, which was less reliable, measured the same four constructs with fewer items in one, one-hour simulation. The in-basket measures, which also were less reliable, were used to measure eight different constructs using a small number of items on each construct. Further research is required to determine if the use of fewer constructs and more items will result in greater reliability both within and across assessment center exercises.

There are also several practical implications from the current study. First, the study provides evidence to support the use of formation activities from a practical perspective as well as from the theoretical perspective discussed earlier. The results indicated that the program increased the likelihood that participants would pursue certification as school administrators, potentially increasing the pool of qualified candidates. In addition, the results demonstrated that formation activities may have an immediate positive effect on schools in that participants reported that they had accepted additional leadership responsibilities. Positive scores from the Assessment Center suggest that participants have improved their abilities to address tasks associated with these responsibilities. Overall, the results suggest that formation activities can be useful in helping prospective leaders to learn about leadership theory and practice and to identify and develop competencies necessary to address leadership challenges and opportunities found in current school environments.

The results also demonstrated that the formation activities convinced some participants that they should not pursue certification. Accordingly, some individuals may be less likely to spend time, effort and financial resources to obtain a certificate they never use. Schools would also be likely to save resources since they often provide financial assistance in the form of tuition payments, educational salary credits, and paid internships to those seeking administrative certification.

Summary

This study is an effort to investigate the effectiveness of one leadership development initiative, which is designed to provide professional formation activities to prospective school leaders. The data suggest that the program was effective at all levels of Kirkpatrick's framework for assessing training program effectiveness. It also suggests that formation activities can be useful in addressing issues related to the quantity and quality of candidates for school leadership positions. The findings may be useful in the development, implementation, and evaluation of other leadership development endeavors.