A Critical Examination of the Impacts and Lessons Learned from a Professional Development Program for Out-of-Field Mathematics Teachers

As international concerns about the prevalence of out-of-field teaching have grown, so have discussions about how to support out-of-field teachers. In Ireland, the Professional Diploma in Mathematics for Teaching, a two-year professional development program, was created for out-of-field mathematics teachers. A pre-test, post-test, and final survey examined the program’s impact on participating teachers’ mathematical knowledge, confidence in teaching curricular content, and classroom practice. Findings offer evidence of development in participating teachers’ mathematical knowledge and self-efficacy after completing the program. They also raise important concerns about persistent weaknesses in participating teachers’ mathematical knowledge, particularly related to key areas of the curriculum.


Introduction
International concerns about the prevalence of out-of-field teaching have prompted calls for more research on the issue, including research specifically focused on out-of-field mathematics teaching (Akiba et al., 2007). As defined by Ingersoll (2002), an out-of-field mathematics teacher is a qualified teacher who is teaching mathematics without the required expertise in mathematics content or mathematics-specific pedagogy. As concerns have grown, so have discussions about how to support these teachers (Goos et al., 2020;Hobbs, 2012;Ingersoll, 2001).
In the Irish context, Ní Ríordáin and Hannigan (2011) revealed that, based on the Irish Teaching Council's qualification requirements, 48% of Ireland's secondary mathematics teachers were out-of-field. They also found that out-of-field teachers were primarily assigned to foundational courses and students with previously low achievement levels. In response to these findings, the Irish Government funded the establishment of the Postgraduate Diploma in Mathematics for Teaching (PDMT), a national, blended-learning professional development program designed to offer mathematical content and pedagogy for out-of-field mathematics teachers.
The nationwide implementation of the PDMT and its cohort model provided a unique opportunity to examine the outof-field mathematics teachers' knowledge development and professional growth over the course of the program. Given that most of the out-of-field teachers had limited studies of mathematics content and pedagogy in their initial pathway to teacher certification, Ní Ríordáin et al. (2017) began with an examination of the mathematical knowledge and confidence levels of these teachers prior to beginning the PDMT. Their findings showed that, upon enrolment in the program, the out-of-field teachers demonstrated weak proficiency levels and a high rate of conceptual errors with curriculum-aligned mathematical content. Using this as a baseline, the research reported here builds on the findings of Ní Ríordáin et al. (2017) with the aim of examining the impact of the PDMT on participating teachers' mathematical knowledge and self-efficacy, both of which have been linked to teacher practice (Desimone, 2009;Goldsmith et al., 2014;Sharplin, 2014). Researchers have identified mathematical knowledge as an influential factor in teachers' Additional research has been conducted separately on the teachers' experiences with the PDMT and its defining features, including its blended-learning model, content, delivery and practices. For example, Lane and Ní Ríordáin (2020) examined the action research reports produced by graduates and found evidence of teacher development in the form of a majority shift towards constructivist beliefs and practices. Goos et al. (2020) identify the professional development needs of out-of-field mathematics teachers and theorize the affordances and characteristics of effective blended learning programs to support such teachers and their needs.

Mathematical knowledge
In setting out to measure the program's impact, the researchers were cognizant of documented difficulties in implementing deep change in teachers' knowledge through professional development (Carney et al., 2016). While teacher learning has been shown to occur incrementally and iteratively, there have also been calls for more research with a stronger focus on teacher learning (Goldsmith et al., 2014). More specifically, there is a widely recognized need for continued exploration of ways to improve out-of-field mathematics teachers' mathematical knowledge for teaching (MKT), which includes both subject matter knowledge and pedagogical content knowledge (Ball et al., 2008;Ní Ríordáin et al., 2017;Nixon et al., 2016;Sharplin, 2014).
An array of theoretical frameworks concerning the knowledge required for teaching mathematics have been developed by key researchers in the area (e.g. Ball et al., 2008;Davis & Renert, 2013;Rowland & Ruthven, 2011;Tatto, et al., 2012). Such models provide guidance for designing experiences that target teacher knowledge development (Chapman, 2013), and particularly in this context, thinking about the development out-of-field mathematics teachers. Both the design of the PDMT and research on its impact (Ní Ríordáin et al., 2017) drew on the MKT framework offered by Ball et al. (2008), with a particular focus on specialized content knowledge (SCK). The pedagogical workshops and summer institute also reflected a focus on knowledge of content and curriculum (KCC) and knowledge of content and teaching (KCT).
In designing this study, the researchers were aware of concerns raised about the challenges of measuring teacher knowledge (Fauskanger, 2015), but also of the established need for such work. While research has shown that direct measures of teachers' mathematical knowledge growth can predict improved student achievement, there have been relatively few empirical studies focused specifically on quantitative measures of teachers' knowledge development (Goldsmith et al., 2014;Hill & Ball, 2004). Ultimately, the researchers chose to measure teachers' mathematical knowledge in close alignment with the Irish curriculum, a strategy that was informed by the work of Krauss et al., (2008) and Schmidt et al., (2008) in which measuring subject matter knowledge involves examining the mathematics content of the intended curriculum in a given context (Adler & Venkat, 2014).
While the external influence of national policies regarding what mathematics is required to qualify as a secondary teacher shaped the content of the program, research has raised questions about whether advanced mathematics is really what teachers need to teach at the secondary level (Ball et al., 2008;Hill & Ball, 2004;Silverman & Thompson, 2008). Given that all participants were teaching mathematics while completing the PDMT, the researchers chose to measure how effectively the PDMT's undergraduate mathematics content could effectively address its teachers' needs with regard to the content of the curriculum they teach.
Ní Ríordáin et al.'s (2017) baseline measures showed plenty of room for growth with participating out-of-field teachers exhibiting low levels of proficiency with curriculum-aligned mathematical content and high occurrence rates for conceptual errors and evidence of incomplete understanding prior to participation in the program. Despite research demonstrating the impact of teachers' subject matter knowledge on their practice, there are surprisingly few studies specifically examining out-of-field mathematics teachers' knowledge development, particularly through large-scale and long-term professional development programs. This study was designed to address this gap.

Self-efficacy
Notwithstanding a clear need for further research focused on mathematical knowledge development for out-of-field mathematics teachers, Chapman (2013) warns that exclusively focusing on a perceived "set of knowledge" (p. 238) can limit an important examination of other aspects of out-of-field teachers' growth through professional development. Thus, considering other aspects, such as self-efficacy, can also be valuable. Bandura's (1997) influential research provides an important understanding of the types and sources of self-efficacy and potential impact on teachers and their actions. It examines the impact of mastery experience, vicarious experience, social persuasion and affective states on cognitive, motivational, affective, and selection processes.
The importance of teacher self-efficacy for practice is further supported by the research of Woolfolk discussed in Shaughnessy (2004) in which teachers with "a high sense of efficacy" and a willingness to act on it were "more likely to have students who learn" (pp. 156-157). Hattie (2012) points to collective teacher self-efficacy as having the greatest impact on student achievement when considered in conjunction with teacher practice. In particular, research has found that students of teachers in a given school who believe, as a whole that they can impact student knowledge, tend to be more successful (Goddard et al., 2000). In Ireland, out-of-field teaching is not limited to an individual teacher within a given school -rather a significant proportion of teachers teaching mathematics in any school are out-of-field. Therefore, a positive impact on the PDMT's out-of-field teachers' self-efficacy has the potential to widely enhance practice and student learning across classrooms as a collective within Irish secondary schools.
In the context of this study, self-efficacy is examined from the perspective of teachers' perceptions of their ability to both effectively teach the content of the mathematics curriculum (Carney et al., 2016) and incorporate new strategies into their practice. In the out-of-field context, when teachers are assigned to teach subjects which do not match their qualifications (Ingersoll, 2002), they must be provided with supports to help develop their competence and confidence in ways that can reduce the potential negative impact of their own limited mathematics education on their learners (Sharplin, 2014). In addition, several researchers have linked teachers' self-efficacy with their openness to and persistence with incorporating new and innovative practices in their teaching (Bitto & Butler, 2010;Gabriele & Joram, 2007). Therefore, it is particularly important to examine such concepts in relation to out-of-field teacher education (Hobbs, 2013), especially given the prevalence and complexity of the issue (Hobbs & Törner, 2019).

Methodology
Three instruments were used to collect both pre-and post-program data. An initial online survey was completed by participating teachers prior to enrolment to gather background information and self-reported confidence levels related to the content of the Irish curriculum (Ní Ríordáin et al., 2017). Then, a paper-and-pencil test designed to examine knowledge of mathematical concepts in the Irish curriculum was administered both prior to and upon completion of the program. A final online survey completed at the end of the program offered participants a chance to report their perceptions of the program's impact on their mathematical knowledge development and self-efficacy.

Paper-and-Pencil Test
The conceptual framework for the Teacher Education and Development Study in Mathematics (TEDS-M) helped to inform item design for the paper-and-pencil test (Ní Ríordáin et al., 2017;Tatto et al., 2008). The test focused specifically on assessing specialized content knowledge (SCK) that was closely aligned with the Irish secondary mathematics curriculum (Ball et al., 2008;Krauss et al., 2008;Schmidt et al., 2008). Andrews (2011) endorses such alignment with the curriculum of the teachers in the study and/or the country in which the research is conducted. In this case, alignment of the Irish curriculum with the PISA mathematical framework also ensures a level of consistency with international expectations for what teachers must know to be able to teach secondary mathematics (Kirwan, 2017;Merriman et al., 2014). Accordingly, all 24 items were designed to specifically measure knowledge of the content that an Irish mathematics teacher is required to teach (Ní Ríordáin et al., 2017).
The test was piloted with a group of 28 preservice and practicing teachers. The results were then discussed in detail with these participants to ensure that they accurately captured their conceptions and ideas, both correct and incorrect or incomplete. This was also an important process for establishing the validity and reliability of this instrument.
The test assessed concepts from all five strands of the Irish secondary mathematics curriculum prior to 2018 ‡ (Statistics and Probability, Geometry and Trigonometry, Number, Algebra, Functions) at two levels -Junior Certificate (JC) and Leaving Certificate (LC). The JC curriculum includes the first three years of post-primary education in Ireland. ‡ While the curriculum was restructured in 2018 to include a Unifying Strand, the content has not changed.
The LC curriculum includes two additional years of secondary mathematics culminating in a terminal exam. Certain items were specifically designed to determine whether the teachers held common misconceptions that would make them more likely to allow such misconceptions to persist and go unaddressed in their classrooms. Others assessed the teachers' ability to synthesise their knowledge across areas of the curriculum. Ní Ríordáin et al. (2017) provide further details regarding the instrument design.
Each item on the paper-and-pencil test was scored twice. First, each item received what the researchers called a cognitive score, which indicated the degree to which the response to that item was correct. Each teacher also received an overall cognitive score -their total points earned across all items expressed as a percentage of the total possible points that could be earned on the test. This helped to give an overview of the teachers' general proficiency with the content on the test.
The cognitive scores for single items were also examined collectively to determine a proficiency rate for each item -the mean score for that item presented as a percentage of the total possible score. For example, a multiple-choice item (scored either 0 or 1) with an overall mean score of 0.68 would have a proficiency rate of 68%. An open-ended question (scored out of 3 possible points) with a mean score of 1.8 would have a proficiency rate of 1.8 out of 3 or 60%. While this is also known as item difficulty, the researchers chose to use the term proficiency rate to focus less on the item and more on participants' proficiency with the mathematical content being assessed (Ní Ríordáin et al., 2017).
The second scoring process analysed solutions to each item for evidence of incorrect or incomplete understanding of the relevant mathematical concepts to determine what the researchers called a conceptual score. This score considered the mathematical concepts involved in the teacher's approach to completing a test item and then determined whether there was any evidence of an incorrect or incomplete understanding of each of these concepts. For each item, the concepts involved with the teacher's chosen solution were given a score of 1 if there was evidence of an incorrect or incomplete understanding of that particular concept and a 0 if the item response offered no such evidence. So if a teacher's chosen approach to completing an item involved 3 different concepts, they could earn a score for that item of either 0/3, 1/3, 2/3, or 3/3, with 0/3 indicating no evidence of incorrect or incomplete understanding of any of the concepts, 1/3 indicating some evidence of incorrect or incomplete understanding of one of the concepts, etc.
Each participant was given an overall conceptual score, expressed as an occurrence rate for either conceptual errors or evidence of incomplete conceptual understanding. To account for different approaches to solving problems, a teacher's conceptual occurrence rate was calculated based only on the number of concepts involved in his or her particular approach to finding solutions for test items. For instance, two teachers that approached the same item in different ways may have drawn on knowledge of a different set (and number) of concepts for their solutions. In this case, if each teacher had evidence of incorrect or incomplete understanding of one concept, but one teacher's approach required knowledge of two concepts while the other's approach required knowledge of four concepts, the first would receive a conceptual score of 1/2, while the second would receive a conceptual score of 1/4. While a mean overall occurrence rate was calculated for the cohort, mean occurrence rates for individual test items were not generally examined because the number of concepts required to solve each item varied from teacher to teacher based on their approach to finding a solution. Ní Ríordáin et al. (2017) offer a more in-depth discussion of this approach to scoring. Given that the cognitive scores are a measure of proficiency and conceptual scores reflect the occurrence of errors or evidence of incomplete understanding, a teachers' knowledge of the mathematical concepts being tested was seen to have improved with an increase in their cognitive score and a decrease in their conceptual score.
Separate scoring tools were used for the cognitive and conceptual scoring processes. Finalizing these scoring tools involved an iterative process in which two researchers scored batches of ten tests at a time, comparing their scores after each batch until a target agreement level was reached for each template. For the cognitive scoring template, there were 52 possible points. Of the 24 items, 14 were considered "debatable," meaning they were open-ended items. These open-ended items were scored out of 3 possible points, while the remaining 10 multiple-choice items were either right or wrong and given a score of either 1 or 0, respectively. The researchers used an agreement percentage target of 86%, based on allowance of 2 discrepancies per test with a maximum rating difference for any single item of 1 point.
In the conceptual scoring template, there were a total of 65 potential concepts across all items. For each concept, the researchers had to make a scoring decision -giving the teacher a 1 if there was evidence of incorrect or incomplete understanding of that concept, a 0 if no such evidence was present, and an NA if the teacher chose to complete the item in a way that did not require knowledge of that concept. So, even if a potential concept was not part of a teacher's solution, the assignment of NA to that concept was considered a scoring decision. The researchers used an agreement percentage target of 95%, based on allowance of 3 discrepancies per test.
The PDMT's cohort model enabled a comparison of data collected from the paper-and-pencil test taken prior to beginning the PDMT with data collected from this same test taken two years later, upon completion of the program. The pre-test sample consisted of all teachers initially enrolled in the PDMT (n=280). The post-test sample included the teachers from this initial cohort that completed the program (n=249). Unique identifiers generated for all participants facilitated alignment between data collection instruments and the resulting pre-and post-program data. Response rates varied between pre-testing (n=202) and post-testing (n=131).
To adjust for the lower post-test response rate, the researchers did two comparisons of pre-and post-test results. The first was a comparison of the full collection of pre-and post-tests (n=202,131). The second was a comparison of a subset of 80 of these participants for which the researchers could be sure that they had both pre-and post-tests. To address concerns about the relatively small size of this subset compared to the overall number of participants, the researchers compared the overall change in scores (from pre-test to post-test) for this subset of 80 participants with the overall change in scores for the complete set of pre-tests (n=202) and the complete set of post-tests (n=131). This was done to determine whether scores from the full set of pre-and post-tests could be considered representative of the findings for participants that we knew for sure took both the pre-and post-test. Table 1 presents both sets of results with values reported as percentages. The pre-test to post-test improvements of 11.5 points and 10.3 points in the full sample and overlapping sample, respectively, correspond to effect size differences of 0.81 and 0.70 standard deviations. A paired samples test conducted with the overlapping samples showed that the 10.3 point difference between the pre-and post-test scores, 95% CI [13.1, 7.7], was statistically significant, t(79) = 7.7, p < .001. The same was found to be true for the changes in conceptual scores, presented in Table 2. The pre-test to post-test conceptual score decreases of 9 points and 9.3 points in the full sample and overlapping sample, respectively, correspond to effect size differences of 0.66 and 0.65 standard deviations. A paired samples test conducted with the overlapping samples showed that the 9 point difference between the pre-and post-test scores, 95% CI [5.7, 12.2], was statistically significant, t(79) = 5.5, p < .001. Thus, with both samples showing statistically significant differences between the pre-test and post-test results, and such a small difference found when comparing the changes in mean scores between the full set and the subset, especially for the conceptual scores, the full pre-test sample and the full post-test sample will be used for all further discussion of the findings from the paper-and-pencil test.

Online Surveys
Participants were also asked to complete an initial online survey prior to beginning the PDMT and a final survey upon completion of the program. Unlike the pre-and post-paper-and-pencil tests, the initial survey and final survey items were not the same, but both surveys did elicit information regarding teachers' self-efficacy and other dispositions. In particular, the final survey asked teachers to rate the impact of the program on their confidence in teaching the main concepts within each area of the curriculum. It also asked teachers to indicate the immediate impact and/or potential future impact of the PDMT on their classroom practice, including their mathematical knowledge, pedagogical choices, and overall improvement in teaching mathematics.
The initial online survey focused primarily on teachers' self-reported confidence levels with regard to teaching curricular content at both the JC and LC levels. Ní Ríordáin et al. (2017) explain that while the teachers' self-reported confidence levels on the initial survey were high, a comparison with the initial paper-and-pencil test results revealed that high confidence levels did not necessarily mean strong content knowledge. In fact, some items revealed high confidence levels in teachers who also demonstrated erroneous or incomplete conceptual understanding of the content, including misconceptions commonly seen among mathematics students.
Given this demonstrated unreliability of teachers' self-reported confidence as a predictor of them having the content knowledge needed to teach the curriculum, the post-program survey focused on asking teachers to report any growth that they directly attributed to the PDMT. The items specifically asked about changes in their confidence with the mathematical content and their self-efficacy related to incorporating newly-learned strategies into their teaching and action research into their classrooms. As a result, discussion of the survey results will not attempt to draw comparisons between the initial survey and the final survey. Instead, it will primarily focus on the results of the final survey. Ní Ríordáin et al. (2017) offer a detailed discussion of the initial survey results which will be referenced where relevant in the interpretation of the findings.
Despite the final survey and post-test being administered to the same sample of 240 teachers, the response rates differed with 125 teachers completing the online survey. Within these two subsets, only 57 of the teachers overlapped, meaning, they completed both the final survey and the post-test. While this was a disappointing result, the researchers still believe that it is valuable to discuss the survey results and will discuss them separately from the pre-and post-test results, and they will serve as a separate but related set of findings that offer the teachers' perceptions of the impact of the program. Findings When considered in conjunction with the results of the pre-test, the results of the post-test and final survey each distinctly offer evidence of teacher growth and development resulting from the professional development program. However, the findings also raise some important concerns.

Post-test results: Impact on teachers' mathematical knowledge
Comparing cognitive and conceptual scores from the paper-and-pencil test taken prior to enrolment in the PDMT with those from the same test administered at the end of the PDMT provides an opportunity to evaluate the program's impact on the participating teachers' mathematical knowledge. These cognitive and conceptual scores are presented in Tables 3 and 4, respectively.  Overall, the mean cognitive score increased by 11.5 percentage points and the occurrence rate of conceptual errors or evidence of incomplete conceptual understanding decreased by 9.3 percentage points. These results show improvement in this cohort of out-of-field mathematics teachers' ability to correctly solve the mathematical questions contained in the test. The appearance of statistically similar changes from the paired samples t-test in the sub-sample for which pre-test and post-test scores were compared for the same set of participants (n=80) helps to address potential speculation that these changes should be attributed to attrition of weaker participants from the pre-test to the post-test (see Tables 1 and 2). Both the 10.3 point difference between the pre-and post-test cognitive scores (95% CI [13.1, 7.7], t(79) = 7.7, p < .001) and the 9 point difference between the pre-and post-test conceptual scores (95% CI [5.7, 12.2], t(79) = 5.5, p < .001) are statistically significant, with effect sizes corresponding to 0.81 and 0.66 standard deviations, respectively.
Also notable is the fact that the highest cognitive score on the pre-test was 69.2%, and over 7% of participants that completed the post-test exceeded this previous maximum score, with two candidates earning a new high score of 90.4%, an increase of over 20 percentage points.
Despite these improvements, a mean cognitive score of 46.1% on a mathematical content test aligned with the curriculum these teachers are expected to teach is still very worrying when considering their effectiveness. Such a low achievement level on a content test is of particular concern when it persists even after the completion of a two-year program specifically designed to target the development of teacher knowledge. This presents a strong motivation for further analysis of the individual items to identify areas where gaps or errors in teachers' knowledge seem to be most persistent . Table 5 offers an overview of the changes in scores for each item. Tables 6 and 7 contextualise these changes with details on which concepts were classified as cohort strengths and weaknesses upon completion of the PDMT (values are presented as percentages). Table 5 presents a range of changes in the cohort's ability to answer particular test items, indicating varying impacts of the PDMT on the out-of-field teachers' knowledge of particular curricular concepts. The largest change in proficiency rates was for the item on sequences and series. This has anecdotally been an area of the curriculum for which, in the past, students have entered undergraduate programs with very limited knowledge. Teachers' avoidance of the topic is often cited as the reason, despite the fact that it has been assessed by the Leaving Certificate exam, Ireland's terminal exam taken upon completion of secondary education. This offers an example of the importance of considering context in the design of professional development for out-of-field teachers, in order to maximize impact. Other variables, including feedback on the effectiveness of instructors for particular modules can also help to explain discrepancies in program impact across the curricular concepts.  While improvement in the proficiency rates for some items meant the cohort developed new strengths in their knowledge of mathematical concepts, for many items, the extent of the teachers' initial weakness in a particular area meant that, despite notable improvement, continued development is still needed in that area. Tables 6 and 7 help to better position the changes in scores with further information on cohort strengths and weaknesses. In Table 6, as in Ní Ríordáin et al. (2017), concepts are identified as strengths if their proficiency rate is at or above 50%, because in the Irish system, a mark of 50% or higher qualifies as an honours distinction. Similarly, Table 7 identifies any item with a proficiency rate below 40% as a weakness, because in the Irish context, anything below 40% is not considered a passing mark. While comparison of pre-test and post-test scores shows improvement in the teachers' ability to correctly answer all items in this collection of cohort strengths, the pre-test scores show that most of these items were already considered cohort strengths prior to enrolment in the PDMT. However, the final two items (concepts in number systems and counting) were originally considered cohort weaknesses. The substantial improvement in the cohort's ability to answer these two items elevated them from a weakness to strengths, suggesting that the PDMT had an important impact on the out-of-field teachers' knowledge of these concepts.
It is also important to note that six of eight topics in this table involve Junior Certificate content, which is the level that out-of-field teachers are typically teaching. This suggests that experience with teaching the content prior to engaging in the program may have played a more significant role in some of these topics being strengths than the PDMT. This is not something that can be determined by the data collected in this study. Several of the items in Table 7 show improvement; however, the cohort's persistent struggle with these items, even after completing the PDMT, highlights areas in which the program fell short of improving out-of-field teachers' knowledge of important curricular concepts. For this finding, it is important to note that the Irish Teaching Council mandated that the PDMT include a sequence of undergraduate courses equivalent to the regular licensure requirements for a secondary mathematics teacher in Ireland. This meant that, although the instrument used to assess the teachers' mathematical knowledge was closely aligned with the Irish secondary curriculum, the PDMT's content modules was more closely aligned with the content of an undergraduate mathematics degree. Additionally, the prevalence of Leaving Certificate level topics on this list would seem attributable to the fact the they are higher-level concepts, but again, it may also be attributable to the fact that out-of-field teachers in Ireland are more commonly assigned to Junior Certificate classes and these teachers were less likely to have any experience teaching mathematics at the Leaving Certificate level.
While changes in standard deviation are also apparent in both tables, discussion of the findings will focus primarily on overall changes in the proficiency rates rather than in the level of variance for these rates. While an analysis of changes in standard deviations would also support an interesting discussion, it is outside the scope of the analysis presented in this paper.

Final survey results: Teacher-reported impact on knowledge development
In addition to the paper-and-pencil test, an online survey item gave teachers an opportunity to rate the effectiveness of the program with regard to development of their mathematical content knowledge. Table 8 presents the percentages of teachers who reported that the program was either effective or very effective in supporting their development in the following ways: When considered in the context of the results of the pre-and post-mathematical knowledge test, the increase in the mean overall cognitive score confirms the development reported by the teachers on this survey item. However, persistent weaknesses in key aspects of the curriculum are still evident in the relatively low level of post-test cognitive achievement reflected in Table 7 and both the overall cognitive scores and conceptual occurrence rates presented in Tables 3 and 4.

Final survey results: Teacher-reported impact on self-efficacy
In addition to the perceived impact on teachers' mathematical knowledge, the final online survey also asked teachers to evaluate how the program had impacted their confidence levels for teaching specific areas of the curriculum and their perceived improvement in teaching mathematics. For nine of the ten PDMT mathematics content modules, the percentage of participants indicating that the module had either some impact or a significant impact on their teaching was between 66.0% and 91.5%. The highest impact was attributed to the Statistics module, while the lowest module (and the one below 66%) was Calculus III. While this is encouraging, it also means that for some modules, over one third of the teachers that completed the survey felt that it had either a slight impact or no impact on their teaching. This was likely related to the perceived relevance and connection of the content to the secondary curriculum.
This was further examined in another survey question that specifically asked the teachers about the program's impact on their confidence in teaching each of the topic areas in the Junior Certificate and the Leaving Certificate curricula. While high confidence levels from the initial survey (Ní Ríordáin et al., 2017) were likely a factor here, the responses to this item still offer further evidence of issues with alignment of the mathematical development fostered by the PDMT and the teachers' immediate needs related to teaching the content within their curriculum. Table 9 shows the topic areas within each strand of the curriculum for which more than 25% of the teachers who completed the survey indicated that they felt no more confident in teaching that particular area after completing the PDMT. The percentage of participants indicating that the PDMT pedagogy workshops had either some impact or a significant impact on their teaching was between 76.6% and 79.8% for each of the five workshops. The online survey also considered the extent to which the program had motivated the teachers to incorporate new ideas or strategies into their practice. This aligns with the important link that Bitto and Butler (2010) draw between teachers' self-efficacy and their willingness to consider and incorporate new ideas and practices into their classrooms. Of those who completed the survey, 41.9% reported having already used at least one strategy from the pedagogy workshops in their classroom, and 20.9% reported having already used several. An additional 25.6% indicated that, while they have not yet implemented any strategies from the pedagogy workshops, they plan to do so in the future.
The survey found equally encouraging results regarding the aim of developing teachers' capacity and willingness to conduct action research in their classrooms. Of those who responded, 83.9% said that the PDMT's required action research project had either some impact or significant impact on their practice, and 82.8% indicated that they were open to incorporating research, likely to incorporate research, or had already incorporated research into their classroom as a result of the PDMT.

Discussion
The post-test and final survey results offer evidence of some development in participating out-of-field teachers' mathematical knowledge and self-efficacy over their two years in the PDMT. When combined with the fact that of over 60% of participating teachers reported having already incorporated at least one, if not several strategies learned from the PDMT into their classrooms, the results are encouraging. At the same time, the findings also raise some important concerns about persistent weaknesses in participating teachers' mathematical knowledge related to key areas of the Irish curriculum.
These findings provide both motivation and guidance for reconsidering the PDMT content and highlight the critical need to ensure that professional development designed to support out-of-field teachers is tailored to their specific needs. This is important for both the overall structure and content of a program such as the PDMT and the specific concepts addressed within content modules. For instance, while there were full content modules in both probability and statistics, equating to 20% of the content focus, after completion of the PDMT, items assessing curriculum concepts in probability and statistics were both some of the cohort's best answered items as well as some of the cohort's worst answered items.
Many factors influenced the development of the PDMT content, structure and delivery. Among these was the need for the program to align with the undergraduate mathematics requirements for secondary mathematics teacher licensure set by the Irish Teaching Council. This was to ensure that teachers who successfully complete the program are eligible to extend their existing teaching qualification to include mathematics. The resulting misalignment between the module content and the secondary curriculum is noted several times as a likely factor in why teachers' scores in some areas were still worryingly low after completing the program. This, combined with some participants rating modules as having little to not impact on their confidence in teaching the secondary curriculum continues to fuel the debate over what mathematical content is actually needed to effectively teach mathematics (Ball et al., 2008;Ingersoll, 2001;Shah et al., 2019).
The fact that the program was, in part, designed to match initial certification standards for mathematics teachers raises the question of whether standard undergraduate mathematics courses are really the type of content knowledge that teachers need to be prepared to teach the secondary curriculum. Particularly for the out-of-field teachers in this study, the results suggest that advanced mathematical studies similar to the content of an undergraduate degree did not adequately address the gaps in their knowledge of the mathematics in the curriculum they are not only expected to teach, but are currently teaching.
While the literature presents compelling arguments for the importance of advanced mathematical studies in shaping future teachers' conceptions about the subject (Paolucci, 2015), the pre-program survey showed that this particular population came into the program with high levels of confidence about their mathematical knowledge and ability to teach the subject (Ní Ríordáin et al., 2017). This likely limited their efforts to draw links between the advanced-level content in some modules and the mathematics in the curriculum that they teach. It is also likely why Calculus 3 was seen as the least valuable module in the program, given that it could easily be perceived as the most removed from secondary mathematics.
Similarly, increases in teachers' mathematical knowledge and self-efficacy along with self-reported impact on practice are positive outcomes; however, the notion of false confidence also requires further investigation in this context. This was initially noted in a disparity between high confidence levels reported on the initial online survey, and low achievement on the initial paper-and-pencil test (Ní Ríordáin et al., 2017). One limitation of the final survey is that the researchers cannot say whether some teachers' reports of feeling no more confident teaching some areas of the curriculum after completing the PDMT is attributable to the fact that their confidence levels were already high in the beginning or that maybe these areas of the content were not well-addressed by the program. Some combination of these is also possible. Regardless, even though many teachers reported increased confidence in teaching areas of the curriculum and a willingness to employ innovative approaches to teaching these areas, their ability to do so will remain, to some extent, dependent on their mathematical knowledge (Desimone, 2009;Goldsmith et al., 2014).

Conclusions
Identifying specific content areas for which weaknesses in the teachers' knowledge persisted is important for improvement of this particular professional development program; however, there are two broader, more widelyrelevant content-related conclusions for the design and delivery of professional development for out-of-field mathematics teachers. The first is that considering and incorporating context is particularly critical to the design of effective professional development for out-of-field mathematics teachers. In this case, context includes current attitudes and culture around teaching mathematics, the extent of the alignment or misalignment between teachers' backgrounds and the content of the curriculum they are expected to teach, and the specific needs regarding development of mathematical knowledge that teachers bring with them into the program. While the content, structure, and delivery of professional development should be informed by relevant research and literature, as with Hobbs (2012) these findings offer evidence that creating and implementing a research-informed professional development program without a context-and cohort-specific examination of the teachers' needs will ultimately limit the effectiveness of the program.
The second broad conclusion is that professional development content should be informed by the needs of the specific teacher cohort for which it is designed, and it should be continuously adapted to effectively target and address these needs, both prior to and during implementation. There was some room for adaptation of the PDMT from year to year; however, the relatively fixed model and content meant little scope for adapting it to specifically address some of the issues highlighted in the results of the pre-test which examined the teachers' mathematical knowledge. This is also a limitation of efforts to provide large-scale solutions to addressing the needs of out-of-field mathematics teachers, even though out-of-field teaching is a large-scale issue (Hobbs, 2012).
While Desimone (2009) identifies content focus as an important feature of professional development, consideration of the appropriate type and level of mathematical content becomes even more important when attempting to address the distinct needs of out-of-field teachers. Often, these teachers' needs are shaped by the fact that they end up teaching mathematics because of external factors, not because mathematics is their subject of choice (Ingersoll, 2001;Shah et al., 2019). This study demonstrates the difficulties in implementing deep change in teachers' knowledge through professional development (Carney et al., 2016). It may also further support Goldsmith et al.'s (2014) conclusion that teacher learning occurs incrementally and iteratively. However, the most compelling conclusion seems to be that, whether it's in the form of a domain of practice (Clarke & Hollingsworth, 2002) or a focus on coherence between the content of the school curriculum and mathematical content in the professional development (Desimone, 2009), if professional development for out-of-field teachers is not sufficiently grounded in their immediate needs and context, its impact on teachers' development of the essential mathematical knowledge that they need to teach their curriculum on a day-to-day basis will be limited (Hobbs, 2012).
Despite not being addressed in the initial survey, the teachers' self-identified commitment to incorporating new teaching methods and action research in their classrooms seen in the final survey results is a positive outcome. When combined with the teachers' rating of the effectiveness of the program with regard to improving their mathematical knowledge for teaching, the teachers seem to believe that they are better prepared. This reflects a positive impact on their self-efficacy (Bitto & Butler, 2010). Of course, when combined with the improvements seen in their test scores, we can indeed say that the program was effective in improving both their knowledge and their self-efficacy, with the caveat that for some areas of the curriculum, there is still a long way to go.
When discussing self-efficacy, it is important to remember that this cohort of teachers came into the program feeling confident in their ability to teach most of what is in the curriculum (Ní Ríordáin et al., 2017). This is important for two reasons. The first is that those who already felt confident and may not have believed that there was substantial room for improvement were among those in the group that rated the program as effective or very effective in developing aspects of their mathematical content knowledge, including their understanding of relationships between mathematical concepts, their ability to explain and communicate mathematical concepts, and their awareness and use of appropriate mathematical language. This suggests that, at some point in the program, they recognised some room for growth. The second is that the prevalence of out-of-field teachers' false confidence in their ability to teach mathematics highlighted by Ní Ríordáin et al. (2017) offers evidence of a larger cultural issue regarding dispositions toward the knowledge required to teach mathematics. The persistence of low achievement levels among these teachers when tested on mathematical knowledge critical to effectively teaching the curriculum is evidence of the need to shift this perception about who is "qualified" to teach mathematics and the critical need for professional development for out-of-field mathematics teachers.

Recommendations
Overall, the results highlight the complex challenges of out-of-field teaching and how much support these teachers need (Hobbs & Törner, 2019). Given the importance of teachers' mathematical knowledge for teaching (Ball et al., 2008), they also demonstrate that further consideration must still be given to how out-of-field mathematics teachers are supported in developing their knowledge of mathematical content, particularly with regard to the curriculum that they are required to teach. The results of this study show the potential for a large-scale, blended-learning professional development program to positively impact the knowledge and preparation of out-of-field mathematics teachers. At the same time, they show the limitations of such a program, and have broader implications for contributing to the international conversation on the amount, level and type of mathematical content that should be required for teachers. The findings from this study provide an important confirmation of the need for such programs but that they must also be appropriate and suitable for the professional development needs of out-of-field mathematics teachers.