Referencia:

Reference:

Cadiz, Janet (2006).  Reanalyzing fourth grade math student achievement  in Chile: applying Hierarchical Linear Models (HLMs) Revista ELectrónica de Investigación y EValuación Educativa, v. 12,  n. 1. http://www.uv.es/RELIEVE/v12n1/RELIEVEv12n1_2.htm. Consultado en (poner fecha).

 

 

 Hits:  Hit counter - Contador de visitas Visitas 

REANALYZING FOURTH GRADE MATH STUDENT ACHIEVEMENT IN CHILE: Applying HIERARCHICAL LINEAR MODELS (HLMs)

[Reanalizando el rendimiento de los alumnos de cuarto grado en matemáticas en Chile: una aplicación de modelos jerárquicos lineales (HLM)]

In pdf format

 

por

 

 En formatopdf

Article record

About the authors

Print this article

Janet Cádiz (jcadiz@puc.cl)

Ficha del artículo

 Sobre los autores

Imprimir el artículo

 

 

Abstract

 The purpose of this article is to present the application of Hierarchical Linear Models (HLMs) in reanalyzing fourth grade math student achievement by using 1996 SIMCE data. This article is part of a study that represents a first attempt to explore more sophisticated statistical techniques— other than those techniques commonly applied thus far— in order to obtain a better understanding of student achievement and the effects of schools in Chile. To achieve this goal, two types of school administration are analyzed: municipal and private subsidized schools, respectively, utilizing the One-Way ANOVA Model and the Random-Intercept Model as the primary HLMs. Results indicate that there are significant differences not only within, but also between municipal and private subsidized schools in math achievement.  However, the significant variation among students and schools remains to be explained.

Resumen

  El objetivo de este artículo es presentar el uso de Modelos Jerárquicos Lineales (HLM) para reanalizar el rendimiento en matemáticas de estudiantes de cuarto grado, usando datos del SIMCE (Sistema de Medición de la Calidad de la Educación de Chile) de 1996. Este artículo es parte de un estudio que representa una primera tentativa de explorar técnicas estadísticas más sofisticadas – diferentes de las técnicas comúnmente aplicadas hasta ahora - para obtener una mejor comprensión del logro de estudiante y del efecto escuela en Chile. Para alcanzar este objetivo, se analizan dos tipos de administración escolar : municipal y privada subvencionada, respectivamente, utilizando el Modelo de ANOVA de una vía y el Modelo Intercepto-Aleatorio como principales análisis de tipo jerárquico. Los resultados indican que hay diferencias significativas no sólo dentro de, sino también entre escuelas municipales y privadas subvencionadas en el rendimiento en matemáticas. Sin embargo, una variación significativa entre estudiantes y escuelas queda aún por ser explicada.

Keywords

The System of Assessing the Quality of Education in Chile (SIMCE), Hierarchical Linear Models (HLMs), student math achievement, municipal schools and private subsidized schools

Descriptores

Sistema de Medición de la Calidad de la Educación, Modelos Jerárquicos Lineales, rendimiento, matemáticas, escuelas municipales, escuelas privadas subvencionadas.

 


Introduction


Comentar este artículo


Artículos relacionados:

Cervini (2002). Desigualdades Socioculturales en el Aprendizaje de Matemática y Lengua de la Educación Secundaria en Argentina: Un modelo de tres niveles


Volumen 12, n. 1


During the early 1980s, the Chilean government developed and implemented an educational reform based on decentralization and privatization policies.  The decentralization of the central government and the privatization of public goods and services were important strategies for improving the quality and efficiency of public institutions[i].  In education, these political strategies allowed the government to increase the amount of decision-making done by local governments and to foster the participation of the private sector in the educational system.

   As a result of these political changes, a new Chilean educational reform was implemented to increase the quality of education and to achieve efficiency in the use of educational resources. To help achieve the goals of this educational reform, new components were introduced into the educational system.  For example, the component of “devolution” was established to diminish bureaucracy and promote decision-making to better represent local characteristics.  That is, devolution strove to give the responsibility of decision-making back to the community.  This devolution process was called municipalization and led to the creation of the Municipal Common Fund (FCM), which concentrated on transferring public schools into municipalities, giving these municipalities greater professional and technical support, and diminishing the inequality of resources between municipalities.

Since the execution of this measure, municipalities have had control of educational expenditures.  However, the central government has continued allocating financial and technical support.  In this case, the most important hypothesis was that the quality of educational services might be improved by increasing the overall level of expenditures and by appropriately matching expenditures with local requirements (Hevia, 1982; Jimenez, 1984; Latorre & Nunez, 1987; and Rounds Parry, 1997).

In relation to the “privatization policies”, the Chilean Ministry of Education gave subventions to public and private schools, which provided a per-student payment or voucher.  In this case, the principal purpose was to encourage the private sector to create “tuition-free” private schools and to make them more competitive with public schools.  In 1987, the element of subvention was created by the Ministry of Education to ensure the equal distribution of vouchers to both free private schools and public schools.  According to Barr (1993):

 “Subsidies to education may be justifiable for both efficiency and equity reasons. In a libertarian world, individuals pay for the private benefits they receive from education, but are subsidized to the extent that external benefits are thereby conferred upon others” (p. 346). 

As noted earlier, several measures were introduced to implement decentralization and privatization policies.  These measures are summarized as follows: (1) public schools were transferred to the authority of municipalities (2) the Ministry of Education encouraged the creation of private subsidized schools, as well as provided subsidies based on average monthly student attendance for municipal and private subsidized schools; and (3) teachers, who for many years worked in the public sector, became private sector employees.  Furthermore, a national system of school supervision was developed in order to give schools technical and pedagogical support, and the curricula were reformulated, aimed at being more flexible and better attune to local conditions.

In order to obtain valid information to describe the quality of Chilean education and explain issues related to decentralization and privatization policies, a national assessment system was implemented. This assessment system was named the System of Assessing the Quality of Education in Chile (SIMCE).  From the time when SIMCE was created, different educational reports have revealed that students who attended municipal schools tended to have lower academic achievement than those who attended private subsidized schools at the mid-low and low socio-economic status (SES) student levels, and those who attended non-subsidized private schools continued to have the highest SIMCE scores at the high and mid-high SES student levels.  In part, these results run counter to assumptions about the effectiveness of decentralization and privatization policies in education.

In addition to the aforementioned issue, several problems have been found to stem from the analysis and report of SIMCE information.  This is because the ways in which SIMCE results are analyzed and reported tend to be very broad in nature.  For instance, to obtain summarized and illustrative school information at the local and the national level, SIMCE regroups some school variables (e.g. type of city and school accessibility are clustered in the geographic indicator).  By using this clustered information, SIMCE builds eighteen theoretical structures in order to group Chilean elementary schools, which have similar characteristics (The Theoretical Structures of School Groups, SIMCE 1996).  Moreover, SIMCE usually summarizes student test outcomes by using three types of average percentages.  These are:  (1) the average percentage of correct answers per student group, (2) the average percentage of correct answers in comparison to other schools which have similar characteristics, and (3) the average percentage at the national level (the SIMCE Report Form, 1996). 

These two ways in which SIMCE staff uses SIMCE information reveals important issues.  In particular, the manner in which SIMCE results are analyzed and reported is extremely broad-based for adequately explaining the nuances and variability of student test scores and understanding the complexity of Chilean education at the local context level (Gomez and Edwards, 1995).  Correspondingly, when we use clustered variables, the meaning of these clustered variables may change in comparison to their single meaning (Murchan & Sloane, 1994).   Indeed, by using excessive clustered information, we may have limited opportunity for developing in-depth explanations in relation to particular variables that affect student achievement.

Another crucial issue in SIMCE assessment is related to the analysis of the SIMCE Math Test.  Through the use of this math test, Chilean educators can obtain a general profile about student math achievement in relation to the average percentage at the school level, at the national level, and at other similar schools.  However, there are not enough statistical analyses to allow us to explain differences in math scores at the student and school levels.  For these reason, we can see that it is necessary to develop further statistical analyses in order to explain the relationship between student and school variables, and their effect on math achievement.

More recent SIMCE information reveals that although the 1996 SIMCE results show an increase in math scores, differences persisted with regard to the type of school administration.  However, there is not enough evidence to account for the scope of differences in math achievement between municipal and private subsidized schools.  Further limiting our understanding of such differences is the fact that other SIMCE variables, such as the geographic accessibility to a school, school size, school population area, and Chilean geographic region, are analyzed in an isolated way. 

As you may have noted in the previous explanations offered here, most of the SIMCE analyses are essentially descriptive.  For more sophisticated statistical analyses, some Chilean researchers use Multiple Regression and Analysis of Variance (ANOVA).  Traditionally, linear models are based on the assumption that subjects respond independently to educational programs.  Nevertheless, this assumption of independence is regularly violated because an educational system is an organizational structure and a multilevel enterprise.  In other words, we can see that students are nested within teachers and classes, teachers and classes are nested within types of schools, and schools are nested within a particular geographic location (Burstein, 1980; Bryk & Raudenbush, 1992; Murchan & Sloane, 1994). 

In sum, all of these traditional statistical analyses do not provide a full explanation of the quality of Chilean education.  In fact, when we examine the SIMCE reports, we can find some conflicting conclusions.  Bryk & Raudenbush (1989) state that: “When [school effects and instruction] vary among individuals and the contexts in which they are educated, traditional data analysis approaches can be very misleading” (p.163).  The issues presented here reveal the need for evaluating the quality of the past SIMCE statistical analyses.  Actually, contemporary educational studies are recommending the application of multilevel analyses for obtaining more accurate standard errors, and examining how/whether the relationship of interest (e.g. the relationship between SES and student achievement) varies across contexts.  Moreover, multilevel techniques allow us to simultaneously analyze the effect of school or classroom variables at different levels upon student outcomes.  Further, researchers are putting together data sets, which will enable them to reanalyze or replicate prior analyses, and to obtain more information. 

Based on previous explanations, four questions led to the development of this article aiming at examination of the application of HLMs for reanalyzing 4th grade students’ math achievement.  These questions are:

·       With respect to 4th grade students, how much do Chilean elementary schools vary in their mean math achievement?

·       What is the extent of difference in math achievement between schools?

·       What is the extent of difference in math achievement between schools after controlling for the students’ SES at the school level?

·       With respect to geographic accessibility to a school, school population areas, and school size, what is the extent of difference in math achievement between schools?

 

Method

Data Source and Participants

Two sets of math data were used to run descriptive and inferential analyses, which come from the System of Assessing the Quality of Education in Chile (SIMCE).  These are:  SIMCE Math Test for fourth grade and the SIMCE School Questionnaire

Given constraints of time and space for this article, I chose to concentrate on the fourth grade student population, with a focus on the 1996 math test for the purpose of this study.  This is because 1996 is one of the years in which SIMCE took into consideration certain kinds of information at the student level.  Taking into consideration municipal and private subsidized schools, there are 227,283 students, which were nested within 5,065 Chilean elementary schools.

In relation to the 1996 SIMCE data, it is important to explain that this data is limited in order to develop HLM analysis.  One of the most important limitations is at the student level (Level-1 model), in which the student achievement is the primary variable.  In other words, the majority of SIMCE variables are at the school level.  For instance, SES is a variable usually related with student `s characteristics, however, the SIMCE data takes this variable as a school variable because it computes the SES average per school. With on these limitations, the SIMCE data allows us to examine school effect on student math achievement in which only the intercept parameter in the Level-1 model is varying at the Level-2 model.

 

Variables

-          The 4th grade students’ math achievement:

Math achievement of 4th grade student is assessed through the application of the SIMCE math test.  The SIMCE math test has 45 items where the first 12 items are related to basic math skills, and the next 33 items are related to specific math curriculum for fourth grade.  The SIMCE math test scored on a scale of 0 to 100. 

-           The type of school administration[ii]:

The type of school administration is classified into two categories:

            Municipal schools: These schools are managed by municipalities and obtain subvention from the Chilean government.

            Private subsidized schools: These schools are managed by particular enterprises and obtain subvention from the Chilean government.

-           The students’ SES at the school level:

To determine the socio-economic status of the student population at the school level, the Ministry of Education and the SIMCE program request information regarding the average of educational expenditures per family and the average level of parents’ education for each school.  The categories are: Low-SES, Mid-low SES, Mid-high SES, and High-SES.

-          School size:

This variable represents the total student enrollment at a school.  It is divided into four categories: Small-size (less than 100 students); Medium-small size (between 101 and 500 students); Medium-large size (between 501 and 1000 students); and Large-size (more than 1000 students).

-           School population areas:

This is a dummy variable related to population areas in which the school is located.  Its categories are: (0) Rural areas: These are schools located in countryside areas, and (1) Urban areas: These are schools located in cities or towns areas.

-           Geographic accessibility to a school:

This variable is related to the difficulty or facility of entering the geographic area in which the school is located.  This indicator is divided into four categories:  Limited accessibility, Poor accessibility, Moderate accessibility, and Good accessibility.

 

Data analysis

The analysis of school effects on math achievement for fourth grade is divided into two types of statistical analyses, descriptive and inferential analyses[iii].  The descriptive analysis allows us to develop an exploratory examination of the SIMCE variables.  The inferential analysis allows us to examine the HLMs developed.

For the inferential analysis, I developed two types of Hierarchical Linear Models, the One-Way ANOVA Model and the Random-Intercept Model.  The first Hierarchical Linear Model allows us to obtain information about how much variation in the math outcome lies within and between Chilean elementary schools, particularly in the 4th grade.  The second Hierarchical Linear Model allows us to examine school effect on student math achievement in which only the intercept parameter in the Level-1 model is varying at the Level-2 model. 

It is important to explain my decision to use the Random-Intercept Model because several crucial predictors in the SIMCE assessment are measured at the school level rather than at the student level.  This type of Hierarchical Linear Model allows us to avoid the following methodological dilemmas: (a) To examine the data only at the student level and to ignore the fact that students are nested in school.  As a result of this inappropriate methodological approach we could have problems with the estimated standard error because it could potentially be too small. (b) To examine the data at the school level and to use the means of student responses as the outcome.  As a result of this approach, we could have problems with the estimation of school effect.  Also, we may have problems including other level-1 predictors into the analysis. 

 

Results

Descriptive analyses

In 1996, 5,592 Chilean elementary schools participated in the SIMCE assessment and 248,364 students took the SIMCE math test.  The general descriptive analysis reveals that the mean math score of Chilean schools is 68.4 and the standard deviation is 11.6.  The most frequently occurring math score value, mode, is 60, and the minimum and maximum student math scores are 25 and 99.  Finally, the middle score, median, is 68.9. 

a.         Municipal schools

In municipal schools, the mean math score of 4th grade students is 65.5, and the standard deviation is 10.28.  The middle of the math score distribution, median, is 66.2.  Moreover, the minimum math score is 25 and their maximum is 96.  As valuable information, it is necessary to explain that the mean math score of municipal schools is lower than the national average (65.5at the municipal level < 68.4at the national level). 

In table 1, we can observe that municipal schools located in urban areas have a higher mean math score (66.4) than those schools located in rural areas (62).  In comparison to the national average for rural schools, municipal schools tend to have a slightly higher mean math score (62at the municipal level > 60.9at the national level).  On the contrary, municipal schools tend to have a lower math math score in comparison to the national average for urban schools (66.4at the municipal level < 69.7at the national level).  In relation to the standard deviation, municipal schools in rural areas have a greater standard deviation than schools in urban areas. 

 

TABLE 1 - The 1996 SIMCE math test by Municipal Schools

Variables

Categories

Mean

Std. Dev.

Min - Max

Range

n

    School population areas

   Rural areas

62

13

30 - 96

66

690

   Urban areas

66.4

9.2

25 - 95

70

2746

    SES

   Low-SES

61.7

12.6

25 - 96

71

956

 

   Mid-low-SES

66.3

8.8

34 - 92

58

2162

 

   Mid-high-SES

71.8

7.2

49 - 89

40

318

 

   High-SES *

 

 

 

 

 

    Geographic accessibility to a school

   Limited-accessibility

59.4

14.5

31 - 93

62

201

   Poor-accessibility

63.1

12.1

32 - 96

64

498

   Moderate-accessibility

64.6

11.1

25 - 95

70

723

 

   Good-accessibility

67

8.4

36 - 94

58

2014

    School size

   Small-size

62

12.8

25 - 96

71

935

 

   Medium-small size

65.6

9.6

34 - 91

57

1400

 

   Medium-large size

67.4

7.2

47 - 88

41

747

 

   Large-size

70.3

6.8

46 - 85

39

354

 

 

Total of Municipal Schools

3436

*    There are no municipal schools whose students have high-SES.

In the table above, we can also see that municipal schools in which students have a mid- high SES obtain the highest mean in math and the smallest standard deviation.  Additionally, it is important to note that 4th grade students with mid-high-SES at the national level have better math achievement than students with the same SES at the municipal level (75.5mean score at the national level > 71.8mean score at the municipal level).  Similar to the results obtained at the national level, municipal schools in which students have low-SES continue to obtain the lowest mean math score (60.7 mean score at the national level < 61.7 mean score at the municipal level) and the greatest standard deviation. 

In relation to the geographic accessibility to a school, results reveal that students who attend municipal schools with limited geographic accessibility tend to have the lowest mean scores in the SIMCE math test (59.4), while students who attend municipal schools with good geographic accessibility tend to have the highest mean (67) and the lowest standard deviation (8.4) in the math test.

The final SIMCE variable in this part of the descriptive analysis is the school size.  We can observe that municipal schools that are medium-large-or large in size tend to have higher mean math scores (67.4medium-large-size and 70.3large-size) than other schools of smaller size (62small-size and 65.6medium-small-size).  They also have smaller standard deviations than those smaller schools.

b.         Private Subsidized Schools

In private subsidized schools, the mean math score of 4th grade students is 69.4; the standard deviation is 11.59; the median is 71.18; the minimum math score is 26 and the maximum is 99.  In relation to the school population areas, we can see that private subsidized schools located in urban areas have a higher math mean score than those schools located in rural areas (table 2).  Furthermore, private subsidized schools tend to have lower mean math scores (55.8) in comparison to the national average of Chilean schools located in the same rural areas (60.9).  In contrast, private subsidized schools tend to have higher math mean score (70.7) in comparison to the national average of Chilean schools located in the same urban areas (69.7).  For private subsidized schools, the standard deviation tends to be greater in rural areas than in urban areas.  It is also significant to see that private subsidized schools located in rural and urban areas tend to have similar standard deviations (13.6rural and 10.4urban) compared to the entire population of Chilean schools located in rural and urban areas, respectively (13.3rural and 10.8urban). 

In table 2, we can observe that private subsidized schools in which students have a high-SES obtain the highest mean in math and the smallest standard deviation (80.6 math mean score and 7.1 standard deviation), while those schools in which students have a low-SES obtain the lowest mean math score and the greatest standard deviation (55.8 math mean score and 14.1 standard deviation).  As additional information, it is interesting to observe that 4th grade students with low-SES at the national level tend to have better math achievement than students with the same SES at the private subsidized level (60.7mean score at the national level > 55.8mean score at the private subsidized level). 

 

TABLE 2 - The 1996 SIMCE math test by Private Subsidized Schools

Variables

Categories

Mean

Std. Dev.

Min - Max

Range

n

    School population areas

   Rural areas

55.8

13.6

26 - 92

66

146

   Urban areas

70.7

10.4

33 - 99

66

1483

    SES

   Low-SES

56.3

14.1

26 - 99

73

214

 

   Mid-low-SES

67.2

9.6

29 - 93

64

743

 

   Mid-high-SES

75.2

7.2

50 - 91

41

574

 

   High-SES

80.6

7.1

47 - 91

44

98

    Geographic accessibility to a school

   Limited-accessibility

55

16

30 - 92

62

55

   Poor-accessibility

56.5

12

26 - 84

58

93

   Moderate-accessibility

61.4

14

33 - 99

65

105

 

   Good-accessibility

71.4

9

36 - 95

59

1376

    School size

   Small-size

56.3

14.1

26 - 99

73

194

 

   Medium-small size

68.5

10.5

29 - 93

64

783

 

   Medium-large size

73.4

8.4

44 - 91

47

431

 

   Large-size

76.2

7.1

54 - 91

37

221

 

Total of Private Subsidized Schools

1629

In relation to the geographic accessibility to a school, it is important to explain that the distribution of math scores in private subsidized schools tends to follow the same national trend in which an appropriate geographic accessibility allows 4th grade students to have better math achievement.  The results show that students who attend private subsidized schools with limited geographic accessibility tend to have the lowest mean in SIMCE math test (55), while students who attend those schools with good geographic accessibility tend to have the highest math mean score (71.4) and the lowest standard deviation (9).

The fourth SIMCE variable is school size.  The descriptive analyses reveal that private subsidized schools of medium-large and large-size tend to have higher mean math scores (73.4medium-large-size and 76.2large-size) than the other schools of smaller sizes (56.3small-size and 68.5medium-small-size).  Also, those schools with medium-large and large-size tend to have smaller standard deviations than those schools of smaller sizes.

In sum, we observe that 4th grade students vary in their math achievement among Chilean schools.  The 1996 SIMCE math test reveals that the SIMCE variables analyzed can help contribute to explaining an important portion of the variation of math scores. 

Inferential Analyses: Hierarchical Linear Models

Based on the descriptive analyses of the SIMCE variables alone, however, we cannot say that these differences in math achievement are significant.  The following inferential analyses will allow us to determine whether or not these differences in math achievement are statistically significant, particularly in relation to municipal and private subsidized schools.  To achieve this purpose, two types of Hierarchical Linear Models are developed in this section.

a.         The One-Way ANOVA Model

The One-Way ANOVA Model allows us to obtain information about the grand mean, and the partitioning of the total variation in math achievement into variation between and within schools.  Furthermore, this model allows us to examine the hypothesis that all Chilean schools have the same mean math achievement.  In this analysis, the key question is: With respect to 4th grade students, how much do Chilean elementary schools vary in their mean math achievement?  In this case, the Hierarchical Linear Model (HLM) is:

Level-1 Model (Student Level): 

Yij =    b0j + rij

Yij =    the 4th grade math outcome score for student (i) in school (j)

b0j =    the intercept for each school’s mean math achievement.

rij ~ N(0, s2 )  in which s2 is the student-level variance

  

Level-2 Model (School Level): 

b0j =    g00 + u0j

b0j =    each school’s mean math achievement

g00 =    this is the grand mean

u0j ~ N(0, t00 )  = the random error in which t00 is the school-level variance

As noted in the previous explanation, this is an unconditional model in which there are no predictors at either Level 1 or 2.  In table 3, we can see the primary results from One-Way ANOVA Model. 

TABLE 3 - Results from One-Way ANOVA Model

   Fixed effect

Coefficient

se

 

 

   Average school mean, g00

67.26

0.14

 

 

   Random Effect

Variance Component

df

X2

p value

   School mean, u0j

() 91.23

5064

66335.02

0.000

   Level-1 effect, rij

(2) 293.46

 

 

 

 

In the table above, we can observe that the estimate for the grand-mean math achievement is 67.26 with a standard error of 0.14.  This estimate is statistically significant at the 0.001 level (0.000<0.001).  In relation to the estimates of the variance components, we can see that the estimate for the within-school or Level-1 variance (2) is 293.46, while the overall variability among the true school means on math achievement () is 91.23.  Based on these estimates, we can indicate that most of the variation in the outcome is at the student level, although a significant proportion is between Chilean schools. 

 In order to determine the proportion of variance in Y between Chilean elementary schools, which is the intra-class correlation, the estimated variance components are substituted for their parameters.  In this case, the intra-class correlation is equal to 91.23/(91.23+293.46)= 0.24.  This result reveals that about 24% of the variance in math achievement is between schools.

For this analysis, the crucial hypothesis is H0: t00 = 0.  That is, we assume that all Chilean Elementary schools have the same mean math achievement.  In the table above, we can observe that the Chi-Square statistic (X2) takes on a value of 66335.02 with 5064 degrees of freedom (J= 5065), and its p-value is 0.000.  This means that it is highly implausible to have the same mean math achievement among schools.  Therefore, the evidence indicates that significant variation among schools in their math achievement remains to be explained.

b.         The Random-Intercept Model

The Random-Intercept Model allows us to examine school effect on student math achievement in which only the intercept parameter in the Level-1 model varies in the Level-2 model.  In this analysis, the Level-1 model is the same as in the One-Way ANOVA Model.  That is, student math achievement scores will be modeled as varying around their school means.  In relation to the Level-2 model, each school’s mean will be predicted by the socio-economic status of the student population at the school level (MEANSES), the type of school administration (ADM), school size (SCHSIZE), geographic accessibility to a school (ACCESS), and school population areas (URB_RUR).

In order to better explain the effects of the crucial SIMCE variables on math achievement, three primary Random-Intercept Models are presented.  The first model takes into consideration only the type of school administration (ADM) in which municipal and private subsidized schools are the primary categories.  For this analysis, the Hierarchical Linear Model is:

 

Level-1 Model (Student Level): 

Yij =    b0j + rij

Level-2 Model (School Level): 

b0j =    g00 +g01(ADM) j+ u 0j

b0j =    each school’s mean math achievement

g00 =    the intercept

g01 =    this is the effect of ADM on b 0j

u0j ~ N(0, t00 )  = Residual. That is, b0j - g00 - g01(ADM)j

t00 =    the residual variability at the school level (conditional model)

In the table below, the results from the Random-Intercept Model are presented.  The results reveal that there is a highly significant association between the type of school administration (ADM) and mean math achievement.  For example, we can see that private subsidized schools have significantly higher mean achievement than municipal schools (g01 = 4.22, t= 14.06).

TABLE 4 - Results from the Random-Intercept Model (I)

   Fixed effect

Coefficient

se

t-ratio

p-value

   Model for school means

 

 

 

 

   INTERCEPT, g00

65.87

0.17

 

 

   ADM, g01

4.22

0.30

14.06

0.000

   Random Effect

Variance Component

df

X2

p value

   School mean, u0j

() 86.43

5063

60719.15

0.000

   Level-1 effect, rij

(2) 293.52

 

 

 

 

In relation to the residual variance between schools, this Random-Intercept Model displays a smaller variance than the One-Way Anova Model (t00 (Random-Intercept Model) = 86.43 < t00 (One-Way Anova Model) = 91.23).  In this part of analysis, it is important to remember that t00 (One-Way Anova Model= unconditional model) represents the total parameter variance in the school means that is potentially explained by alternative level-2 models for B0j.   

By computing the proper equation[iv], we observe that the estimated proportion of variance between schools explained by the model with type of school administration (ADM) is 0.05.  That is, 5% of the true between-school variance in math achievement is accounted for by the type of school administration.

Another important analysis is related to the conditional intra-class correlation.  As noted in the One-Way Anova Model, the intra-class correlation was 0.24.  In the Random-Intercept Model, after removing the effect of the type of school administration, the correlation between pairs of math scores in the same school is now reduced to 0.22.

The final analysis refers to the homogeneity of residual school means.  For this type of analysis, the null hypothesis is t00 = 0, where t00 is now the residual variance.  In this case, the crucial question is: do school achievement means vary significantly once the type of school administration is controlled?  In the table above, we can see that the value of the Chi-Square (X2) is 60719.15 with 5063 degrees of freedom (J= 5065), and its p-value is 0.000.  This result indicates that the null hypothesis is easily rejected.  Therefore, after controlling for the type of school administration, significant variation among school mean math achievement remains to be explained.

The second Random-Intercept Model takes into consideration the type of school administration (ADM) and the MEAN SES of school.  For this analysis, Level-1 and Level-2 are structurally similar to the previous model.  The only difference is that this model includes one more predictor, which is MEAN SES.

 

Level-1 Model (Student Level): 

Yij =    b0j + rij

 

Level-2 Model (School Level): 

b0j =    g00 +g01(ADM) j+ g02(MEANSES) j u 0j

Table 5 presents the primary results from the second Random-Intercept Model.  We can observe that the type of school administration (ADM) and MEAN SES are positively related to students’ math achievement.  That is, there is a highly significant association between the type of school administration and mean math achievement (g01 = 0.91, t= 3.12); and between the school MEAN SES and mean math achievement (g02 = 6.54, t= 31.90).  It is also important to emphasize that private subsidized schools continue to have higher mean achievement than municipal schools after controlling for the effect of MEAN SES However, the differences between these types of schools in math achievement tend to decrease when MEAN SES is incorporated into the analysis.

TABLE 5 - Results from the Random-Intercept Model (II)

   Fixed effect

Coefficient

se

t-ratio

p-value

   Model for school means

 

 

 

 

   INTERCEPT, g00

66.52

0.15

 

 

   ADM, g01

0.91

0.29

3.12

0.002

   MEANSES, g02

6.54

0.20

31.90

0.000

   Random Effect

Variance Component

df

X2

p value

   School mean, u0j

() 69.10

5062

47908.67

0.000

   Level-1 effect, rij

(2) 293.55

 

 

 

 

With respect to the residual variance between schools, this second Random-Intercept Model shows a smaller variance than the One-Way Anova Model (t00 (Random-Intercept Model) = 69.10 < t00 (One-Way Anova Model) = 91.23).  The estimated proportion of variance between schools explained by the model based on the type of school administration (ADM) and MEAN SES is 0.24.  That is, 24% of the true between-school variance in math achievement is accounted for in this model.

As mentioned in the previous analysis, the intra-class correlation for the One-Way Anova Model was 0.24.  In this Random-Intercept Model, after removing the effects of the type of school administration and MEAN SES, the conditional intra-class correlation is 0.19.  In other words, the results reveal that the correlation between pairs of math scores in the same school is now reduced from 0.24 to 0.19.

Here, it is also important to determine the homogeneity of residual school means.  In other words, it is essential to determine how much school achievement means vary when the type of school administration and MEAN SES are controlled.  Here, the null hypothesis is t00 = 0, where t00 is now the residual variance.  In table No 4, we can observe that the value of the Chi-Square (X2) is 47908.67 with 5062 degrees of freedom (J= 5065), and its p-value is 0.000.  By using these results, the null hypothesis is once again easily rejected.  Therefore, we can say that significant variation among school mean math achievement remains to be explained, after controlling for the type of school administration and MEAN SES.

The third Random-Intercept Model takes into consideration the following SIMCE variables: the effect of the accessibility to a school (ACCESS), the MEAN SES of school (MEAN SES), school population area (URB_RUR), school size (SCHSIZE), and the type of school administration (ADM).  For this analysis, the Level-1 and Level-2 are:

Level-1 Model (Student Level): 

Yij =    b0j + rij

 

Level-2 Model (School Level): 

b0j =    g00 +g01(ACCESS)j + g02(MEAN SES)j + g03(URB_RUR)j + g04(SCHSIZE)j + g05(ADM) j+ u 0j

b0j =    each school’s mean math achievement

g00 =    the intercept

g01 =    this is the effect of ACCESS on b 0j

g02 =    this is the effect of MEANSES on b 0j

g03 =    this is the effect of URB_RUR on b 0j

g04 =    this is the effect of SCHSIZE on b 0j

g05 =    this is the effect of ADM on b 0j

u0j ~ N(0, t00 )  = Residual. That is, b0j - g00 - g01(ACCESS)j- g02(MEAN SES)j - g03(URB_RUR)j- g04(SCHSIZE)j- g05(ADM)j

t00 =    the residual variability at the school level (conditional model)

In Table 6, the results from the third Random-Intercept Model are presented.  We can observe that the schools’ MEAN SES, school size, and the type of school administration are positively related to students’ math achievement.  In other words, there is a highly significant association between the schools’ MEAN SES and mean math achievement (g02 = 5.33, t= 20.91); between school size and mean math achievement (g04 = 1.23, t= 6.77); and between the type of school administration and mean math achievement (g05 = 1.16, t= 3.984).  In relation to the type of school administration, private subsidized schools still have significantly higher mean achievement than municipal schools.  The effect of the accessibility to a school (ACCESS) and the school population area (URB_RUR) are not statistically significant. 

 TABLE 6 - Results from the Random-Intercept Model (III)

   Fixed effect

Coefficient

se

t-ratio

p-value

   Model for school means

 

 

 

 

   INTERCEPT, g00

66.71

0.12

 

 

   ACCESS, g01

0.15

0.36

0.43

0.666

   MEANSES, g02

5.33

0.25

20.91

0.000

   URB_RUR, g03

0.56

0.79

0.71

0.475

   SCHSIZE, g04

1.24

0.18

6.77

0.000

   ADM, g05

1.16

0.29

3.98

0.000

   Random Effect

Variance Component

df

X2

p value

   School mean, u0j

() 67.81

5059

46141.38

0.000

   Level-1 effect, rij

(2) 293.57

 

 

 

 

In this Random-Intercept Model, we can see that the residual variance between schools is smaller than the One-Way Anova Model (t00 (Random-Intercept Model) = 67.81 < t00 (One-Way Anova Model) = 91.23).  Furthermore, by using the variances obtained in the One-Way ANOVA model (the unconditional model) and the Random-Intercept Model (the conditional model), we can look at the estimated proportion of variance between schools, which is explained by the predictors at level-2.  In this case, the estimated proportion of variance between schools explained by the type of school administration, school size and schools’ MEAN SES is 0.26.  That is, 26% of the true between-school variance in math achievement is accounted for by these SIMCE variables.

In relation to the conditional intra-class correlation, after removing the effect of the type of school administration, school size and schools’ MEAN SES, the correlation between pairs of scores in the same school, which had been 0.24 for the One-Way Anova Model, is now reduced to 0.18

As a final point, it is important to determine how much school achievement means vary when the type of school administration, school size and schools’ MEAN SES are controlled.  In table No 4, the value of the Chi-Square (X2) is 46141.38 with 5059 degrees of freedom (J= 5065), and its p-value is 0.000.  By using these results, the null hypothesis (t00 = 0) is, as we have seen with the two other models presented here, easily rejected.  Therefore, we can say that significant variation among school mean math achievement remains to be explained, after controlling for the type of school administration, school size and schools’ MEAN SES.

 

Discussion

Based on the previous analyses, we observe that most of the variation in the math outcome is at the student level.  However, a significant portion of the variation in the math outcome is at the school level, particularly in relation to municipal and private subsidized schools.  The results reveal that 4th grade students who attend private subsidized schools have higher math achievement scores than those who attend municipal schools.  In fact, private subsidized schools continue to show higher mean achievement than municipal schools after controlling for the effect of MEAN SES

In relation to the other SIMCE variables, we observe that school size is positively related to students’ math achievement.  Although the effect of the accessibility to a school (ACCESS) and the school population area (URB_RUR) are not statistically significant, these are crucial SIMCE variables.  In this case, it is important to continue developing alternative analyses to determine the real contribution of these variables.

In sum, the type of school administration, school size and schools’ MEAN SES explain an important proportion of variance between schools.  Moreover, after removing the effect of these SIMCE variables, the correlation between pairs of scores in the same school is substantively reduced.  However, , these exploratory analyses tell us that significant variation among students and schools remains to be explained. 

In future studies, it will be necessary to identify other variables that affect math achievement at the individual and organizational levels.  Of course, it is important to develop other instruments that will allow us to acquire more information at the student and school levels.  In other words, the SIMCE assessment needs to include other sources of information.  This will lead us to develop better Hierarchical Linear Models to more fully understand and explain the variability in math achievement in Chilean schools.

 

References

Barr, N. (1993).  The economic of the welfare state.  Stanford: Stanford University Press.

Bryk, A. & Raudenbush, S. (1989).  Toward a more appropriate conceptualization of research on school effects: A three level hierarchical linear model.  In R. D. Bock (Ed.), Multilevel analysis of educational data (pp. 159-204).  San Diego: Academic Press.

Bryk, A. & Raudenbush, S. (1992).  Hierarchical Linear Models.  Newbury Park, Sage.

Burstein, L. (1980).  The analysis of multilevel data in educational research and evaluation.  Review of Research in Education, 8, 158-233.

Gomez, M & Edwards, V.  (1995).  Equidad y educación básica en Chile: Análisis comparativo de la equidad de la educación por regiones y comunas según los datos del SIMCE de 1992Santiago: UNICEF and PIIE.

Hevia, R. (1982). El mecanismo de la municipalización educacional en Chile.  In E. Schiefelbein and V. Apablaza (Eds), La regionalización educacional en Chile: municipalización o alcalización?  Santiago: Corporación de Promoción Universitaria.

Jimenez, J. (1984).  Un enfoque universitario del proceso de municipalización educacional Chileno: responsabilidad frente a la formación de administradores educacionales.  In E. Schiefelbein and V. Apablaza (Eds), La regionalización educacional en Chile: municipalización o alcalización?  Santiago: Corporación de Promoción Universitaria.

Latorre, L. & Nunez, I. (1987).  El financiamiento de la educación en Chile: evolución histórica y alternativas futuras.  Santiago: Academia de Humanismo Cristiano.

Murchan, D. & Sloane, F. (1994).  Conceptual and statistical problems in study of school and classroom effects: An introduction to multiple modeling techniques.  In I. Westbury et al. (Eds.), In search of more effective mathematics education (pp. 247-272).  New Jersey: Ablex Publishing Corporation Norwood.

Raudenbush, S. & Willms, J. (1991).  Schools, classrooms, and pupils: International studies of schooling from a multilevel perspective.  Academic Press.

Rondinelli, D. (1989).  Decentralizing urban development programs: concepts and guideline for analysis.  Washington, D.C.: United States Agency for International Development.

Rounds Parry, T (1997).  Decentralization and privatization: Education policy in Chile.  Journal Public Policy, 17, 1, 107 - 133.

SIMCE (1996).  The SIMCE Guidebook; the SIMCE School Questionnaire; the Theoretical Structures of School Groups; and the SIMCE report form. Documentos SIMCE. Santiago-Chile.

 

NOTES


[i] Decentralization: three components help us to define decentralization.  The first component of decentralization is deconcentration.  The goal is to shift work from central government officials in the capital city to those working in regions, provinces or districts.  The second component is delegation. The goal is to foster the delegation of decision-making from the central government to the local government.  As a result of this delegation, the central government might decrease their bureaucratic involvement in local level administrations.  The third component is devolution.  The goal is to give total authority and autonomy to particular public services, which may make local decisions without the prior permission of the central government.  Privatization: usually, decentralization is related to the privatization process.  Governments allow private enterprises to provide goods and services previously provided only by the central government.  That is, privatization refers to privatize organizations, enterprises, associations, professional groups etc., which were originally administered through the public sector (Rondinelli, 1989)

[ii] For the purpose of this study, this variable was transformed into a dummy variable. The dummy variable is categorized as follows: Municipal Schools is coded (0) and Private Subsidized Schools is coded (1)

[iii] In the dissertation for the degree Doctor of Philosophy in Education (Cadiz, 2001), several inferential analyses where developed by using Hierarchical Linear Models (HLMs) in order to examine the extent of differences in 4th grade Mathematics achievement among Chilean elementary schools

[iv] The following equation allows us to determine the proportion of variance explained        in b 0j 

Proportion of Variance =

Explained in b 0j

 


ABOUT THE AUTHORS SOBRE LOS AUTORES

Janet Cádiz Henríquez (jcadiz@puc.cl). Ph. D in Education – Research methodology, evaluation and measurement - UCLA, USA.  Es especialista en metodología de la investigación, evaluación y medición.  Actualmente realiza docencia y asesoramiento de tesis en los programas de pregrado y postgrado de la Facultad de Educación, Pontificia Universidad Católica de Chile. Su dirección postal es: Facultad de Educación,  UC. Av. Vicuña Mackenna 4860, Macul. Casilla 306.  Correo 22  -  Santiago de Chile. Buscar otros artículos de esta autora en Scholar Google

 

 


ARTICLE RECORD / FICHA DEL ARTÍCULO

Reference /

Referencia

Cadiz, Janet (2006).  Reanalyzing fourth grade math student achievement in Chile: applying Hierarchical Linear Models (HLMs).

Title / Título

Reanalyzing fourth grade math student achievement in Chile: applying Hierarchical Linear Models (HLMs). [Reanalizando el rendimiento de los alumnos de cuarto grado en matemáticas en Chile: una aplicación de modelos jerárquicos lineales (HLM)]

Authors / Autores

Janet Cádiz

Review

Revista

Revista ELectrónica de Investigación y EValuación Educativa (RELIEVE), v. 12, n. 1 

ISSN

1134-4032

Publication date /

Fecha de publicación

 2006  (Reception Date:  2005 July 07; Approval Date: 2006 February 26; Publication Date: 2006 February 28)

Abstract /

Resumen

  The purpose of this article is to present the application of Hierarchical Linear Models (HLMs) in reanalyzing fourth grade math student achievement by using 1996 SIMCE (System of Assessing the Quality of Education in Chile) data. This article is part of a study1 that represents a first attempt to explore more sophisticated statistical techniques— other than those techniques commonly applied thus far— in order to obtain a better understanding of student achievement and the effects of schools in Chile. To achieve this goal, two types of school administration are analyzed: municipal and private subsidized schools, respectively, utilizing the One-Way ANOVA Model and the Random-Intercept Model as the primary HLMs. Results indicate that there are significant differences not only within, but also between municipal and private subsidized schools in math achievement.  However, the significant variation among students and schools remains to be explained.

   El objetivo de este artículo es presentar el uso de Modelos Jerárquicos Lineales (HLM) para reanalizar el rendimiento en matemáticas de estudiantes de cuarto grado, usando datos del SIMCE (Sistema de Medición de la Calidad de la Educación de Chile) de 1996. Este artículo es parte de un estudio que representa una primera tentativa de explorar técnicas estadísticas más sofisticadas – diferentes de las técnicas comúnmente aplicadas hasta ahora - para obtener una mejor comprensión del logro de estudiante y del efecto escuela en Chile. Para alcanzar este objetivo, se analizan dos tipos de administración escolar : municipal y privada subvencionada, respectivamente, utilizando el Modelo de ANOVA de una vía y el Modelo Intercepto-Aleatorio como principales análisis de tipo jerárquico. Los resultados indican que hay diferencias significativas no sólo dentro de, sino también entre escuelas municipales y privadas subvencionadas en el rendimiento en matemáticas. Sin embargo, una variación significativa entre estudiantes y escuelas queda aún por ser explicada.

Keywords

Descriptores

The System of Assessing the Quality of Education in Chile (SIMCE), Hierarchical Linear Models (HLMs), student math achievement, municipal schools and private subsidized schools.

Sistema de Medición de la Calidad de la Educación, Modelos Jerárquicos Lineales, rendimiento, matemáticas, escuelas municipales, escuelas privadas subvencionadas.

Institution /

Institución

Pontificia Universidad Católica de Chile

Publication site /

Dirección

http://www.uv.es/RELIEVE 

Language / Idioma

 English (Title, abstract and keywords in spanish)

 

Volumen 12, n. 1

 

© Copyright, RELIEVE.  Reproduction and distribution of this article  is authorized if the content is no modified and its origin is indicated (RELIEVE Journal, volume, number and electronic address of the document).

© Copyright, RELIEVE.  Se autoriza la reproducción y distribución de este artículo siempre que no se modifique el contenido y se indique su origen (RELIEVE, volumen, número y dirección electrónica del documento).

   

  [Principal]  [La Revista] [Artículos publicados]  [Publicar en RELIEVE]

 [Home Page]  [ About us]  [ Published Articles]  [To publish in RELIEVE]

Revista ELectrónica de Investigación y EValuación Educativa

  http://www.uv.es/RELIEVE

email: relieve@uv.es

Statistics  Estadísticas