Data Issues

Waves 1 to 7
Data issues - Waves 1 to 7 – February 2019

30 Academic Rating Scale score in Wave 7

The Longitudinal Study of Australian Children (LSAC) uses the Academic Rating Scale (ARS) as one measure of children's academic development. The ARS is also used in the Early Childhood Longitudinal Study (ECLS-K) in the United States (see National Center for Education Statistics [NCES], 2002, 2004). The ARS in the earlier years of ECLS-K is divided into three domains: Language and Literacy, Mathematical Thinking and General Knowledge.

The original ARS was adapted for use with Australian children for LSAC. Only the Language and Literacy and Mathematical Thinking domains were used with the K cohort in Waves 2 through 6, and with the B cohort in Waves 4 through 6. In Wave 7, only the Language and Literacy domain was used, and only with the B cohort.

This section describes the procedures followed to obtain scores for the Academic Rating Scale in Wave 7.

In LSAC, the ARS is administered as part of the Teacher Questionnaire. The nine Language and Literacy items in the questionnaire ask the study child's English teacher about the child's skills, knowledge and behaviours as evidenced in the child's current achievement and motivation, compared to other children in the same year level.6 There are five levels of rating: Not yet, Beginning, In progress, Intermediate and Proficient. Teachers can also indicate if the skill has not yet been introduced at the year level.

30.1 Method

LSAC ARS scores were calculated in the same manner as the ARS scores in ECLS-K, using the Rasch rating score model. This is the procedure followed in previous waves of LSAC for both the K and B cohorts.

Only children who were rated on more than 60% of items were assigned rating scale scores. In Wave 7, the Language and Literacy domain comprised nine items; ratings were therefore required on six or more items. Children with scores on fewer items were not included in the analyses and were not assigned scores. The numbers of children who were and were not assigned scores in each wave are contained in Table 42.

Table 42: Number of children assigned scores on the Academic Rating Scale, Language and Literacy, Wave 7
  Language and Literacy
ARS score assigned 2,524
Some items rated but no score assigned 29
No score on any item 828
Total 3,381

Scores on each of the nine items were rated from 1 to 5, according to the skill level assigned by the teacher. The initial analyses indicated that there was no overlap of the steps within each item (see Appendix A).

Principal component analysis indicated that a single component could be extracted from the nine items, accounting for 71.5 % of the variance (see Appendix B). Subsequent analysis used the rating scale option of the Rasch model, based on Wright and Masters (1982) and implemented in Quest (Adams & Khoo, 1996). The Rasch analysis showed that the reliability of the estimates of children's ability in Language and Literacy was very high (see Table 43). These estimates have remained above 0.90 for both cohorts, with decreases showing once the children have entered secondary school. The analysis assigned case estimates to each child.

Table 43: Internal consistency statistics for the Academic Rating Scale, Language and Literacy, by cohort and wave
Cohort Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7
K cohort 0.95 0.96 0.95 0.93 0.92 - -
B cohort - - - - 0.96 0.95 0.94 0.92

Perfect scores were estimated by adding 1.1 logits to the Rasch estimate of the second highest score. 'Zero' scores were estimated by subtracting 1.1 logits from the Rasch estimate of the second lowest score. The Rasch model does not calculate estimates for perfect or zero scores - 'extreme scores' - so some estimation is required. Wright (1998) has suggested that these extreme scores should be at least 1.0 logit and no more than 1.2 logits away from the next scores, unless some justification can be made for using a greater distance. Examination of the distances between the near-perfect and near-zero scores showed that the addition/subtraction of between 1.0 and 1.2 logits for an extreme score was appropriate, and that 1.1 logits would provide a reasonable result.

Once case estimates were obtained for the pattern of ratings on the ARS items, the estimates were transformed to ARS scores that reflect the range of scores available to the children's teachers; that is, the lowest possible score on the ARS scale is 1 and the highest is 5. Rasch case estimates were then transformed to ARS scores using a linear transformation. Again, this is consistent with the procedures used in ECLS-K. The equation used to convert the Rasch estimates to ARS scores is:

ARS = 2.9513 + (0.2784 x estimate)

Table 44 presents the conversion data for the ARS in each domain for children who obtained scores on all items in the scale. The table shows the raw score, the ARS score and the standard error associated with each score. As noted above, ARS scores were assigned to children with ratings on at least 60% of the items in a scale.

Table 44: Raw score to Academic Rating Scale score conversion tables, Language and Literacy, B cohort, Wave 7
Language and Literacy
Raw score ARSLIT score Standard error (s.e.)
9E 1.00 --
10 1.31 0.30
11 1.54 0.23
12 1.69 0.20
13 1.81 0.18
14 1.91 0.17
15 2.00 0.17
16 2.09 0.16
17 2.17 0.16
18 2.24 0.16
19 2.32 0.15
20 2.39 0.15
21 2.47 0.15
22 2.54 0.15
23 2.61 0.16
24 2.69 0.16
25 2.76 0.16
26 2.83 0.16
27 2.91 0.16
28 2.98 0.16
29 3.06 0.16
30 3.14 0.16
31 3.22 0.16
32 3.30 0.17
33 3.39 0.17
34 3.47 0.17
35 3.56 0.17
36 3.65 0.18
37 3.75 0.18
38 3.84 0.18
39 3.95 0.18
40 4.05 0.18
41 4.16 0.19
42 4.30 0.21
43 4.46 0.24
44 4.69 0.31
45E 5.00 --

Note: Extreme scores are indicated with E; standard errors not available for extreme scores.

6 It is assumed that all children in the cohort have commenced secondary education, with different teachers for each learning area.