3.2 Sampling procedures for the selection of subjects

3.2.1 The two main groups of the sample: First Language (L1) and second Language (L2) groups

3.3 Structure and administration of the English proficiency tests

3.3.1 The cloze tests

3.3.1.1 Theoretical overview

3.3.1.2 The cloze tests used in the study

3.3.2 The essay tests

3.3.2.1 Theoretical overview

3.3.2.2 The essay tests used in the study

3.3.3  Error recognition and mixed grammar tests

3.3.3.1 Theoretical overview

3.3.3.2 Error recognition and mixed grammar tests used in the study

3.3.4  The Dictation test

3.3.4.1 Introduction

3.3.4.2 Theoretical overview

3.3.4.3 The dictation tests used in the study

3.3.4.4 Presentation of the dictation tests

3.3.4.5 Method of scoring of the dictation tests

3.4 Summary of Chapter 3

3.1 Introduction

This chapter describes the sample of subjects and sampling procedures (of subjects and tests) and provides a detailed theoretical overview and description of the battery of English proficiency tests.

The sample of subjects consists of two main groups: First Language (L1) and Second Language (L2). A major issue in this study is between L1 and L2 levels of language proficiency. This issue has become a controversial distinction in South Africa where more and more applied linguists and educationists argue that the L1-L2 distinction should be jettisoned. (I discuss specific authors on this issue in Chapter 6). I use the labels L1 and L2 slightly differently from what is normally meant by these labels. The way these labels are used are explained shortly.

A literature review is provided for each of the test methods used in which an overview of the relevant theoretical issues is provided. After each literature review follows a detailed description of the structure and administration of the specific tests used.

3.2 Sampling procedures for the selection of subjects

The sampling procedures form a crucial part of the method rationale and are described in detail. The crucial issue is how to classify the subjects into different levels of proficiency.

There were 90 entrants to Grade 7 in January 1987 who also sat the Grade 7 end-of-year examinations. Owing to the fact that the battery of English proficiency tests was administered during the first week of the school year, there was some absenteeism during the three-day testing period. Thus, not all of the learners did all of the tests, and four learners did not do any of the tests. These four learners were not included in the sample. The other 86 learners (44 boys, 42 girls) comprise the sample of subjects.

3.2.1 The two main groups of the sample: First language (L1) and second language (L2) groups

Figures 3.1 and 3.2 provide a clear picture of the details of the L1 and L2 groups. The reader may want to consult these figures in conjuction with the verbal descriptions of the sample below.

At the school there were mother-tongue speakers from diverse linguistic backgrounds, e.g. Tswana, Sotho, English, Afrikaans and some expatriates, e.g. Greek and Filipino. (The exact numbers are provided in Table 3.1, which we shall come to later on). About two thirds were Tswana mother-tongue speakers. All learners had to take English as the medium of instruction at the School.

The Tswana speakers could choose from the following language subject combinations:

- Tswana as a First Language and English as a Second Language. (After 1987 Afrikaans was also offered as a first language).

- Tswana as a First Language and English as a First Language.

- English as a First Language and Afrikaans as a Second Language. Tswana speakers never took this option.

The English and Afrikaans speakers and speakers of other languages (expatriates and those using English as a “replacement” language) could choose from the following language subject combination:

- English as a First Language and Afrikaansas a Second Language.

- English as a First Language and French as a Second Language. This combination was taken by the expatriates, because they had not studied Afrikaans in primary school as South Africans had done. The “replacement” learners took Afrikaans as a second language.

All the L2 learners were Bantu mother-tongue speakers, most of whom were Tswana speakers. The L1 learners were a mixture of English mother- tongue speakers, Tswana mother-tongue speakers, mother-tongue speakers of other languages. The latter consisted of (i) expatriates from other countries and (ii) South Africans who speak other South African non-Bantu languages such as Afrikaans and Gujarati. It was not always certain who among the L1 group (i.e. those who took English as First language at the School) were mother-tongue speakers because some of them identified themselves as mother-tongue speakers of English and/or another language, e.g. Afrikaans, Sotho. However, there was little doubt that many in (ii) were using English as a “replacement” language. (I show why this is so at the end of section 4.8). It is, of course, possible to have more than one “mother” (or “father”, or “native”) tongue.

I say something briefly about the notion of mother tongue and relate it to the notions of native language and replacement language. (These notions are discussed in more detail in section 6.2). The notion of native-speaker is not a simple one. According to Paikeday, the “the native speaker is dead!”. Indeed, it is difficult to identify a native speaker, who is someone who apparently should have a more thorough knowledge than the nonnative speaker. When one adds the notion of mother-tongue speaker, first-language speaker and second/additional-language speaker to the pot, it becomes difficult to see the wood for the trees. I discuss this issue in greater depth in section 6.2.

The fact that the Tswana learners had a choice between taking English as First Language or as a Second Language and that the “replacement” language learners had to take English First language becomes important in decisions of how to categorise the different levels of proficiency. I explain the problem later on. (A detailed analysis of the sample follows shortly).

All language subjects were taught in separate classes. With regard to theother subjects, L1 and L2 learners were taught through the medium of English in homogeneous groups in different classrooms, where each class contained about an equal proportion of L1 and L2 learners. For example, Grade 7 was divided into four classes. Each class bore the initial of the surname of the relevant class teacher. This four-class arrangement was maintained for the administration of the tests.

It was general practice at MHS that on entrance to the School, learners decided themselves whether they preferred to take English as a First Language subject or as a Second Language subject. As far as the battery of English proficiency tests was concerned, the subjects in the sample were requested to indicate on their protocols whether they had decided to take English First Language as a subject or English Second Language as a subject.

The results of the tests were not subsequently used by the School administration to make any changes to the choices the entrants had made regarding the English group (L1 or L2) they wanted to belong to. There were several possible reasons for this:

- The School did not wish to force the labels of “English second language” (L2) onto learners.

- Limited English proficiency learners might benefit in a class of high English proficiency learners, because the former might benefit from listening to a higher standard of English than their own.

- The School might not have been sure of the actual level of English proficiency of each individual entrant, in spite of the fact that it was aware that the level of English proficiency of disadvantaged entrants was generally low, but once these entrants had become part of the School, it would have been possible to make more accurate judgements of their English proficiency.

- Finally, the School might have been reluctant to use the results until I had produced solid evidence that these tests were valid predictors of academic achievement.

It was not a simple matter to decide how to classify the subjects. The following variables had to be taken into account (the descriptions are specific to the sample):

(1) Some were or said they were English mother-tongue speakers, while others were mother-tongue speakers of Tswana and other languages.

(2) Some had English as the medium of instruction from Grade 1 (Bantu speakers and non-Bantu speakers), while some had English as the medium of instruction from Grade 5 (only Bantu speakers).

(3) All had the freedom to choose at the beginning of Grade 7 whether they wanted to take English First Language or English Second Language.

The problem is whether one can make a clear separation between these subjects that would indicate a difference in levels of English proficiency. An obvious division in theory is mother-tongue speaker/non-mother-tongue speaker, where mother-tongue English proficiency was regarded as the level of English to aspire to. Thus, when the essays for this study were marked they were judged in terms of mother-tongue proficiency, and so non-mother–tongue English speakers’ essays were not marked more leniently than those of mother-tongue English speakers. There were difficulties in deciding on the norms for the other tests, which were all previously standardised published tests, because it is only after the test has been performed on the test-bench that it is possible to decide whether the test is too easy or too difficult. If there are mother-tongue speakers and non-mother-tongue speakers in the same sample, as in this study, one needs to consider whether the norms of the two kinds of speakers should be separated or interlinked. One can only do this if subjects have been precisely classified into mother-tongue/non-mother-tongue groups. This was not a simple matter in the sample for the following reasons:

In the truly multicultural setting of MHS a composite of the following cultural-ethnic groups said that they were English mother-tongue speakers: Ghanaian, Sri Lankan, Indian (South African and expatriate) and Coloured. There was also a Greek, a South Sotho, and a Fillipino who said that they had two mother tongues, one of them being English.Although all the above (N=18) obtained an English proficiency test score (in this case a composite of the cloze, dictation and essay test) of 60% and over, there were also quite a number of Bantu mother-tongue speakers (N=10, mostly Tswana) who also obtained a score of 60% and over. Further, there were five subjects who said that they were mother-tongue speakers but obtained scores between 50% and 55%. As I shall show later such a score is not a good score for somebody claiming to be an English mother-tongue speaker. In the light of this evidence, it was difficult to tell from the results of the proficiency tests who were mother-tongue or native-speakers of English. Although it was difficult in the sample to pinpoint mother-tongue English speakers this does not mean that the notion of “native-speaker” is a figment. (More about this in section 6.2). All I could specify about the sample in this regard was that it consisted of a wide range of English proficiency . (Recall that the labels L1 and L2 in the sample simply refer to those taking English First or Second Language as a subject, respectively).

When the English proficiency test scores were examined without resorting to summary statistics but merely sorted in ascending order there was a clear distinction between the subjects who had chosen to do English as First Language and English as a Second Language, respectively. These are called the L1 and L2 groups, respectively. In terms of the test results this meant that the L1 group on average were substantially better than the L2 group. (Six of the subjects who had decided to change form L1 to L2 in Grade 8 had obtained over 60% on the English proficiency test).

Classifications of human groups, especially in the human sciences, are not expected to be objectively “precise”; what matters is that the parameters of the classifications be delineated and used consistently. In this study I use the labels “L1″ and “L2″ to refer to “high proficiency” and “low proficiency” indicated by English First Language and English Second Language, respectively.

Also important is English as a medium of teaching and learning. The tests in this study aimed “to select lexical and structural items relevant to the demands of the appropriate syllabuses”, i.e. relevant to English as the medium of instruction. What these tests aim at testing is directly related to the task-demands of academic study.

At MHS L1 learners and L2 learners are not treated as two separate groups, because both groups are taught in the same classroom – where English is used as the medium of instruction – and write the same examinations except for the language subjects, i.e. English, Tswana, Afrikaans and French. This has important implications for the interpretation of the results, where it might be argued that the L1 and L2 groups belong to separate populations and therefore cannot be grouped together in a correlational analysis. (See section 4.6 for further discussion of this issue). Tables 3.1 and 3.2 provide a detailed analysis of the sample.

The subjects originated from 36 different schools. The L1 subjects (N=49)

originated from (i) CM Primary School (N=37), (ii) a “white” school (N=1), (iii) a “coloured” school (N=4), and (vi) several DET schools (N=7). (One Sri Lankan came from a DET school where his mother was a teacher).

The 37 L2 subjects originated from 28 DET schools, three church schools, a Coloured school and an Indian school.

Of the total sample of 86 subjects, there were 60 South African blacks, of which 52 were Tswanas and eight were non-Tswanas. These eight non-Tswana South African blacks, like all the Tswana subjects, had to take Tswana as a first language.

TABLE 3.1 Detailed Analysis of the L1 Subjects

.

There were 10 TL1 subjects who changed from L1 English as a subject to L2 English as a subject in Grade 8 (January 1988). I did not have any information on why these changes were made. One plausible reason for this change could have been that the School recommended to the learners concerned that it was in their best interest to change, because the change to L2 English as a subject at a later stage might have given them a better chance of passing English. Another plausible reason is that learners decided themselves to make the change, because it wasn’t necessary to take English First Language, because they already had Tswana as a First Language. Whatever the reason for the change to L2, the fact is that other members of the TL1 group who had obtained Grade 7 English achievement scores in the same range as those who changed to English Second Language did not change from English First Language to English Second Language. For example, the Grade 7 English scores of the 10 L1 subjects who changed to L2 in Grade 8 were (in ascending order in percentages) 50, 51, 53, 53, 55, 55, 58, 58, 61 and 63. (Most of these also had English proficiency scores in the 55% to 70% range). The Grade 7 English scores of five L1 subjects who did not change to L2 in Grade 8 were 45, 52, 53, 59, 62. As a matter of interest, in the first set, eight out 10 obtained a matriculation exemption, while in the second set, two left the school after passing a grade, two obtained a matriculation exemption, and one failed before reaching Grade 12 and left the school.

The “replacement language” learners were required to take English First language because they had no other First Language, while the Tswanas in TL1 group could take Tswana as a First Language. The initial choice of language group (L1 or L2) at the beginning of Grade 7, as pointed out earlier, was voluntary.

Most of the L2 subjects were mother-tongue speakers of Tswana who came from rural or peri-urban schools where English was used on a limited scale in the classroom, hardly at all in the playground, and not at all at home, which was probably the reason for the low level of English proficiency of most of them. A few L2 subjects, however, did have a high level of English proficiency. There were no L2 subjects from CM Primary School, because all learners at this school had English as the medium of instruction from Grade 1. L2 subjects even if they did very well in English Second Language did not change to English First Language. This was probably because there was no need to complicate their lives unnecessarily. Five L2s obtained over 70% in Grade 7 English achievement. Four of these five passed Grade 12 and three of them obtained a matriculation exemption. A detailed analysis of the L2 group follows.

TABLE 3.2. Detailed Analysis of the L2 Subjects

As shown in Table 3.2, four of the black L2 subjects were non-Tswanas. These took Tswana as a first language at the School.

In sum, there are two groups in the sample: the L1 group (a composite of the Tswana L1 and the Non-Tswana L1 groups) and the L2 group. The L2 group is also referred to as the TL2 (Tswana L2) group, because 33 of the 37 L2 subjects are mother-tongue speakers of Tswana.

3.3 Structure and administration of the English proficiency tests

Subjects were divided into four groups and the tests were administered in four classrooms by four Grade 7 teachers, where each classroom contained a combination of L1 and L2 subjects. The time allocated for each test will be indicated in the description of the administration of the individual tests.

The possibility exists that fatigue resulting from a three-day test period may have affected the results of all the tests, but this seems unlikely because subjects were released from all lessons and from all school activities during this three-day period. Also, the test sessions were interspersed with ample rest periods. The structure and administration of the English proficiency tests follows. A theoretical review precedes the description of each of the tests. Before I describe the tests I would like to point out that no sample of tests can adequately represent the vast variability of language, nor does it have to “because of the generative nature of language which acts as its own creative source”. The controvery, as far as general language proficiency is concerned, is which sample of tests to use: the “reductionist” kind of tests used in this study or “holistic” (formal and informal) outside-of-the-classroom “cocktail party”, “tea-party, “cooking club” type tests. In the academic context, what is important is the relationship between general, or overall, proficiency, communicative competence and academic achievement.3.3.1 The cloze test 3.3.1.1 Theoretical overviewCloze tests are deceptively simple devices that have been constructed in so many ways for so many purposes that an overview of the entire scope of the literature on the subject is challenging to the imagination not to mention the memory.

Cloze tests are deceptively simple devices that have been constructed in so many ways for so many purposes that an overview of the entire scope of the literature on the subject is challenging to the imagination not to mention the memory.

(Oller, 1973:106)

Since 1973 the literature on cloze has more than doubled, adding even more challenges to the imagination if not – thanks to the printed word – to the memory.

The aim of a cloze test is to evaluate (1) readability and (2) reading comprehension. The origin of the cloze procedure is attributed to Taylor (1953), who used it as a tool for testing readability. Of all the formulas of readability that have been devised, cloze tests have been shown, according to Geyer (1968), Weintraub (1968) and Oller (1973:106), to be the best indicators of readability. It is also regarded as a valid test of reading comprehension. Oller (1973:106) cites Bormuth (1969:265) who found a multiple correlation coefficient of .93 between cloze tests and other linguistic variables that Bormuth used to assess the difficulty of several prose passages. Bormuth (1969:265) maintains that cloze tests “measure skills closely related or identical to those measured by conventional multiple choice reading comprehension tests.”

Many standardised reading tests use cloze tests, e.g. the Stanford Proficiency Reading Test. Johnson and Kin-Lin (1981:282) believe that cloze is more efficient and reliable than reading comprehension, because it is easier to evaluate and does not, as in many reading comprehension tests, depend on long written answers for evaluation. (But it is also possible to use multiple choice reading tests; see Bormuth [1969] in the previous paragraph). Johnson and Kin-Lin’s implication is that although cloze and reading comprehension are different methods of testing, they both tap reading processes. Anderson (1976:1), however, maintains that as there is no consensus on what reading tests actually measure, all that can be said about a reading test is that it measures reading ability. On the contrary, far more can be said about reading: notions associated with reading are “redundancy utilization” (Weaver & Kingston, 1963), “expectancies about syntax and semantics” (Goodman, 1969:82) and “grammar of expectancy” (Oller, 1973:113). All these terms connote a similar process. This process involves the “pragmatic mapping” of linguistic structures into extralinguistic context (Oller, 1979:61). This mapping ability subsumes global comprehension of a passage, inferential ability, perception of causal relationships and deducing meaning of words from contexts (Schank, 1982:61). According to Bachman (1982:61),

[t]here is now a considerable body of research providing sound evidence for the predictive validity of cloze test scores. Cloze tests have been found to be highly correlated with virtually every other type of language test, and with tests of nearly every language skill or component.

Clarke (1983), in support of Bachman, is cautiously optimistic that the cloze procedure has a good future in reading research. Alderson (1979:225) who is less optimistic, maintains that

individual cloze tests vary greatly as measures of EFL proficiency. Insofar as it is possible to generalise, however, the results show that cloze in general relates more to tests of grammar and vocabulary than to tests of reading comprehension.

Hughes (1981), Porter (1978) and Alderson (1979) found that individual cloze tests produce different results. Johnson and Kin-Lin (1981) and Oller (1979), contrary to Alderson (1979), found that a great variety of cloze tests correlates highly with tests such as dictation tests, essay tests and reading tests as well as with “low order” grammar tests.The concept of closure is important in cloze theory. Alderson (1979:225) asks

whether the cloze is capable of measuring higher-order skills. The finding in Alderson (1978) that cloze seems to be based on a small amount of context, on average, suggests that the cloze is sentence – or indeed clause – bound, in which case one would expect a cloze test to be capable, of measuring, not higher-order skills, but rather much low-order skills…as a test, the cloze is largely confined to the immediate environment of a blank.

This means that there is no evidence that increases in context make it easier to complete items successfully. Oller (1976:354)  maintains, contrary to Alderson, that subjects “scored higher on cloze items embedded in longer contexts than on the same items embedded in shorter segments of prose”. Oller used five different cloze passages and obtained similar results on all of them.

Closure does not merely mean filling in items in a cloze , but filling them in a way that reveals the sensitivity to intersentential context, which measures “higher-order skills” (Alderson above). A cloze test that lacks sufficient closure would not be regarded as a good cloze test.

There are two basic methods of deletion: fixed deletion and rational deletion, or “selective” deletion). In the former, every nth word is deleted; which may range between every fifth word – which is considered to be the smallest gap permissible without making the recognition of context too difficult – and every ninth word. Pienaar’s (1984) tests, which are used in this study, are based on a rational deletion procedure.

Alderson (1980:59-60) proposes that the rational deletion procedure should not be referred to as a “cloze” but as a “gap-filling” procedure. Such a proposal has been accepted by some researchers, e.g. Weir (1993:81), but not accepted by others, e.g. Bachman’s (1985) “Performance on cloze tests with fixed-ratio and rational deletions”, Maclean’s (1984) “Using rational cloze for diagnostic testing in L1 and L2 reading” and Markham’s (1985) “The rational deletion cloze and global comprehension in German.” There is nothing wrong with the proposal that the rational-deletion procedure be called a gap-filling test, if it remains nothing more than that – a proposal.

Alderson (1979:226) suggests that what he calls cloze tests (namely, every nth word deletion) should be abandoned in favour of “the rational selection of deletions, based upon a theory of the nature of language and language processing”. Thus, although Alderson proposes that the rational selection of items, his “gap-filling” should not be called a cloze procedure, he still favours “gap-filling” tests over “cloze” tests. This superiority of the rational deletion method is supported by a substantial body of research, e.g. Bachman (1982) and Clarke (1979). However, it should be kept in mind that language hangs together, and thus the every-nth-word cloze test is also, in my view, a good test of global proficiency. In other words, whether one uses “fixed” deletions or “rational” deletions, both methods test global proficiency.

Having considered the arguments for the validity of the cloze test as a test of reading, it seems that cloze tests are valid tests of reading strategies, i.e. they can test long-range contextual constraints. One must keep in mind, however, that deletion rates, ways of scoring, e.g. acceptable words or exact words, and types of passages chosen in terms of background knowledge and of discourse devices, may influence the way reading strategies are manifested. But it is debatable whether one should make to much of these differences.

3.3.1.2 The cloze tests used in the study

In a review of Pienaar’s pilot survey called “Reading for meaning”, Johanson refers to the “shocking” low reading levels in many schools in the North-West Province revealed by Pienaar’s survey.

Pienaar tested a variety of learners from different schools: learners whose (1) first language (i.e. mother tongue or a language the learner knows best) was English, (2) replacement language was English, (3) first language was Afrikaans, (4) first language was a Bantu language, mostly Tswana speakers. Categories (1) and (2) and (3) came from upper middle class families, while category (4) was split into two sub-categories: (4a) Bantu speakers who generally came from working class families and (4b) Bantu speakers who lived in sub-economic settlements in the environs of Mmabatho. Many of the parents of (4b) were illiterate or semi-literate and are either unemployed or semi-employed. (The category labels used by Pienaar are different to mine: I have changed them for clarity sake). Pienaar’s major finding was that 95% of learners in the North West Province (Grade 3 to Grade 12), most of whom belonged to category 4b, were “at risk”, i.e. they couldn’t cope with the academic reading demands made on them.

There are four reasons why Pienaar’s cloze tests are used in this study:

(1) they have already been used in many schools in the North West Province and have produced a solid body of results, (2) their purpose is “to select lexical and structural items relevant to the demands of the appropriate syllabuses”, (3) Pienaar’s (1984) cloze tests are based on a rational deletion method, where it is possible to select gaps in such a way that closure , i.e. long-range constraints, is ensured, and (4) Pienaar’s data will be compared with the data in this study.

Pienaar’s (1984) tests comprise five graded levels – “Steps” 1 to 5, where each level consists of four short cloze passages (Form A to Form D) with 10 blanks in each passage (Pienaar 1984:41):

Step 1 corresponds to Grades 3 and 4 for English first language and to Grades 5 to 7 for English second language.

Step 2 corresponds to Grades 5 and 6 for first language and Grades 7 to 9 for second language.

Step 3 corresponds to Grades 7 and 8 for first language and to Grades 9 to 11 for second language.

Step 4 corresponds to Grades 9 and 10 for first language and to Grades 11 and 12 for second language.

Step 5 corresponds to Grades 11 and 12 for first language and to Grades 12 + for second language.

If one Step proves too easy or too difficult for a specific pupil, a higher or a lower Step could be administered. For example, if Step 2 is too difficult, the pupil can be tested on Step 1. In this way it is possible to establish the level of English proficiency for each individual pupil.

Owing to the fact that (1) many of the L1 group were not mother-tongue speakers of English and (2) I had to give the same test to both the L1 and L2 groups in order to make reliable comparisons, I used Step 2 for the L1 group and the L2 group.

I did not use the other Steps, because it was irrelevant to the purpose of study, which was not to place learners in the level they belong to, i.e. for teaching purposes (which was the purpose of Pienaar’s tests), but to test for proficiency at the Grade 7 level, and in the process to test the tests themselves, i.e. examine whether the passages chosen for the Step 2 level were valid for that level. Although annual predictions between English proficiency and academic achievement might yield higher correlations than long-term predictions, the aim is to investigate what chance Grade 7 learners who entered the School in 1987 would have of passing Grade 12.

According to Pienaar, a perfect score on a cloze test indicates that the pupil has fully mastered that particular level. A score below 50% would indicate that the learner is at risk.

Pienaar maintains that English second language learners are generally two to three years behind English first language learners in the acquisition of English proficiency, and there is often also a greater age range in the English second language classes, especially in the rural areas. Pienaar’s implication is that to be behind in English language proficiency is also to be behind in academic performance.

Pienaar standardised his tests in 1982 on 1068 final year JSTC (Junior Secondary Teacher’s Certificate) and PTC (Primary Teacher’s Certificate) students from nine colleges affiliated to the University of the Transkei. These standardised results became the table of norms for Pienaar’s tests (Pienaar 1984:9). Below are the weighted mean scores achieved by the students of the nine colleges (Pienaar 1984:10):

Step 1 Step 2   Step 3    Step 4   Step 5

Weighted means:    67%    53%       37%       31%      24%

Most of the colleges performed similarly on all five Steps. These results confirmed the gradient of the difficulty of the various steps.

During 1983 Pienaar administered a major part of the test instrument to a smaller group of college students selected from the original large group. No significant difference between the scores of the two administrations was found, which confirmed the test-retest reliability of the instrument..

The tests underwent ongoing item analysis and refinement. By the time the final version was submitted to school learners in the Mmabatho/Mafikeng area in 1984, 30% of the items had been revised. As a result of continuous item analysis, a further 18% of the items were revised.

An important point is that these results claim to represent the reading ability of college students, who are supposed to be more proficient in English than school learners. However, final year student teachers only obtained a score of between 40% and 60% on Step 2 – see Pienaar’s mean scores above. (Step 2 has been used in this study for Grade 7 learners). These low scores indicate that the reading level of the student teachers, who were to start teaching the following year, was probably no higher than the level of many of the learners they would eventually teach. Thisalarmingstate of affairs would probably have had a detrimental effect on the academic performance of these learners.

Pienaar’s original tests had four passages for each level and that he tried to establish the equivalence in difficulty between the passages for each level. In this study I used two cloze passages for Step 2 instead of four. This was done for two reasons: (1) to see whether only two passages were sufficient, and (2) the other two passages, in their “unmutilated” form, were used for the dictation tests, because they belonged to the same level, namely, Step 2. The question is whether two passages – in the cloze and dictation tests – were enough to ensure reliability and validity. The results of the tests (Chapter 4) deal with this question.

In the test battery I used Pienaar’s Form B and Form D passages of Step 2, as shown below:

Pienaar’s Practice exercise

(Pienaar does not provide the answers for this practice exercise. Possible answers are provided in brackets).

The 1 (rain) started falling from the sagging black 2 (clouds) towards evening. Soon it was falling in torrents. People driving home from work had to switch their 3 (headlights) on. Even then the 4 (cars, traffic) had to crawl through the lashing rain, while the lightning flashed and the 5 (thunder) roared.

Cloze passage 1: Form B Step 2 (Pienaar 1984:59):

A cat called Tabitha

Tabitha was a well-bred Siamese lady who lived with a good family in a shiny white house on a hill overlooking the rest of the town. There were three children in the family, and they all loved Tabitha as much 1 she loved them. Each night she curled up contentedly on the eldest girl’s eiderdown, where she stayed until morning. She had the best food a cat could possibly have: fish, raw red mince, and steak. Then, when she was thirsty, and because she was a proper Siamese and did 2 like milk, she lapped water from a blue china saucer.

Sometimes her mistress put her on a Cat show, and there she would sit in her cage on 3 black padded paws like a queen, her face and tail neat and smooth, her black ears pointed forward and her blue 4 aglow.

It was on one of these cat shows that she showed her mettle. The Judge had taken her 5 of her cage to judge her when a large black puppy ran into the hall. All the cats were furious and snarled 6 spat from their cages. But Tabitha leapt out of the judge’s arms and, with arched 7 and fur erect, ran towards the enemy.

The puppy 8 his tail and prepared to play. Tabitha growled, then, with blue eyes flashing, she sprang onto the puppy’s nose. Her 9 were razor-sharp, and the puppy yelped, shook her off, and dashed for the door. Tabitha then stalked back down the row of cages to where she had 10 the judge. She sat down in front of him and started to preen her whiskers as if to say, “Wait a minute while I fix myself up again before you judge me.” She was quite a cat, was Tabitha!

Answers. (The words in round brackets are Pienaar’s suggested alternative answers. The words in square brackets are suggested alternative answers):

1. as; 2. not; 3. her [four, soft]; 4. eyes (eye); 5. out; 6. and; 7. back (body); 8. wagged, twitched (waved, lifted); 9. claws (nails); 10. left (seen, met).

Item 9 could also be “teeth”. (There are cats who jump on to faces and bite, rather than scratch). Even in easy cloze passages, “acceptable” answers can be a problem.

Cloze passage 2: Form D Step 2 (Pienaar 1984:61):

A dog of my own

When I was ten all 1 wanted was a dog of my own. I yearned for a fluffy, fat, brown and white collie puppy. We already had two old dogs, but my best friend’s pet collie had 2 had seven fluffy, fat, brown and white puppies, and I longed for one with all my heart. However, my mother said no, so the seven puppies were all sold. I had horses, mice, chickens and guinea-pigs, and as my 3 said, I loved them all, but I wasn’t so keen on finding them food. Since she had five children to look after, it made here angry to 4 hungry animals calling, so she said crossly, “No more dogs.”

This didn’t stop me wanting one though, and I drew pictures of collie dogs, giving 5 all names, and left them lying around where she would find them. As it was 6 Christmas, I was sure that she would relent and give me a puppy for Christmas.

On Christmas morning I woke up very excited, 7 the soft little sleepy bundle that I wanted at the bottom of the bed wasn’t there. My mother had given me a book instead. I was so disappointed that I cried to myself, yet I tried not to 8 her how sad I was. But of course she noticed.

Soon after that my father went off to visit his brother and when he came back he brought me a puppy. Although it 9 a collie it was podgy and fluffy, and I loved him once. My mother saw that I looked after him properly and he grew up into a beautiful grey Alsation. We were good friends for eleven happy 10 before he went to join his friends in the Animals’ Happy Hunting Ground.

Answers.

1. I; 2. just, recently; 3. mother (mummy, mum, mom); 4. hear; 5. them; 6. near (nearly, nearer, close to; 7. but, however (though); 8. show (tell); 9. wasn’t (was not); 10. years.

Pienaar allotted six minutes for each passage. I allotted 12 minutes. Ability is dependent on speed of processing and so if a test-taker does badly with an allotted time of six minutes,perhaps results would not be much significantly better after six minutes. However, the effect of time on performance is not easy to gauge.

3.3.2 The essay tests

3.3.2.1 Theoretical overview

Language processing involves various components such as linguistic knowledge, content knowledge, organisation of ideas and cultural background. All these factors mesh together into a proficiency network of vast complexity, which makes objective evaluation of essay performance very difficult. It is this vast complexity that makes the written discourse, or essay writing, the most pragmatic of writing tasks and the main goal of formal education.

Essay writing is arguably the most complex of human tasks. If getting the better of words in writing is usually a very hard struggle for mother-tongue speakers, the difficulties are multiplied for the second language learner, and very difficult for disadvantaged or second language learners such as those described in this study. Many of the disadvantaged Grade 7 subjects are similar to young mother-tongue speakers of English first learning to write in that much mental energy is expended on attention to linguistic features rather than to content.

What makes essay writing a pragmatic task, in contradistinction to tasks at the sentence level, is that essay writing involves writing beyond the sentence level. This does not mean that non-pragmatic tasks are not integrative. As discussed in section 2.5, all language resides along a continuum of integrativeness, where pragmatic tasks are the most integrative.

Owing to the fact that the production of linguistic sequences in essay writing is not highly constrained, problems of reliability arise in essay scoring. (In this respect, essay tests have much in common with oral tests). One problem is that owing to the fact that it is inferential judgements that have to be converted into a score, “[h]ow can essays or other writing tasks be converted to numbers that will yield meaningful variance between learners?” (Oller, 1979). Oller argues that these inferential judgements should be based on intended meaning and not merely on correct structural forms. That is why in essay rating, raters should rewrite (in their minds, but preferably on paper) the intended meaning. Perhaps one can only have an absolutely objective scoring system with lower-order skills (Allen in Yeld), but Oller is not claiming that his scoring system is absolutely objective, but only that as far as psychometric measurement goes, it is a sensible method for assessing an individual’s level within a group.

Whatever one’s paradigm, structural (“old”) or communicative (“new”), when one marks an essay one can only do so through its structure. The paradox of language is that the structure must “die” so that the meaning may live; yet, if structure is not preserved, language would not be able to mean.

In the normal teaching situation, marking is done by one rater, namely the teacher who teaches the subject. Sometimes if a test is a crucial one, for example an entrance test or an end-of-year examination, more than one rater, usually two, are used. In a research situation, the number of raters depends on the nature of the research and the availability and proficiency of raters. The raters used for the essay tests in this study were all mother-tongue speakers of English, and recognised as such by their colleagues. (In the dictation tests there were three English mother-tongue presenters and one non-mother-tongue presenter).

With regard to the level of English proficiency of raters, it does not follow that because a rater (or anybody else) is not a mother-tongue speaker (of English in this case) that his or her English proficiency is necessarily lower than a mother-tongue speaker of English. In the academic context, there are many non-mother-tongue speakers of English who have a higher level of academic English proficiency (CALP) than mother-tongue speakers of English. A major reason for this is not a linguistic reason, but because these non-mother-tongue speakers are more academically able, i.e. they have better problem-solving abilities and abilities for learning content.

Kaczmarek (1980) reports high correlations between essay raters, while Hartog and Rhodes (1936) and Pilliner (1968), contrary to Kaczmarek, found essay tests to have low interrater and low intrarater reliability. With regard to scoring procedures in essay testing. Mullen (1980:161) recommends the use of four “scales” (criteria) of writing proficiency: structure, organisation, quantity, vocabulary. According to Mullen (1980), a combination of all four scales is required to validly predict proficiency. A major issue in scoring is whether marks should be separately allocated to each of the criteria, i.e. whether one should use an analytic scoring procedure or whether marks should be allocated globally. Global scoring usually refers to two ways of scoring: (1) “Overall impressions” and (2) “Major constituents of meaning”, which takes into account global errors, e.g. cohesion and coherence, but not local errors, e.g. grammar and spelling.

The following terms are used interchangeably in the literature: global rating, overall impressions, holistic scoring and global impressions (specific authors will be mentioned). The term holistic scoring is used by Perkins to refer to overall impressions, which takes into account global as well as local errors. With regard to the two ways of global scoring mentioned in the previous paragraph, it is possible that a rater’s “overall impressions” may include quick, yet thorough attention to major constituents of meaning as well as to local errors. In such a case the distinction within global scoring between “overall impressions” and “major constituents of meaning” would no longer be very useful.

With regard to the relative reliability of analytic and global scoring procedures, Kaczmarek and Ingram have shown that analytic scoring procedures are not more reliable than global scoring. According to Perkins, “[w]here there is commitment and time to do the work required to achieve reliability of judgement, holistic evaluation of writing remains the most valid and direct means of rank-ordering students by writing ability.”

Zughoul and Kambal (1983:100) also report “no significant difference between the two methods”, and Omaggio (1986:263) maintains that “holistic scoring has the highest validity” (reliability?).

According to Oller (1979:392) “judges always seem to be evaluating communicative effectiveness regardless whether they are trying to gauge ‘fluency’, ‘accentedness’, ‘nativeness’, ‘grammar’, ‘vocabulary’, ‘content’, ‘comprehension’, or whatever. Even if one does this quickly, say a minute per page, the brain of the rater still has to consider the trees to get an overall idea of the wood. The greater the experience and competence of the rater the more unconscious and quicker, but no less rational, is the judgement.

It is arguable whether judges always seem to be evaluating communicative effectiveness, as Oller’s believes. Although it seems reasonable that in essays one should be looking at the overall impact of a piece of writing (the whole) and that the only way to do this is to look at the various aspects of the writing such as those mentioned by Oller, I would question whether raters do regard communicative effectiveness as the overarching criterion. Unfortunately, I did not manage to obtain the judgements of the raters of the MHS essay protocols and so could not investigate this important issue. I did, however, at a later stage, use some of the MHS protocols on a different set of raters to verify whether Oller’s cautious observation was reasonable. (More about this in section 4.8.1).

If global or holistic scoring is more effective or even as effective as analytic scoring, then for reasons of economy global scoring should be used. However, global scoring ideally requires at least three raters (Ingram 1968:96), who would presumably, and hopefully, balance one another out. The effectiveness of global scoring depends on factors such as availability, willingness and qualifications of raters.

The unavailability of raters is often a problem. In special circumstances such as proficiency tests used for purposes of admission or placement at the beginning of an academic year, it may be possible to obtain the help of three or four raters. However, in the normal testing situation during the school year, only one rater may be available, who is usually the teacher involved in teaching the subject. Hughes (1989:87) recommends four raters because this has been shown to be the best number. Four raters were used in this study.

To ensure high interrater reliability there should only be a narrow range of scores and judgements between raters. If three or four raters are considered to be required for reliability a serious problem is what to do in the normal education situation where at most two and usually only one rater is available. I shall discuss this issue in the final chapter where “improving testing in South Africa” is dealt with.

3.3.2.2 The essay tests used in the study

There were two essays:

Essay 1: Everybody in this world has been frightened at one time or another. Describe a time when you were frightened. Write between 80 and 100 words.

Essay 2: Do (a) or (b) or (c). Do only one of the following topics. Don’t forget to write the letter (a) or (b or (c) next to the topic you choose. The topic you choose must not be shorter than 80 words and not longer than 100 words.

(a) Describe how a cup of tea is made.

(b) Describe how shoes are cleaned.

(c) Describe how a school book is covered.

Both the L1 and L2 group’s essays were judged in terms of English-mother- tongue proficiency.

The TOEFL Test of Written English (Hughes 1989:86) recommends spending a rapid one and a half minutes per page using a holistic scoring method. I would imagine that when working at such a speed, the scoring criteria are assumed to be known to the point of automaticity. Raters in this study were recommended to spend about one and a half minutes on each protocol, where protocols were much shorter than a page in length.

As I discussed earlier, clarity and consistency of judgements are difficult to ensure. The TOEFL scoring method seems to be the same as the “overall impressions” approach of Perkins (1983), which takes into account global as well as local errors.

Four raters, including myself, rated 86 protocols. The other three raters were Grade 7 teachers at the School, who were the same teachers who had also participated in the administration of the test battery. As I mentioned earlier (section 3.3.3.1) these three raters were all mother-tongue speakers of English, and were also recognised to be such by their colleagues, of which I was one. Each rater, in turn, was given the original 86 protocols and was requested to give an impressionistic score based on such considerations as topic relevance, content and grammatical accuracy.

Raters did not provide any judgements. The reason for this was because these raters, who were also the Grade 7 teachers at the School, were fully involved in the three-day administration of the test battery; accordingly I did not want to overload them with too much extra work after the three days were over, because they had to return to a full teaching load. Thus, they merely gave a score based on a global impression. It would have been useful to compare raters’ scores and judgements because this would have provided insights into the knotty problem of interrater reliability. I did manage at a later stage to obtain data on the same essay test (given to the Grade 7 subjects in this study) from a workshop on language testing that I conducted (Gamaroff 1996c, 1998c). (More about this in section 4.8.1).

Bridgeman (1991:9) recommends that each rater assign a holistic score on a six point scale, where zero is given if the essay is totally off the topic or unreadable. A nine-point scale was used in this study: from Scale 1; 0 to 1 point = totally incomprehensible, to Scale 9; 9 to 10 points = outstanding. The points were converted to percentages.

Raters did not record their scores on the protocols but were each provided with a copy of the list of the names of subjects on which they had to record their scores. Raters were requested not to consult one another on the procedures they used to evaluate the protocols. The results, as with all the tests in the study, are presented in the next chapter.

3.3.3  Error recognition and mixed grammar tests

3.3.3.1 Theoretical overview

Grammar is an important component in most standardised test batteries, e.g. English as a Foreign Language (EFL), English Language Testing Service (ELTS), Testing English as a Foreign Language (TOEFL) (O’Dell 1986). Error recognition has been used in various studies on language proficiency testing. Olshtain, Shohamy, Kemp and Chatow (1990:31) use it as part of a battery of first language proficiency tests to predict second language proficiency. The emphasis in Olshtain et al. is on appropriacy, i.e. language use, and not on acceptability, i.e. grammatical correctness. Different to Olshtain et al.’s aim of trying to find a connection between error recognition and language use, Irvine, Atai and Oller (1974:247) use the multiple-choice error recognition test from the TOEFL battery of tests to find out whether integrative tests such as cloze tests and dictation tests correlate more highly with each other than they do with the multiple-choice tests of TOEFL.

Henning, Ghawaby, Saadalla, El-Rifai, Hannallah and Mattar’s (1981) revised GSCE (Egyptian General Secondary Certificate Examination) test battery contains an error identification test and a grammar accuracy test. Henning et al. (1981) found that the highest correlation with their “composition” subtest was with Error identification (.76). They accordingly maintain that “Error Identification may serve as an indirect measure of composition writing ability.” (Henning et al. 1981:462).

A grammar component has always featured prominently in all the standardised English first and second language proficiency and achievement tests of the Human Sciences Research Council (HSRC). The recent tests of the HSRC range across various school levels from junior secondary school to senior secondary school (Barry, Cahill, Chamberlain, Lowe, Reinecke and Roux, undated).

3.3.3.2 Error recognition and mixed grammar tests used in the study

The error recognition test and the mixed grammar test in this study are both multiple choice tests that have been designed for learners who have completed the “elementary” stage (Bloor et al. 1970) of second/foreign language learning. These two tests each comprise 50 items. Bloor et al. (1970:ix – Teacher’s book) state that tests were analysed by the authors over an extended period subjected to an item analysis and validated under test conditions. No data on this item analysis or validation under test conditions were provided by the authors, probably because that kind of data would not appear in a Teacher’s book. I shall show (Chapter 4) that their tests have high (split-half) reliability and high correlations with other test methods.

Bloor et al. (1970) divide their tests into three levels; First Stage/ Elementary Stage, Second Stage/Intermediatory Stage, and Third Stage/Advanced Stage. Although the levels have been designated relative to one another, these are merely guidelines, and therefore the tester must use discretion in fitting the level to the relevant group of students. I have used the First Stage level for the Grade 7 subjects.

The tests were administered at the same sitting over a one-and-a- half-hour period, the emphasis being on completing the task and not on speed. It is true, however, that ability is dependent on speed of processing.Sample items from the error recognition and mixed grammar tests are now provided.

Error recognition test: Sample items

The test used was Test 1 from Bloor et al. (1970:70-77; Book 2).

Instructions: In some of the following sentences there are mistakes. (There are no mistakes in spelling and punctuation). Indicate in which section of the sentence the mistake occurs by writing its letter on your answer sheet. If there is no mistake, write E.

Example: (THE LETTERS ARE TOO MUCH TO THE RIGHT BUT IT IS NOT DIFFICULT TO SEE WHERE THEY BELONG)

A                                     B

Although he has lived in England/since he was fifteen,/he

C                             D

still speaks English/much badly. Correct – E.

Answer: D.

A                          B                            C

Item 8. Both Samuel and I/are much more richer/than we/

D

used to be. Correct – E.

Answer: B.

A                                  B

Item 19. Some believe that/a country should be ruled/by

C                                  D

men who are/too clever than ordinary people. Correct – E.

Answer: D.

A                              B

Item 25. His uncle is owning/no fewer than ten houses,/

C                          D

and all of them/are let at very high rents. Correct – E.

Answer: A.

A                              B

Item 27. As I have now studied/French for over three

C                                          D

years/I can be able to/make myself understood when I go to France. Correct – E. Answer: C

Mixed Grammar test: Sample items

The test used is from the “First Stage: Test 2 in Bloor et al., (1970:35-40; Book 1). (By “mixed” is meant a variety of grammatical structures were tested).

The test consists of choosing the correct alternative that fits into the gap within a sentence. The following are the instructions and an example from the test, followed by five selected items from the test:

Instructions: Choose the correct alternative and write its letter on your answer sheet.

Example: His sister is….than his wife. A) more prettier B) prettier C) very pretty

D) most pretty. Answer: B.

Item 1. They often discuss…. A) with me B) about whether there is a problem C) the problem D) about the problem with me. Answer: C.

Item 28. This dress was made… A) by hands B) by hand C) with hands D) with hand.

Answer: B.

Item 30. When the door-bell…., I was having a bath. A) rang B) rings C) rung D) ringed. Answer: A.

Item 38. My friend always goes home….foot. A) by B) with C) a D) on. Answer: D.

Item 50. We….our meat from that shop nowadays. A) were never buying B) do never buy C) never buy D) never bought. Answer: C.

The mixed grammar tests and error recognition tests of this study (Bloor et al. 1970) are commonly used tests. Compare these test items with equivalent test items from the Egyptian study of Henning et al. (1981). Consider the following two items from their test battery:

Grammar Accuracy

Ahmed enjoys….us. A. helping B. to help C. help to D. helping to

The item requires the selection of one of the four options. This test has the same format as Bloor et al.’s (1970) mixed grammar.

Error Identification

A                     B                       C                           D

In my way to school,/ I met a man/ who told me that/ the school was on fire.

One has to choose the incorrect segment in a sentence, i.e. item A. This format is almost identical to Bloor et al.’s (1970) error recognition test. The difference is that Bloor et al.’s test has five options, which makes Bloor’s test more difficult.

I judged Bloor et al.’s “elementary stage” to be appropriate for the Grade 7 subjects. If this is correct it suggests that the standard of grammatical knowledge that is required for Egyptian university entrants (Henning et al. 1981) is very similar to the standard required for Grade 7 non-mother-tongue speakers of English. Quite alarming. Further, as mentioned, Henning et al’s Error Identification test is easier than Bloor’s, because it only has four options whereas Bloor’s error recognition test has five options. The greater the number of options the more difficult it is to guess.

3.3.4 The dictation tests

3.3.5.1 Introduction

Listening comprehension “poses one of the greatest difficulties for learners of English” (Suenobu et al. 1986:239). This section examines language proficiency through the dictation test, which is the most demanding type of listening comprehension test, because it forces the test-taker to focus on structure as well as meaning.

3.3.4.2 Theoretical overview

Language tests involve one or various combinations of the four language modes, namely listening, speaking, reading and writing. Although the most important combination in education, except for the early school years, is usually reading-writing (Lado 1961:28), the

listening-writing combination, which is what a dictation test measures, also plays an essential role. The dictation test is a variation of a listening comprehension test where subjects write down verbatim what they listen to.

Some authors regard the dictation test merely as a test of spelling or of grammar (Lado 1961; Froome 1970:30-31). For Lado (1961) dictation was only useful as a test of spelling because dictation, he argued, did not test word order or vocabulary, both of which were already given; neither did it test aural comprehension, owing to the fact that the learner could often guess the context. For protagonists of the audio-lingual method the dictation test was considered to be a “hybrid test measuring too many different language features, mixing listening and writing, and giving only imprecise information” (Tönnes-Schnier and Scheibner-Herzig 1988:35). Savignon (1983:264) maintains that dictation does not test communicative proficiency. The reasons for its popularity, she suggests, is that it has high concurrent validity, is easy to develop and score, and has high reliability.

Contrary to these negative views, other authors regard dictation as a valid test of “pragmatic” language (Oller 1979:42 1983; Bacheller 1980:67; Cziko 1982; Larsen- Freeman 1987; Bott and Satithyudhakarn 1986:40; Tönnes-Schnier and Scheibner-Herzig 1988). Dictation for these authors is a robust test of the ability to reconstruct surface forms to express meaning at the sentence level and beyond. For Oller (1976:61) and Tönnes-Schnier and Scheibner-Herzig (1988:37-38), dictation tests are valid measures of communicative proficiency. Spelling, which for Lado was the dictation test’s only justification, is disregarded by some of these authors (e.g. Oller 1979:278; Bacheller 1980:69).

There is substantial evidence in the interdisciplinary co-operation between research in linguistic pragmatics and reading comprehension to show that reading and listening employ the same underlying processing strategies (Horowitz and Samuels 1987; Hoover and Gough 1990; Vellutino et al. 1991). Vellutino et al. (1991:107, 124) found significant correlations between reading and listening comprehension in young children and in adults. Of the four language skills, reading performance was found to be the best predictor of listening performance, and vice versa. (For different views on the relationship between listening and reading see Atkin, Bray, Davidson, Herzberger, Humphreys and Selzer 1977a, 1977b; Jackson and McClelland 1979; Carroll 1993:179-80).

Cloze and dictation have been found to reveal similar production errors in writing (Oller 1976:287ff; Oller 1979:57), and a combination of cloze tests and dictation tests have been used effectively in determining general language proficiency (Stump 1978; Hinofotis 1980). Oller (1979:61) maintains that all pragmatic tasks such as dictation tests probe the same underlying skill. The reason why a dictation and a cloze test (which are apparently such different tasks) intercorrelate so strongly is that both are effective devices for assessing the efficiency of the learner’s developing grammatical system, or language ability, or pragmatic expectancy grammar. This underlying skill is overall or general language proficiency. Spolsky (1989:72) describes the overall proficiency claim in the following “necessary” condition: “As a result of its systematicity, the existence of redundancy, and the overlap in the usefulness of structural items, knowledge of a language may be characterized as a general proficiency and measured.”

Tönnes-Schnier and Scheibner-Herzig (1988) compared Oller’s method and Bacheller’s (1980:71) “scale of communicative effectiveness” – where both methods distinguish between spelling errors and”real” errors – with the relatively much simpler traditional method, where different errors are not distinguished. Tönnes-Schnier and Scheibner-Herzig (1988:38) maintain that “thissimplifiedway of marking errors in dictations chosen in accord with the learners’ level of structures and vocabulary proves an effective way to rank a class of learners according to their communicative capacities.” The fact that a superficial and reductionist method such as the traditional method of counting surface errors could rank learners according to their communicative capacities shows that reductionist methods of testing can predict “pragmatic” language, i.e. language that straddles sentences. (Analogously, an eye that is involved in an eye test is no less alive looking at letters on an optician’s screen than reading a book or looking at the sunset). Recall (section 2.5) Spolsky’s (1989:61) suggestion that the “microlevel” is “in essence” the “working level of language, for items are added one at a time”, keeping in mind that “any new item added may lead to a reorganisation of the existing system, and that items learnt contribute in crucial, but difficult to define ways to the development of functional and general proficiency.” The micro-level is not less alive than the micro-level, just as an eye in a socket is not less alive than the brain that turns sensation into perception. Accordingly we can talk about a symbiotic relationship between “reductive”, or “discrete”, micro-levels and “holistic”, or “pragmatic”, macro-levels.

For Ur (1996:40), dictation “mainly tests spelling, perhaps punctuation, and perhaps surprisingly, on the face of it, listening comprehension”. Tönnes-Schnier and Scheibner-Herzig’s (1988) “surface” (discrete?) findings expose our “abyss of ignorance” (Alderson (1983:90). Constructs may not be lurking beneath the surface after all, but staring us in the face; or more accurately lurking beneath the surface and staring us in the face. The German term aufheben (sublation) illustrates the paradox. This term means “to clear away” as well as “to preserve” : the simultaneous preservation and transcendence of the structure/meaning antithesis. Language (i.e. language structure) has to be cleared away and preserved in order to convey its meaning. Coe (1987:13) usessublation to explain the paradox of structural preservation and transcendence. According to Scholes (1988), however, sublation remains inconsequential from a practical point of view. Certainly not inconsequential from the practical point of view of language testing.

3.3.4.3 The dictation tests used in the study

Excerpts from two restored cloze tests of Step 2 of Pienaar’s (1984) “Reading for meaning” were used. Step 2 corresponds to Grade 5 and Grade 6 for English mother-tongue speakers, and Grades 7 to 9 for English non-mother-tongue speakers. These dictation passages were different from the passages that were used for the cloze test (which were Forms B and D). For the dictation tests, I used the restored texts of Forms A and C of Pienaar’s Step 2. Thus, all four passages – two for the cloze test and two for the dictation test – belong to the same level. (Recall that Step 2 corresponds to Grades 5 and 6 for English First Language and to Grades 7 to 9 for English Second Language).

I judged the conceptual difficulty of the word sequences to be within the range of academic abilities required of Grade 7 learners who have to use English as the medium of instruction. I decided to pitch the dictation test at the L2 level and not at the L1 level, because I suspected that STEP 3, which was meant for the Grade 7 L1 level, would be too difficult for the Grade 7 L2 subjects. Accordingly, I used the passages of STEP 2, which were aimed at the Grade 7 L2 level.

Test 1

The fire

We were returning from a picnic up the river when the fire-engine raced past us. Of course we followed it. We hadn’t gone far when we saw black smoke pouring from an old double-storey house in the high street. When we drew nearer we saw angry tongues of flame leaping from the downstairs windows. There was already a curious crowd watching the fire, and we heard people say that there was a sick child in one of the upstairs bedrooms. A black cat was also mentioned.

(86 words)

Test 2

A close call

It was early evening and we were driving at a steady ninety when a small buck leapt into the road about a hundred metres ahead of us. At the last moment it swerved and ran directly towards us. I flicked on the headlights and swerved at the same time. The car slithered to a halt in a cloud of dust, and it was only then that we saw why the buck had changed direction. A number of sinister shapes were hard on the Duiker’s* heels. Wild dogs!

(87 words)

*Duiker is a South African species of small buck.

The reason for the choice of these dictation passages was the same as Pienaar’s for the choice of his cloze passages (see section 3.3.1.2) which was “to select lexical and structural items relevant to the demands of the appropriate syllabuses” (Pienaar 1984:3), i.e. relevant to English as the medium of instruction. However, the relevant demands of the appropriate syllabuses cannot be separated from general language proficiency, which is often the hardest part of learning English for ESL learners (Hutchison and Waters 1987).

3.3.4.4 Presentation of the dictation tests

The following procedures were used:

1. The degree of difficulty of the texts was regulated by controlling factors such as speed of delivery and length of segments between pauses. The text was read at a speed that preserved the integrative nature of the sequences, while catering for subjects who might not have been able to write at the required speed. The length of sequences between pauses was also sufficient, which satisfied the requirements of both mechanical speed and speed of information-processing.

2. The background noise level was kept to a minimum.

3. The text was presented three times. Once straight through, which involved listening only, a second time with pauses, and a third time without pauses, but at a speed that allowed for quick corrections.

4. And very importantly for this study, more than one presenter was used. This procedure is explained in the next section.

It is normal procedure in a dictation test to use one presenter for all subjects – in this case all four groups. It has been argued that a ” dictation can only be fair to students if its presented in the same way to them all” (Alderson, Clapham and Wall 1995:57), i.e. using only one presenter; and old procedure for an “old” method/test.

In this study, I used “old” tests, but the procedure of presentation was new. If one is using indirect tests such as dictation, this does not mean that one has to stick to “old” procedures. One can still try to be exploratory.

The normal procedure in a dictation test is to use one presenter even when subjects are split up into different venues/classrooms. Owing to the exploratory nature of the dictation tests, four presenters (including myself) were used. The presenters then repeated the process on a rotational basis so that each of them presented the two dictation tests to all four groups. The dictation scores used in this study were the scores of the first presentation of each presenter. Thus, I did not use any scores for the statistical analysis from dictations that had been heard more than once by the subjects. Table 3.3 shows the procedure of the first rotational presentation.

TABLE 3.3

The reason why I chose such a design instead of doing the usual and simple thing of using one presenter was because I wanted to investigate whether different presenters, i.e. different methods of presentation, would have any significant effect on the results. Three of the presenters were English mother-tongue speakers, and one was a Tswana mother-tongue speaker. In order to test for any significant difference in the means between the results of the four different presenters, I did an analysis of variance (One-way ANOVA). The ANOVA results are presented and discussed in section 4.1. The reason why an ANOVA was used is because I had to test whether there was any significant difference between the results of the four procedures of administration: each rater’s presentation represents a different procedure. An ANOVA deals with the four sets of data simultaneously, whereas a T-test can only deal with two at a time.

The results of an ANOVA are more reliable than the results of six T-tests (AB, AC, AD, BC, BD, CD), which would be required if the four sets of data (represented by the results of the four different presentations) had to be submitted to T-tests. The more T-tests you have the less the reliability of the results: “If means are to be cross-compared, you cannot use a t-test” (Hatch and Farhady 1982:114). The cross comparison, or, as Brown (1988:170) calls it “multiple t-tests”, may be between more than two groups, or more than one test on the same two groups. Brown (1988:170) gives the following example of this pitfall:

[A] researcher might want to compare the placement means, including four sets of subtest scores, of the males and females at a particular institution. In such a siutation, it is tempting to make the comparisons between the means for males and females on each of the four subtests with four different t tests. Unfortunately, doing so makes the results difficult to interpret. yet I have seen studies in our literature that make 2,3,4,5, and more comparisons of means on the same groups ot test scores.

(See Hatch and Farhady [1982:114] who provide the statistics that show why multiple t-tests increase the likelihood of being able to reject the null hypothesis).

3.3.4.5 Method of scoring the dictation tests

Various methods of scoring dictation are examined and reasons are given for the scoring method used in this study:

Cziko (1982) uses the method of scoring by segment using an exact-spelling criterion where a point is awarded for a correct segment on condition that there is no mistake in the segment. Bacheller (1980) created a scale of communicative effectiveness, where spelling, unlike in Cziko above, was disregarded. In Bacheller’s (1980) method each segment is rated on a scale of 0 to 5 according to how much meaning is understood. A score of zero indicates that none of the intended meaning of the segment has been captured; a score of 3 indicates that the subject apparently understands the meaning of the segment”; a score of 5 indicates that the meaning is understood (see also Tönnes-Schnier and Scheibner-Herzig 1988). Owing to the fact that the emphasis in Bacheller is on the top-down process of coherence/meaning, this method tends to be subjective, especially if only one rater is involved.

Tönnes-Schnier and Scheibner-Herzig (1988) compared Bacheller’s method with the “traditional German method” of scoring dictation (henceforth referred to as the traditional method), where all words are counted, including spelling – one point for each word. Thus, the total score is the number of possible correct words minus the number of errors.

In the method used by Oller (1979:276, 282) and Stump (1978:48) each correct word is worth one point. One point is deducted for each deletion, intrusion or phonological or morphological distortion. Spelling errors, punctuation and capitalisation are not counted. As in the case of the traditional method, the total score is the number of possible correct words minus the number of errors. In the traditional method of counting errors, the “different kinds of errors” are not distinguished. Thus spelling errors – unlike in Oller’s method – are lumped together with omissions, intrusions, lexical and grammatical errors.

Cziko (1982:378) found that his “exact-spelling segment scoring procedure…was three to four times faster than an “appropriate-spelling word-by-word scoring system”. The reason is that one only has to look for one mistake in each segment, whereas in Oller’s procedure one has to take into account each and every error. Cziko (1982:375) found a correlation of .89 between his method and Oller’s method. In my method I did not have to count every word or mistake but decided on a maximum possible score of 20 (before the test was given), which was determined by the difficulty of the test. In this study I used a variation of the traditional method, where errors were subtracted from a possible score of 20 points. One point was deducted for any kind of error, including spelling, and the actual score was deducted from a possible score of 20. This was done because in my opinion this method yielded a valid indication of the level of proficiency of individual subjects. If one is only interested in norm-referenced tests, it wouldn’t matter what the possible score was, because in norm-referenced tests one is only interested in the relative position of individuals in a group, not with their actual scores. One could then measure the correlation between this procedure and Oller’s procedure. If the correlation is found to be high, one could use the shorter procedure. I did a correlational analysis on the dictation tests between Oller’s method and my variation of the traditional method (a possible 20 points). High correlations were found. These are reported and discussed after the ANOVA results in section 4.3).

All the dictation protocols (N=86) were marked by myself. The main reason for this was that the teachers/presenters did not have time for a lot of marking, because the administration of the tests had to be done within the limited time allotted within the School’s programme. Recall that the dictation tests were only a part of a large battery of tests. The teachers/presenters of the dictation tests were involved in the administration of the whole battery. At the end of section 4.2 I discuss the issues of rater reliability when using a single rater.

3.4 Summary of Chapter 3

The sampling procedures, and the structure, administration and scoring procedures of the tests were described. Although subjects are divided into an a L1 sub-group and an L2 sub-group the two sub-groups should be treated as a composite group in the correlational analyses that are to follow in Chapters 4 and 5. The reason for this is that the following conditions were the same for all the subjects:

(1) The admission criteria to the School.

(2) The English proficiency tests and their administration (this investigation).

(3) The academic demands of the School.

(4) The treatment they were given at the School. What is relevant to the statistical rationale of this investigation is not the fact that the entrants to the School had received different treatment prior to entering the School, where some may have been disadvantaged, but only the fact that the all entrants received the same treatment after admission to the School.

(5) The proportion of L1 and L2 learners (as I have defined these labels) was similar from year to year at the School.

All five conditions show that the 1987 Grade 7 sample represented subjects who came from the same population of Grade 7 learners at the school from year to year, specifically from 1980 to 1993, irrespective of their origin and whether they are divided into “L1″ and “L2″ groups.

Various methods of administration and scoring  procedures in the different tests were appraised to show the reason for the methods of administration and scoring procedures chosen for  this study. The next chapter presents the results and statistical analysis of the battery of proficiency tests.

2.2 Ability, cognitive skills and language ability

2.3 Competence and performance

2.4 Proficiency

2.5 The discrete-point/integrative controversy

2.6 Cognitive and Academic Language Proficiency (CALP) and “test language”

2.7 Language proficiency and academic achievement

2.8 Validity

2.8.1 Face validity

2.8.2 Content validity

2.8.3 Construct validity

2.8.4 Criterion validity: concurrent and predictive validity

2.9 Reliability

2.9.1 Approaches to the measurement of reliability

2.10 Ethics of measurement

2.11 Summary of Chapter 2

2.1 Introduction

Whatever we say about language – or about anything – originates from a theory, i.e. a combination of knowledge, beliefs, wants and needs. For some authors^[1] ( the testing  of language proficiency has been a circular enterprise. Vollmer^[2] maintains that “language proficiency is what language proficiency tests measure” and  this circular statement is all that can be firmly said when asked for a definition of  language proficiency. Perhaps this is all that can be firmly established about language proficiency, but we swim on in the hope of  reaching terra firma.

2.2  Ability

I examine the notion of ability first and then discuss language ability. In the next sections I move on to competence and performance, which I shall relate to ability. I then discuss proficiency and the discrete-point/integrative controversy in language testing. As I mentioned in the first paragraph of the study, language testing draws on three areas: the nature of language, assessment, and language ability . The latter is closely related to language proficiency.

The precise definition of ability is not only seldom explicated, but, unlike competence, is often not even considered, in spite of the fact that the term is  used widely in everyday language as well as in scientific circles.  Important issues in the study of abilities, of which language  is only one ability, are:

- The fixety of abilities. If  abilities were highly variable over time they would reflect a state rather than a trait or an attribute. The two latter terms imply a fixed structure, or construct,  rather than a variable process. The constructs that this study is concerned with belong to the domain of language acquisition. (Construct validity is discussed in section 2.8.3).

- Consistency. For example, if an athlete  in a one-off streak “accident” breaks a world record, but is never able to repeat the performance, or even get near the record again, we still say that he or she has the ability to break a world record. We cannot apply the same logic to cognitive abilities, where  consistency of output, not records, is the name of the game. Consistency does not only apply to the ability of learners but also of teachers, who are usually also testers. The consistency, or the reliability, of judgements and scoring is a major issue in language testing. This issue is dealt with in various parts of study.

- When we say people have the ability to perform academically we mean that they are able to achieve a certain liminal level, i.e. minimum or threshold level. In trying to set a minimum level, one is concerned with what the individual can do in terms of established criteria. What the individual can do cannot be separated from what others can do. Hence the importance of norm-referenced tests .

- The variability in ability between individuals obeys a “bell-curve” distribution, as in the case of nature as a whole. The “bell-curve” or “normal” distribution  is the foundational principle of psychometrics.^[3]

Taking  the four points above into account, Carroll^[4] suggests the following definition of ability:

As used to describe an attribute of individuals, ability refers to the possible variations over individuals in the liminal levels of task difficulty (or in derived measurements based on such liminal levels) at which, on any given occasion in which all conditions appear favorable, individuals perform successfully on a defined class of tasks.

Several modern theories of education and psychology reject the notion of traits, i.e. the fixety of psychological constructs^[5]. In traditional trait theories, e.g. Carroll (above), psychological constructs are like any other human trait, animal trait, or plant trait, where biological differences between living things are distributed according to a bell curve and differences in human abilities are also distributed according to the bell curve. This does not mean that people cannot improve, but only that the degree of improvement depends on fixed psychobiological constraints.^[6]

A few comments on Carroll’s idea that ability is a fixed psychological trait is in order. Ability is closely connected to the notion of proficiency (to be discussed shortly) and proficiency is certainly something that can improve by using the correct strategies to enhance learning. The notions of “transferable” and “transferring skills” are used to explain the idea of “fixed” ability.

A major problem with learners with limited academic ability is the underdevelopment of “transfer skills”^[7]. There are two kinds of transfer skills: (1) lower order “transferable skills” and (2) higher order “transferring skills”^[8]. Transferable skills are skills that are learnt in one situation or one kind of subject-matter that are transferable to another. Examples are:  (i) a reading skill such as scanning that is learnt in the English class can be transferred to the geography class, (ii) using a dictionary, (iii) making charts and diagrams, (iv) completing assignments, (v) reviewing course material, (vi) learning formulas and dates, (vii) memorising material. “Transferring skills” are “metacompetences” of a far higher order. These metacompetences are : (i) A sensitive and intelligent discernment of similarities and differences, (ii) Cognitive equipment that one  uses to modify, adapt and extend, and (iii) attitudes and dispositions that support both of the above.^[9]

These three “metacompetences” are interrelated. For example, without the “cognitive equipment” that enables one to modify, adapt and extend, it would not be possible to sensitively and intelligently discern similarities and differences. With regard to Bridges’ third “metacompetence” of “attitudes and dispositions”, which have to deal with intention, motivation and resulting approach to a task, I suggest that its successful development is to a large degree dependent on the successful development of the other two “metacompetences”. If one has the right healthy cognitive equipment, as well as the desire and opportunity to develop it, one will understand more; consequently, one will be more motivated to learn. Of course, socialisation into a community of learners and the correct mediation/intervention procedures between learner and task also  play an important role in cognitive development, e.g. the development of critical awareness and learning strategies.

Bridges’^[10] distinction between lower order “transferable skills” and higher order “transferring skills” is useful in understanding the nature of the problem of transfer. The problem of transfer refers mostly to the higher order “transferring skills”. The question is whether higher order cognitive skills (i.e. Bridges’ “transferring skills”) can be acquired at all (whether independently or through teaching). Millar^[11] maintains that courses in skills development (e.g. the development of executive processes) pursue the “impossible” because processes such as classifying and hypothesising cannot be taught, but can only develop (i.e. they are part of inborn potential, or ability). For Millar the challenge is to find ways of “motivating pupils to feel that it is personally valuable and worthwhile to pursue the cognitive skills (or processes) they [children] already possess to gain understanding of the scientific concepts which can help them make sense of their world”^[12] (square brackets and italics added).

According to Millar, these cognitive skills, especially the higher order transferring skills (e.g. a sensitive and intelligent discernment of similarities and differences), can only be developed if they are based on something that learners already possess, namely, academic potential, or ability. I have raised some highly controversial issues, but they needed to be raised in order to explain what I meant by “fixed” ability. I cannot pursue the matter further here.^[13]

I now discuss how the term ability is used in relation to language. In section 1.1,  I mentioned the four major test uses, achievement, proficiency, aptitude and diagnosis. These are all manifestations of what Davies’ calls “language ability”.^[14]

I mentioned above that “fixed” ability in Carroll’s sense does not mean that people cannot develop and become better. If this were not so, it would be nonsensical to talk about things such as transitional competence and interlanguage, which feature, justifiably, so strongly in the applied linguistic literature. In the next section I discuss the notions of competence and its sibling, performance.

2.3  Competence and performance

The notions of competence and performance are essential to understanding language assessment.

For Chomsky^[15] competence is the capacity to generate an infinite number of sentences from a limited set of grammatical rules. This view posits that competence is logically prior to performance and is therefore the generative basis for further

learning.^[16] Competence, on this view, is equivalent to “linguistic” (or “grammatical”) competence. Chomsky distinguishes between “performance”, which is ”the actual use of language in concrete situations”, and  “competence” or “linguistic competence” or “grammatical competence”, which is “the speaker-hearer’s knowledge of his language”.^[17] Chomsky’s description of language involves no “explicit reference to the way in which this instrument is put to use…this formal study of language as an instrument may be expected to provide insight into the actual use of language, i.e. into the process of understanding sentences.”^[18] Chomsky’s great contribution was to focus on linguistic introspection, without giving introspection (linguistic intuitions) the final word.^[19]

Canale and Swain^[20] make a distinction between knowledge of use and a demonstration  of this knowledge. Knowledge of use is often referred to in the literature as “communicative competence”^[21], and the demonstration of this knowledge as performance. Communicative competence has come to subsume four sub- competences: grammatical competence, sociolinguistic competence, discourse competence and strategic competence^[22]:

(1) Grammatical competence is concerned with components of the language code at the sentence level, e.g. vocabulary and word formation.

(2) Sociolinguistic competence is concerned with contextual components such as topic, status of interlocutors, purposes of communication, and appropriateness of meaning and form.

(3) Discourse competence is concerned with: (i) a knowledge of text forms, semantic relations and an organised knowledge of the world; (ii) cohesion – structural links to create meaning, and (iii) coherence – links between different meanings in a text; literal and social meanings, and communicative functions.

(4) Strategic competence, which is concerned with (i) improving the effectiveness of communication, and (ii) compensating for breakdowns in communication. Strategic competence means something very different in Bachman and Palmer , namely, metacognitive strategies, which is central to communication. For these authors “language ability” consists of “language knowledge” and “metacognitive strategies”.^[23] (See Skehan^[24]).

According to Widdowson communicative competence should subsume the notion of performance:

[T]he idea of communicative competence arises from a dissatisfaction with the Chomskyan distinction  between competence and performance and essentially seeks to establish competence status for aspects of language behaviour which were indiscriminately collected into the performance category.^[25]

How does ability fit into the competence-performance distinction? Chomsky equates “ability” with “performance” (“actual use”),  which he regards as a completely different notion from “competence” or knowledge”.

Characteristically, two  people who share the same knowledge will be inclined to say quite different things on different occasions. Hence it is

hard to see how knowledge can be identified with ability…Furthermore, ability can improve with no change in knowledge.^[26]

Thus, as Haussmann points out, “it should be noted that Chomsky’s original definition of the term [i.e. competence] always excluded this idea [i.e. ability].”^[27] There doesn’t seem to be any reason, however, why “ability” cannot refer to (linguistic/grammatical) competence, (which is Chomsky’s interest), as well as to the knowledge one has of how to use the language in appropriate situations. We can retain performance  to mean the actual use of this knowledge. For example, Bachman and Clark use the term “ability” in the following way:

We will use the term “ability” to refer both to the knowledge, or competence, involved in language use and to the skill in implementing that knowledge, and the term “language use” to refer to both productive and receptive performance.^[28]

Weir equates “ability” with “competence” as well:

There is a potential problem with terminology in some recent communicative approaches to language testing. References are often made in the literature to testing communicative ‘performance’ [e.g. B.J. Carroll 1980^[29]]. It seems reasonable to talk of testing performance if the reference is to an individual’s performance in one isolated situation, but as soon as we wish to generalise about ability  to handle other situations, ‘competence’  would seem to be involved.^[30] (Square brackets added)

This is Skehan’s position as well: “it is defensible to speak of competence-orientated abilities.”^[31]

In other words, different performances point back to the underlying  competence or ability.

“Competency-based education and training” has a different set of concepts for the labels of competence, performance and ability to those discussed above.CBET is discussed in section 6.4 where the future of assessment in South Africa is dealt with.

2.4  Proficiency

Proficiency is closely related to ability, competence and performance discussed above. Proficiency is used in at least two different ways: it can refer to  (1) the “construct or competence level”^[32], which is at a given point in time independent of a specific textbook or pedagogical method^[33] or to (2) the “performance level”^[34], which is a reflection of achievement in the test situation. The construct level or competence level is the knowledge of the language, and the performance level is the use of language.

Proficiency, like the notions of competence and performance,  is very much of a “chameleon” notion^[35], because it can be defined not only in terms of knowledge (the construct or competence level) and in terms of specific tasks or functions (the performance level), but also in terms of degrees of behaviour that are observed at different stages (minimum to native-like^[36]), in terms of language development (e.g. interlanguage studies), in terms of situations that require some skills but not others, or in terms of general proficiency, where no specific skill is specified.

Porter^[37] uses the term  “communicative proficiency”, which seems to subsume the notions of  “communicative competence” and “performance” discussed above. According to Child^[38], “proficiency” is a “general `across-the-board’ potential”, while “performance” is the “actualised skill”, the “mission performance” involved in “communicative” tasks, i.e. the output. Child has much in common with Alderson and  Clapham^[39], who distinguish between “language proficiency” and “language use”, where proficiency, not use, is part of output.

I would like to spend some time on Landtolf and Frawley’s^[40] views on language proficiency because they epitomise the opposition to the view that I am arguing for in this study. These authors will be referred to as L and F. In their abstract they state that they “argue against a definitional approach to oral proficiency and in favor of a principled approach based on sound theoretical considerations.” (The L and F reference throughout is their 1988 article). The authors use oral proficiency as a backdrop to their views on language proficiency in general. L and F,  in their criticism of ”reductionism” in the assessment, of language proficiency,  leave few authors unscathed; authors that many would consider to be in the vanguard of the real-life/communicative movement, e.g. Hymes, Omaggio and Widdowson.

To adumbrate: in the second section of their article “The tail wagging the dog”, L and F use Omaggio’s^[41] section of her manual entitled :”Defining language proficiency” to lament that the “construct of proficiency, reified in the form of the [American Council on the Teaching of Foreign Languages- ACTFL] Guidelines, has begun to determine how the linguistic performance of real people mustbe perceived”:

In her discussion, she considers various models of communicative competence, including those of Hymes, Munby, Widdowson, and Canale and Swain, all of which are reductionist approaches to communicative competence, because they define communicative competence by reference to a set of constitutional criteria. She then proceeds to a subsection entitled “From Communicative competence to Proficiency.” However, nowhere in her analysis is there any in-depth consideration of proficiency that is independent of the proficiency test itself.^[42]

Strange that L and F consider Widdowson^[43] a reductionist, who I would think fully appreciates the distinction between language structure and language in use, where grammar plays a vital role. By “grammar”, Widdowson does not mean merely morphology, phonology and syntax but lexico-grammar, where semantics is included under “grammar”. The inclusion of semantics under “grammar”, or “linguistic knowledge”, is what modern linguistics understands by these terms). The papers of the Georgetown University Round Table Conference^[44] were concerned with the reality and authenticity of communicative language proficiency, where Widdowson argued that grammar  is not dead, but the life blood of language, communication and social meaning. Such a view is not reductionist!

L and F  reject the ACTFL’s  (American Council of the Teaching of Foreign Languages) adoption of a uniform yardstick for the measurement of foreign language ability based on real-life  behaviour^[45]. The ACTFL’s tail (the series of real life descriptors) that is wagging the real dog is not, according to L and F, a real tail. The unreal tail for L and F is the unreal “construct”; the real dog being wagged is real people. The metaphor is clear: it is researchers who have fabricated the “construct”, and fabrications have no psychological reality.  In other words the construct constricts the reality of ”the nontest world of human interaction”^[46]. The test world, which represents the “construct” for these authors, “has come to determine the world, the reverse of proper scientific methodology”.^[47]

Recall that L and F are arguing in “favor of a principled approach based on sound theoretical considerations” (italics added), which L and F  seem to think authors such as Widdowson do not  use. Yet Widdowson (who was probably not unaware of L and F’s criticism) ends his  “Aspects of language teaching” with the following:” There needs to be a continuing process of principled pragmatic enquiry. I offer this book as a contribution to this process – and as such, it can have no conclusion”^[48] ( italics added). (See Gamaroff 1996a^[49]).

Widdowson perceives the content of both the structural and the notional syllabus to be, in Nunan’s words, “synthetic” and “product-orientated”^[50], i.e. the content of both syllabuses is static and lacks the power to consistently generate communicative behaviour. Widdowson’s argument against structuralist and notional syllabuses is that “[i]t has been generally assumed…that performance is a projection of competence…that once the rules are specified we automatically account for how people use language.”^[51] His argument is that structural and functional-notional syllabuses do not link in past experiences with  new experiences, because they lack proper learner involvement^[52]. Widdowson also believes that “the most effective means towards this achievement [i.e. "complete native-speaker mastery"] is through an experience of authentic language in the classroom.”^[53]

L and F, and Widdowson are backing the same communicative horse. The main difference between them appears to lie in the value they place on school learning. All three believe in teaching language as communication, with the difference that much of Widdowson’s work is concerned with academic achievement and school learning rather than with real-life “natural” contexts.

I examine more closely the cogency of the distinction between “natural” contexts in “real-life” and “unnatural” contexts in the classroom. According to L and F “tasks cannot be authentic by definition”^[54], which implies that very little in school is authentic, i.e. natural. The nub of L and F’s criticism is that the exchange between tester and test-taker is not a natural one,  therefore any kind of test cannot be a natural kind of communication. Communicative testing, it seems, would be for L and F a contradiction in terms. What is more, communicative school “tasks” would also be a contradiction in terms. In that case, school, which may be defined as an institution whose role it is to guide learners by defining and dispensing tasks, is another tail wagging the world (of  “reality”). The ACTFL Guidelines, according to L and F, draw a line between the world and the individual. L and F regard such a situation as scientifically unprincipled and morally untenable. There is very little in “tasks” such as  instructional activities and nothing in tasks such as tests that L and F find authentic in the Proficiency literature. L and F want language tasks to  be contextualised in natural settings such as cooking clubs.

Byrnes and Canale caution that the danger of ”the proficiency movement” as espoused by L and F and others as “with any movement is that a rhetoric of fear and enthusiasm will develop which is more likely to misrepresent and confuse than to clarify the crucial issues.”^[55] One confusing issue is that of “natural environment”.

“Just what is a ‘natural environment’ as far as learning or acquiring a second language under any circumstances is concerned?’” asks Morrissey^[56] :

There is no environment, natural or unnatural, that is comparable with the environment in which one learns one’s mother tongue. Furthermore, it seems to me that there is a teaching (i.e. unnatural?) element in any L2-L1 contact situation, not just in cases of formal instruction. This element, even if it only consists in the awareness of the communicants that the [teaching or testing] situation exists, may be a more significant factor in L2 learning  and L2 acquisition [and L2 testing] than any other factor that is common to [the natural setting of]  L1 acquisition and L2 acquisition.^[57]

L and F are seeking a testing situation analogous to the L2 “acquisition” situation (which by Krashen’s^[58] definition is “natural”). But as Morrissey above suggest much of language and learning, like culture, consists of  extrapsychological elements, and in this sense are an “imposition” upon nature. However, although the test situation, i.e. school,  may be less “authentic” in the sense that the test is more concerned with learning language than with using it, with regard to the laws of learning, the dichotomy between “natural” and “unnatural” is a spurious one. It is incorrect to assume that “natural” approaches (e.g. Krashen and Terrell^[59]) and immersion programmes mirror natural language acquisition and that the ordinary classroom doesn’t.^[60] The swimming club, cooking club, tea party or cocktail party are in a sense not more neither less natural than the traditional classroom. That is why one doesn’t have to go outside the classroom in of search for “real reality”^[61]. The learning brain needs stimulation, and it can get it in the classroom or at an informal (or formal) cocktail party. In other words, there is much informal learning in classrooms and much formal learning outside the classroom. But both kinds of learning are completely “natural” to the brain that is doing the learning.

2.5  The discrete-point/integrative controversy

L and F point out that in real life one uses far less words than one would use in school “tasks”, and this is one of the reasons why they maintain that tests are  inauthentic by definition.^[62] However, as Politzer and Mcgroarty^[63] show, it is possible to say or write few words (as one often does in natural settings) in a “communicative competence” test by using a “discrete-point” format. When one uses far less words in natural settings than one would use in many “artificial” school tasks, one is in fact using a “discrete-point” approach to communication. One doesn’t merely look at the format of a test to decided whether it is a “discrete-point” test, one looks at what the test is testing.

It is now opportune to examine what tests are testing. The way I have chosen to do so is by means of an examination of  the discrete-point/integrative controversy.

This controversy can only fully understood within the context of a parallel controversy: the structuralism/functionalism controversy. The discussion of the former controversy will serve as a background to the discrete-point/integrative controversy.

It is impossible to test the structures and functions of language without understanding how it is learnt. Language learning is language processing. The central issue in testing is assessing this language processing skill. Language processing, as with all knowledge, exists within a hierarchical organisation: from the lower level atomistic “bits” to the higher level discoursal “bytes”. The lower level bits traditionally belong to  the  ”structuralist” levels, while the higher level bytes belong to the “functionalist”  levels. It is difficult to know where structure ends and function begins.^[64].

The following continuum, adapted from Rea^[65], includes the concepts on competence and performance discussed earlier.

TABLE 2.1

Functionalist and Structuralist Levels of language

This mutually exclusive classification of functionalism and structuralism is a highly controversial one. The structuralist/functionalist controversy is about whether the semantic meaning of words and sentences (structuralism) can be distinguished from the  pragmatic (encyclopaedic, i.e. world knowledge) meaning of  discourse (functionalism).

Halliday^[66] proposes two meanings of the term function, namely,  “functions in structure” and “functions of language”. “Functions in structure” is concerned with the relationship between different words of a sentence. Structuralism is traditionally associated with the study of language at the sentence level and below. “Functions of language”, on the other hand, goes beyond individual linguistic elements or words (Saussure’s “signs”) to discourse. Functionalism is traditionally associated with the study of discourse.

I understand the terms linguistic knowledge, lexico-grammar (which is what recent modern linguistics understands by grammar ) and Halliday’s “functions in structure” to be synonymous. Therefore, lexico-grammar only deals with linguistic knowledge at the sentence level and below that level. Halliday’s “functions of language”, what I call functionalism deal with discourse, i.e. the intersentential domain. (The division of language  into a sentence level and an intersential level is itself problematic).

Functionalism rejects the Chomskyan idea that grammar is logically and psychologically the origin of “functions in language” (Halliday above). For functionalists like Halliday^[67], the grammar of a specific language is merely “the linguistic device for hooking up the selections in meaning which are derived from the various functions of language”.

In functionalism it is communication that is claimed to be logically and psychologically prior to grammar. Givon^[68], for whom the supreme function of language is communication, criticises Chomsky for trying to describe language without referring to its communicative function. Givon argues: “If language is an instrument of communication, then it is bizarre to try and understand its structure without reference to communicative setting and communicative function.”^[69] Rutherford), whose view is similar to Givon’s communicative view, rejects the “mechanistic” view that grammatical structure (Givon’s “syntax”) is logically or psychologically prior to  communication.^[70] Rutherford sees language as a  dynamic process and not a static “accumulation of entities”.^[71]

In this regard Spolsky^[72] suggests that the “microlevel” is “in essence” the “working level of language, for items are added one at a time”, keeping in mind that “any new item added may lead to a reorganisation of the existing system, and that items learnt contribute in crucial, but difficult to define ways to the development of functional and general proficiency.” Thus, according to Spolsky, building up the language from the microlevel to the macrolevel need not  be a static “accumulation of entities” (Rutherford above), but maylead toa dynamic “reorganisation of the existing system” (Spolsky above). Alderson’s view is similar to Spolsky’s:

Another charge levelled against (unidentified) traditional testing is that it views learning as a ‘process of accretion’. Now, if this were true, one would probably wish to condemn such an aberration, but is it? Does it follow from an atomistic approach to language that one views the process of language as an accretion? This does not necessarily follow from the notion that the product of  language learning is a series of items (among other things). (Original emphasis).^[73]

The process and product methodologies “are too often perceived as generally separate”, i.e. they suffer from an “oppositional fallacy”.^[74] The product is considered to be discrete, static and, accordingly, is not party to language processing,while the process is considered to be integrative and dynamic, and accordingly, the process is seen as belonging to language processing. (More about this in section 6.4). It is this oppositional fallacy that is the battleground of the discrete-point/integrative controversy.

It is widely believed that tests such as essay tests test the “use” of language, i.e. authentic communicative language, while tests such as error recognition tests and grammar accuracy tests test the “usage” of language^[75], i.e. the elements of language. Such a distinction between the two kinds of tests, which Farhady describes as the “disjunctive fallacy”^[76], is an oversimplification.

Many studies report high correlations between “discrete-point tests” and ”integrative tests”.^[77] It may be asked how the construct is able to account for this: “Shouldn’t supposedly similar types of tests relate more to each other than to supposedly different types of  tests?”  An adequate response presupposes three further questions: (1) “What are similar/different types of tests?” (2) Wouldn’t it be more correct to  speak of so-called discrete-point tests and so-called integrative tests?  (3) Isn’t the  discrete/integrative dichotomy irrelevant to what any test is measuring?

I consider some of the issues in the discrete-point/integrative controversy that are related to the questions posed above. The notion of “real-life” tests is also critically examined.

The terms “integrative” and “discrete-point” have fallen out of favour with some applied linguists, while for others these terms are still in vogue. For example, Fotos equates “integrative”  skills with “advanced skills and global proficiency”^[78], which he contrasts with Alderson’s”basic skills”^[79]. These “basic skills” are Alderson’s “low order” skills^[80]. Alderson prefers to distinguish between “low order” and “higher order” tests  than between “discrete- point” and “integrative” tests.^[81] Alderson in 1979^[82] refrained from talking about discrete-point and integrative tests, but preferred to talk of “low order” and “higher order” tests. Yet, in his later collaborative textbook on testing, one of the book’s test specifications is that “tasks” should be “discrete point, integrative, simulated ‘authentic’, objectively assessable”.^[83] These tests specifications would dovetail with the notion that although these tests do not mirror life, they are nevertheless “good dirty methods [of testing] overall proficiency”.^[84]

Whatever one’s classification, all tests, except for the most atomistic of tests, reside along a continuum of “integrativeness”.^[85] For example,  consider  two items from Rea^[86]:

1. How —-  milk have you got?

(a) a lot (b) much of (c) much (d) many

2. —- to Tanzania in April, but I’m not sure.

(a) I’ll come (b) I’m coming (c) I’m going to come (d) I may come.

Item 1 is testing a discrete element of grammar. All that is required is an understanding of the “collocational constraints of well-formedness”^[87], i.e. to answer the question it is sufficient to know that “milk” is a mass noun (see also Canale and Swain^[88]). Item 2 relates form to global meaning. Therefore, all parts of the sentence must be taken into account, which makes it an integrative task. To use Rea’s terminology, her item 1 is testing “non-communicative performance”, while her item 2 (above) is testing what she calls “communicative performance”.^[89] Other discrete-point, or “low order”, items could be shown to be more integrative, or “higher order”, than the items described above. (The terms in inverted commas are Alderson’s [1979^[90]]).

Above I described an “objective” type of test. Consider now a test that lies toward the “pragmatic” extreme of the integrative continuum: the cloze test. Although cloze answers are short, usually a single word, the cloze test can still be regarded as an “integrative” test. A distinction needs to be made between integrative and discrete formats and integrative and discrete processing strategies. The salient issue in a cloze test or in any test is not the length of the answer or the length of the question,  but whether the test measures what it is supposed to measure, in this case integrative processing strategies. One should distinguish between the structure of the test  -  long answer, short answer, multiple choice  -  and what one is measuring. One is measuring the natural ability to process language and one component of this ability is the behaviour of supplying missing linguistic data in a discourse.

According to the “pop”^[91] view, it is only in language use that natural language processing can take place. Although the “pop” view may not conflict with the idea of a continuum of integrativeness, such a view would nevertheless hold that language tests should only test language “use”, i.e. direct language, or authentic language. For language “naturalists” the only authentic tests are those presented in a direct real-life situation. Spolsky^[92] maintains that  “authenticity of task is generally submerged by the greater attention given to psychometric criteria of validity and reliability, where face validity receives no more than “lip service”. For Spolsky and others^[93], authenticity is closely related to communicative language, i.e. to direct language. Authentic tests for Spolsky would be “direct” tests  in contradistinction to “indirect” tests. Owing to the lack of clarity on the relationship between a global skill like composition writing and the components of composition writing, e.g. vocabulary, punctuation and grammar, Hughes recommends that it is best, in terms of one’s present knowledge, to try and be as comprehensive as possible, and the best way to do this would be to use direct tests.

“Direct” testers argue that in language use we do not process language in a multiple choice way as in the case of discrete-point tests. Yet, many multiple choice tests do test processing strategies, e.g. the making of predictions. Furthermore, multiple choice tests are neutral in themselves, i.e. they can serve any purpose; communicative or non-communicative. Rea^[94] gives the following reasons why indirect tests should be used:

1. There is no such thing as a pure direct test.

2. Direct tests are too expensive and involve too much administration.

3.  Direct tests only sample a restricted portion of the language, which makes valid inferences difficult. (Of course, no battery of tests can sample the whole language. Rea’s point seems to be that indirect tests are able to be much more representative than direct tests).

If it could be shown that indirect test performance is a valid predictor of direct performance, this would be the best reason for using indirect tests. Even if indirect performance is accepted to be a valid predictor of direct “natural” performance, one may object that indirect tests are unnatural, and consequently lack face validity. As mentioned earlier.

The laws of learning and testing apply to all contexts, “naturalistic”^[95] and otherwise. One can have authentic indirect tests, because tests are authentic activity types in their own right^[96]. The quality of learning outcomes depends, of course, on the quality of input – and more importantly on the quality of intake.

There is a sense, though, in which “real-life” “authentic” tasks in the classroom, if not a contradiction in terms, are not possible: in the sense that learners are aware that life in the classroom is a preparation for, and simulation of, life outside the classroom – an understanding of life, which comprises not only life skills but content knowledge in specific disciplines and an understanding of their relationship. But this “preparation for life” view of the classroom, does not justify, I suggest, the radical rupture between “real-life” and the classroom, described by Lantolf and Frawley (see end of section 2.4). Tritely, life is one big classroom; and less tritely, the classroom is one small part of life. This does not mean that the classroom has to be turned into a cooking club or a cocktail lounge to get learners to respond authentically to a recipe or to something “stronger” – for instance, a test.

If by some good fortune we come of age in our understanding of what an “authentic” task is (and, accordingly, isn’t), it still doesn’t follow that it is necessary to do “authentic” tasks in order to prove that we are proficient to do them, because communicative tasks can be tested successfully through indirect tests^[97]. For example, an eye test doesn’t directly, or “holistically”, measure whether someone can see the illuminated road clearly, but its a jolly good predictor of whether one will be able to see, if not avoid, that oncoming road-hog on that same illuminated road.

In sum, both direct tests and an indirect tests – as in all direct and indirect classroom activities – have communicative, or real-life, language as their aim. The difference lies in this: direct  tests, or outcomes, or activities are based on the view that communicative language should be directly taught and tested, while indirect tests are based on the view that indirect teaching materials and tests are a prerequisite and solid basis for ultimate real-life language. But I would go even further and agree with Widdowson that “semantic meaning is primary” (Chomsky’s dated! “linguistic competence”) where semantic meaning should (naturally, i.e. obviously) be internalised  to provide for “communicative capacity”^[98]; which is the same idea as Spolsky’s^[99] (mentioned above) where the building up of language from the microlevel to the macrolevel may be a dynamic and not necessarily a static “accumulation of entities” (Rutherford^[100] above), which in turn leads toa dynamic “reorganisation of the existing system”.^{[101] [102]}

2.6 Cognitive and Academic Language Proficiency and “test language”

Cognitive and Academic Language Proficiency (CALP) is closely related to the ability to do tests. Its features are better understood when  compared and contrasted with Basic Interpersonal and Communicative Skills (BICS).^[103] BICS refers to salient basic features such as fluency (speed of delivery) and accent, and not to advanced social and communicative skills, which is a cognitively demanding task.  For example, the skills of persuading or negotiating in face-to-face communication require relatively much more cognitive involvement than a BICS task, and are therefore cognitively demanding CALP tasks. Thus, it would be incorrect to equate BICS with all face-to-face communication, because face-to-face communication may involve informal as well as formal speaking. Formal speech acts such as persuading and negotiating belong to advanced communicative skills, and are consequently part of CALP. Spoken language can be just as complex as written language. They differ in that speech is dynamic while writing is synoptic, and writing is lexically denser than speech: “written language does not have to be immediately apprehended in flight and does not need to be designed to counter the limitations of processing capacity”.^[104]

Cummin’s BICS and CALP  have affinities with Bernstein’s “restricted code” and “elaborated code”, respectively^[105]. The “elaborated code” has the following features:  precise verbalisations, large vocabulary, complex syntax, unpredictability, low redundancy, individuality differences between speakers; in contrast, the “restricted code” has the following features: loose verbalisations, limited vocabulary, simple syntax, high redundancy where assumptions are based on shared social experience.

Wald makes a distinction between “test language” (spoken and written), which he equates with CALP, and “spontaneous language”/”face-to-face” communication.^[106] For Wald, test skills are CALP skills, which can involve all four language skills: listening, speaking, reading and writing. For example, an oral cloze test would be a CALP task.  In terms of these distinctions, it would be possible to have tests of basic language (grammar tests). Basic language tests  involve CALP because, according to Wald, they are tests. All tests are formal, no matter how “natural” one tries to make them. In terms of Wald’s definition of CALP as test language, the tests in this study are CALP tasks because they are tests. Accordingly, if  Wald is correct, and I think he is,  one could not have a BICS test.

Ur uses the term “informal”^[107] not in the sense of natural, but in the sense that test takers are not told in advance what they need to know for a test. One could, for example, spontaneously test learners on their homework. On such an interpretation of “informal”, it follows that there can be “informal” tests (“informal” CALP tasks).

2.7  Language proficiency and academic achievement

Much of the research in second language acquisition involves finding factors which affect language proficiency. In such a research scheme, factors such as intelligence, motivation, mother-tongue interference and socio-economic standing are defined as the independent variables and language proficiency is defined as the dependent variable. Language proficiency in such a research context does not look beyond itself to its effect on academic achievement.

In the investigation of academic achievement the focus changes from considering language proficiency as the dependent variable (the criterion) to considering academic achievement as the dependent variable, as in Saville-Troike^[108].

Consider the following schema. (The schema is highly simplified and is merely meant to present some of the predictor variables that could be involved in academic achievement and is therefore not a comprehensive “model”):

FIGURE  2.1

Second  Language Proficiency as a Criterion Variable and as a Predictor Variable

FIRST FOCUS – Second language proficiency as the criterion

Predictor variables Criterion variable

Intelligence

Motivation  (active participation)Second language

Mother-tongue interference                                                             Proficiency

Socio-economic standing

Personality (e.g. emotional maturity)

SECOND FOCUS – Academic achievement as the criterion

Predictor variables                                                                  Criterion variable

Intelligence

Motivation                                                                               Academic achievement

Mother-tongue interference

Socio-economic standing

Second language proficiency

Subject learning

One needs to know how language proficiency, which is embedded in other factors, promotes or hinders academic achievement. These other factors comprise an complex network of variables such as intelligence, learning processes and styles, organisational skills and content knowledge,  teaching methods, motivation and cultural factors. Owing to the complexity of the interaction between these variables, it is often difficult, perhaps impossible, to isolate them from language proficiency, which means that any one or any combination of these above-mentioned variables might be the cause of academic failure. Therefore, care must be taken not to make spurious causal links between any of these variables and academic failure. Although prediction does not necessarily imply causation, this does not mean that prediction should be ignored; on the contrary, prediction plays a very important role in the selection and placement of candidates. What is important is that these predictions be valid. (I stress that this study focuses mainly on those causes of academic failure that are related to the testing situation, e.g. rater unreliability).

Although the distinction between the two kinds of focus (Figure 2.1) can be useful, the change from one focus to the other does not merely involve rejuggling the variables that were previously used to predict language proficiency (the first focus) and then assigning them to the new game of predicting academic achievement (the second focus), where language proficiency, previously a criterion variable, would then become another predictor variable among those that were previously used to predict it. The reason is that when academic achievement is brought into the foreground, the predictive mechanism becomes far more complex. One cannot merely shift variables around, because in the second focus, learning in or through a second language is added to the demands of learning the second language itself.

Upshur^[109] distinguishes between two distinct general questions: “Does somebody have proficiency?” and “Is somebody proficient?”  The first question considers such issues as grammatical competence and the use of language in discourse. The second question is concerned with the ability of language proficiency tests to predict future performance in tasks that require language skills, i.e. with the “prerequisites for a particular job or course of study”^[110]. It is this second question,  namely, “Is somebody proficient?” (to do a particular task) that is the main concern of educationists.

2.8  Validity

Validity is concerned with “the purposes of a test”^[111], which is basically concerned with the meaning of scores and the ways they are used to make decisions. A major difficulty in this regard is ensuring that one’s descriptions of validity are validly constituted, which involves reconciling “objective” reality with one’s own interpretation of ”objective” reality – a daunting and  probably circular task.

For some researchers, validity comprises face validity, content validity, construct validity and criterion validities (concurrent and predictive validity), whereas for others, especially those belonging to the American Psychological Association^[112] (APA), construct validity itself is validity. Face validity does not feature in the APA’s definitions of validity. The reason for this is explained in the next section.

2.8.1  Face validity

Face validity is concerned with what people (which includes test analysts and lay people) believe must be done in a test, i.e. what the test looks like it is supposed to be doing.

For Clark^[113], face validity, oddly, covers the “whole business” of tests, i.e. looking “at what it’s got in it, at the way it is administered, at the way it’s scored.” Clark’s definition is unusual, because it covers everything to do with testing. Clark’s meaning of face validity is not what a test looks like to the non-tester but what it is to the tester, who should know from what it looks like what it is, i.e. “what it’s got in it”.

Spolsky’s meaning of face validity has affinities with Clarke’s. Spolsky equates face validity with “authenticity”: “authenticity of task is generally submerged by the greater attention given to psychometric criteria of validity and reliability”, where “face validity receives no more than lip service”.^[114]

For Davies “face validity is desirable but not necessary, cosmetic but useful because it helps convince the public that the test is valid.”^[115] The reason why face validity is desirable, according to Davies, is that, in spite of its “cosmetic” nature, it can still have a “major and creative influence for change and development”^[116]. Yeld maintains that face validity should be capitalised on as a point of entry into testing for those “who have not been trained in the use of techniques of statistical analysis and are suspicious of what they perceive as ‘number-crunching’.”^[117]

Thus, face validity (what Stevenson calls “pop” validity is so popular today because many language teachers have a poor knowledge of language testing and educational measurement, i.e. they are “metrically naive” ^[118]. Accordingly, they could remain satisfied with superficial impressions.

There are others who reject face validity altogether, because it relies too much on the subjective judgement of the observer^[119]:

Adopting a test just because it appears reasonable is bad practice; many a `good-looking’ test has failed as a predictor… If one must choose between a test with `face validity’ and no technically verified validity and one with technical validity and no appeal to the layman, he had better choose the latter.^[120]

Gardner and Tremblay^[121] consider face validity to be the lowest form of validity, and should, accordingly, not be generally recommended as a research strategy. The difficulty with face validity in its usual connotation of what a test appears to be is that the prettier the package the worse may be the inherent quality of the tests. No matter what one’s opinions of face validity, it does have the following useful features: it increases a learner’s motivation to study for the test; it keeps sponsors happy; and it sustains the parents’ resolve to pay the ever-escalating school fees.

2.8.2 Content validity

Face validity and content validity can overlap, because what must be done in a test involves content. The latter subsumes subject matter as well as skills. Content validity  “implies a rational strategy whereby a particular behavioural domain of interest is identified, usually by reference to curriculum objectives or task requirements or job characteristics”^[122]. Content validity is concerned with how test items represent the content of a syllabus or the content of real-life situations. Content validity is not only a match between (the situation, topic and style of ) tests and  real-life situations but also a match between tests and school life, both of which are part of “real” life.

2.8.3  Construct validity

The constructs, or human abilities, that this study is interested in belong to the domain of language acquisition. As I mentioned earlier (section 2.2), abilities are fixed attributes, or constructs (in the sense of consistent, not immutable). If behaviour is inconsistent it would be impossible to find out what lies behind the behaviour, i.e. discover the construct. The problem for scientists, whether physical scientists or linguistic scientists, is figuring out the nature and sequence of the contribution of (abstract) theory and (concrete) experience to construct validity.

Consider how evidence for construct validity is assembled. There are two main stages:  (1) hypothesise a construct and (2) construct a method that involves collecting empirical data  to test the hypothesis, i.e. develop a test to measure the construct. Hypothesising is concerned with theory, while the construction of a method, although inseparable from theory, is largely an empirical issue. The problem is that it is not clear whether theory should be the cart and experience  the horse, or vice versa, or some other permutation. Consider some of the problems in assessing the relative contribution of theory and experience in construct validatiFor Messick^[123] construct validity is a unitary concept that subsumes other kinds of (sub-)validities, e.g. content validity and criterion validity. Messick^[124] defines validity, which for him is construct validity, as  a

unitary concept that describes an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferencesand actions based on test scores or other modes of assessment. (Original emphasis)

“Test scores” above refers to “quantitative summaries” (Messick 1987:3), which is the commonly understood meaning of the term. Messick’slonger and much denser definition of construct validity

implies a joint convergent and discriminant strategy entailing both substantive coverage and response consistency in tandem. The boundaries and facets of a behavioural domain are specified, but in this case as delineated by a theory of the construct in question. Items are written to cover that domain in some representative fashion, as with content validity, but in this approach the initial item pool is deliberately expanded to include items relevant to competing theories of the construct, if possible, as well as items theoretically irrelevant to the construct… Item responses are then obtained, and items are selected that exhibit response homogeneity consistent with the focal theory of the construct but are theoretically distinct from theoretically irrelevant items or exemplars of competing theories. ^[125]

What Messick’s definition loses in brevity and simplicity it gains in scientific  rigour. In Messick’s definition one should have a theory and only then a method. And the theory must be able to specify the problem without prejudging a solution: something very difficult to do. We try and ensure a “substantive coverage” (Messick above) of entities or qualities that are similar (a convergent strategy) and of those that are different (a discriminant strategy), which is “delineated by a theory of the construct in question” (Messick above). The problem is knowing what items to include or exclude in a test, because owing to the infinity of the corpus and the fact that elements and skills hang together^[126], it is difficult to distinguish between “items relevant to competing theories of the construct, if possible” and “items theoretically irrelevant to the construct”? (Messick above; italics added). The Unitary Competence controversy is concerned with the nature and degree of interdependence between elements and skills, i.e. how, how much and which elements and skills hang together. Difficulties exist in the discrimination between items. One of these difficulties is distinguishing between low order (so called “discrete”) items and higher order (so called “integrative”) items, as was shown in the discussion of the discrete-point/ integrative controversy (section 2.5).

The group-differences approach to construct validity that is used in this study is now explained: The aim of testing is to discern levels of ability. If one uses academic writing ability as an example of a construct, one would hypothesise that people with a high level of this ability would have a good command of sentence structure, cohesion and coherence, while people with a low level of this ability would have a poor command of these. Tests are then administered and if it is found that there is a significant difference between a group of high achievers and a group of low achievers, this would be valid evidence for the existence of the construct. Second language

learners are often relatively less competent than fist language or mother-tongue users.^[127]

Important for the arguments presented and the validation of the sample of subjects is that those who take English First Language as a subject are generally more competent than those who take English Second Language as a subject. If a test fails to discriminate between low-ability and high-ability learners there are three possible reasons for this:

- The construction of the test is faulty, e.g. the test may be too easy or too difficult for all or most of the test takers participating in the test.

- The theory undergirding the construct is faulty.

- The test has been inaccurately administered and/or scored, which would decrease the reliability, and hence also the validity of the test. (Reliability is discussed shortly).

2.8.4  Criterion validity: concurrent and predictive validity

Criterion validity is concerned with correlating one test against an external criterion such as another test or non-test behaviour. Ebel^[128] maintains that “unless a measure is related to other measures, it is scientifically and operationally sterile.” Criterion validity should not be confused with criterion-referenced tests. Criterion-referenced tests deal with profiles, i.e. with setting a predetermined cut-off score for an individual.

Criterion validity, which relies mainly on empirical methods, ignores the theoretical contribution of construct and content validity. For this reason, some researchers, particularly those of the American Psychological Association, prefer to dissociate validity from descriptions of the criterion, e.g. Loevinger^[129] and Bachman^[130]. Bachman^[131] prefers the term “concurrent relatedness”  to  “concurrent validity”, and  “predictive utility”  to “predictive validity”.

A term used by Messick is “criterion-related validity”, where the latter “implies an empirical strategy whereby items are selected that significantly discriminate between relevant criterion groups or that maximally predict relevant criterion behaviours”.^[132] Messick’s “criterion-related validity” is the same notion as the simpler term “criterion validity”, which was defined in the first sentence of this section.

Criterion validity consists of concurrent validity and predictive validity.  Concurrent validity is also concerned with prediction because there is only a chronological  difference between concurrent and predictive validity.^[133] So, we could distinguish between concurrent prediction and  prediction proper, which is concerned with the ability of one test to predict another test where the predictor and the criterion are not given concurrently and thus are separated from each other by a reasonable period of time.

The reason why predictive validity is easier to measure than other kinds of validity is that predictive validity does not depend on the nature of the test items, but on the consistency of the predictions of performance.^[134] It would be possible to ignore all the other kinds of validity and still have a high degree of predictive validity. The question is whether one should be satisfied with predictive validity alone. No. That is why this study is also concerned  with construct validity.

If  the  construct validity  of one  test  always depends  on  the  validity  of  another test, there cannot exist any one test that  stands by itself such as an equivalent of a “Prime Mover”. Lado’s solution is to compare all tests in terms of  “some other criterion whose validity is self-evident, e.g. the actual use of the language.”^[135] The question is: What is self-evident? Is there a self-evident test that pre-exists all other tests?  There isn’t, because “the buttressing validity of an external criterion is often neither definable nor, when found, reliable”.^[136] This does not mean, of course, that any test or battery of tests, direct or indirect, will do. The problem, however, remains: what tests will do? ^[137]

Having said that, we don’t need to worry about the difficulty of establishing construct validity if we are merely interested in predictive validity. If Test A is a good predictor of Test B, then it seems we don’t Test C as a second predictor, because Test A is doing a good job already. However, recall the discussion of the “One Best Test” question: we can never be sure, and furthermore, it doesn’t look (face validity) fair to use only one test as a predictor. To do so would be regarded by some researchers as highly unethical. Spolsky^[138] (1995:358) is a case in point:

Only the most elaborate test batteries, with multiple administrations of multiple methods of testing the multiple traits or abilities that make up language proficiency, are capable of producing rich and accurate enough profiles to be used for making critical or fateful decisions about individuals.

Such Herculean conditions, however, would probably “paralyze”^[139] most testing endeavours, because it is impracticable in its unrealisability. We shouldn’t only start  measuring when we are clear about what we are measuring; rather we should do the best we can; always taking into account generally accepted theories, but not necessarily following them slavishly if we have cogent reasons why we shouldn’t.

2.9  Reliability

If the validity of a test depends on its close approximation to real life then validity would relate to subjectivity. We try to be as objective as possible in the test compilation, administration and  assessment. This search for objectivity is the domain of reliability. Reliability in testing is concerned with the accuracy and consistency of scoring and of administration procedures. The less the accuracy and consistency, the more the measurement error.

A major difficulty in testing is how to make the “leap from scores  to profiles “^[140], i.e. how to define the cut-off points. In norm-referenced testing, one  defines cut-off points by computing the measurement error. In criterion-referenced tests, one makes a value judgement of what is progress enough for a specific individual.

To the extent that one can decrease the measurement error, one increases the reliability of the test. Measurement error has important ethical implications. It would be unjust to fail students  because they get 49% – perhaps even 47%, where does one draw the line? – instead of 50%. In subjective tests such as essay tests the problem is more serious, because even the best essay test, owing to its subjective scoring

procedures, is often not more than 80%-90%^[141] reliable, and therefore measurement error should be calculated in order to make more equitable judgements.

The following “aspects” are germane to reliability:

- Facets: These refer to such factors as the (1) testing environment, e.g. the time of testing and the test setting, (2) test organisation, e.g. the sequence in which different questions are presented, and (3) the relative importance of different questions and topic content. Facets also include cultural and sexual differences between test takers, the attitude of the test taker, and whether the tester does such things as point out the importance of the test for a test-taker’s future.^[142]

- Features or conditions: These refer to such factors as clear instructions, unambiguous questions and items that do or do not permit guessing.

- The manner in which the test is scored. A central factor in this regard is rater consistency. Rater consistency becomes a problem mainly in the kind of tests that involve subjective judgements such as essay tests. (I discuss interrater and intrarater consistency in the next section). According to Ebel and Frisbie^[143],  consistency is not only concerned with the correlations between raters, but also with the actual scores, more specifically, the equivalence in scores between tests and between raters. (I discuss this issue in section 4.8.1.2).

I  clarify a possible confusion between rater reliability and concurrent validity.  Rater reliability has to do with the consistency between raters’ judgements on one test, e.g. an essay test.Concurrent validity, in contrast, has to do with the correlation between two or more different tests e.g. a dictation test and an essay test. In the next section more details are provided on the approaches to reliability, which may help clarify the concepts discussed.

2.9.1  Approaches to the measurement of reliability

There are five approaches to measuring reliability. Owing to the structure of this study, only approaches 2, 4 and 5 are used:

1. Stability, i.e. consistency over time. The method used to measure stability is the test-retest method, which involves giving the same test a second time and comparing the scores of the two test trials. If the scores are equivalent, the test is considered to be stable. A disadvantage of the test-retest method is that students may not be motivated to do the test a second time, which might affect performance on the retest.

2.  Internal consistency. This approach, also called the “split-half” test, divides the test into two halves. The two halves of the test are regarded as two parallel tests. For each student there is a separate score for each half of the test. It is possible to correlate the two sets of scores as if they were parallel tests.

3.  Rater reliability. Rater reliability is particularly important in non-objective tests such as essay tests, where there are liable to be fluctuations in scores between (1) different raters, which is the concern of interrater reliability, and (2) within the same rater, which is the concern of intrarater reliability. In this study I use essay assessment to examine interrater reliability (section 4.8.1).

4. Equivalence (in the form of the test). There are two meanings of equivalence:  firstly, the equivalence between test scores, and secondly, between the facets of the tests. The method used to measure equivalence is the parallel test method. In parallel tests it is difficult to ensure equivalent conditions within the many facets of a test, especially whether the content of the two parallel are equivalent. The problem doesn’t exist for multiple-choice type tests because the split-half method is used. In the case of “pragmatic” tests, however,  such as the cloze, dictation and essay tests in this study, there is a problem. This problem is examined in the discussion of the parallel reliability of the pragmatic tests in the study.

5.  A combination of stability and equivalence (in forms). The method used is a parallel test which is administered a period of time after the first test. The difficulties are compounded here, because they include the problems of both equivalence and of stability mentioned above.

The degree of reliability required depends on the relative importance of the decisions to be made. For example, an admission test would require more reliability than a placement test, because decisions based on a placement test can be more easily adjusted than decisions based on an admission test. A final evaluation for promotion purposes would require the most reliability of all.

2.10  Ethics of measurement

Validity should not be separated from what Sammonds^[144] refers to as the following “ethical” questions. Most of these ethical questions are scientific questions as well: (The kinds of validity corresponding to each question are the appellations given by Sammonds).

1. Are the measures that are chosen to represent the underlying concepts appropriate? (“Construct validity”).

2. Has measurement error been taken into account, because in all measurement there is always a degree of error? (“Statistical conclusion validity”; in other words, reliability).

3. Are there other variables that need to be taken into account? (“Internal validity”).

4. Are the statistical procedures explained in such a way that a non- statistical person can understand them? Or sometimes even a statistical person. One reader may find an explanation superfluous or too detailed, while another  may find the same information patchy. Much depends on the background knowledge of readers and/or what they are looking for. Pinker^[145] maintains that expository writing requires writers to overcome their “natural egocentricism” where “trying to anticipate the knowledge state of the generic reader at every stage of the exposition is one of the important tasks of writing well.” True, but there is much more, namely the basic expository problem of negotiating a path between under-information and overkill. Getting experts to read and provide comments before one submits one’s work to public scrutiny is one way of  reducing the expository problem. It may also compound the problem, however, owing to the diversity of beliefs in the world: of interpretations of interpretations (see section 6.2).

4. Has the description of the sample and the data analysis been properly done so that generalisations can be made from it? (“External validity”). This important issue is dealt with in the last chapter (section 7.1).

2.11  Summary of Chapter 2

The first part of the chapter dealt with theoretical issues in language proficiency, language learning, language testing and academic achievement. Key concepts such as authenticity, competence, performance, ability, proficiency, test language, integrative continuum and achievement were explained.

The second part of the chapter was concerned with explaining the key concepts in summative assessment. The two principal concepts in summative assessment are validity and reliability. Different kinds of validity were discussed, namely, content validity, face validity, construct validity and criterion validity, where the latter comprises concurrent and predictive validity. Other kinds of validity were also referred to in the context of the ethics of measurement. The group-differences approach to construct validity, to be used in the study, was described. Different approaches to the examination of reliability were discussed and those chosen for the study were specified.

^[1]Ingram, E.Assessing proficiency: An overview on some aspects of testing, 1985, p.218. (2) Vollmer, H.J.Why are we interested in general language proficiency, 1981, p.152).^[2]Ibid.

^[3]“Norm” is used in the sense of an idealisation against which comparisons are made of what scientists call the “real” world. Although this “normal” curve is a mathematical abstraction it is based on the reasoning that if there were an infinitely large population then human abilities (and the milk yeld of cows) would be represented by a perfect bell curve.

^[4]Carroll, J.B.Human cognitive abilities: A survey of factor analytic studies, 1993, p.10.

^[5]Minick, N.J.L.S. Vygotsky and Soviet activity theory: New perspectives on the relationship between mind and society, 1985, pp.13-14.

^[6]The concept of fixety in the social sciences carries the stigma of colonialism and racism, “an ideological constriction of otherness” (Bhaba 1994:66; see also Leung, Harris and Rampton 1997). This came out clearly in an article by Phatekile Holomisa, president of the Congress of Traditional Leaders of South Africa  (Financial Mail, October 16, 1998, p.22), who maintained that one of the reasons why Transkeians who have made good outside the Transkei do not return to help the rural poor is because they believe that they “might be seen as promoting ubuXhosa (Xhosa-ness), in contradiction to the ideal of nonracialism.” There is also the danger that the concept of fixety could translate into the constriction of the historical individual, which the generalising mode of science considers to have significance only insofar as it reveals a universal rule (Cassirer 1946:27).

^[7]Botha, H.L. and Cilliers, C.D. ‘Programme for educationally disadvantaged pupils in South Africa: A multi-disciplinary approach.’ South African Journal of education, 13 (2),  55-60 (1993).

^[8]Bridges, P.’Transferable skills: A philosophical perspective.’ Studies in Higher   Education, 18 (1), 43-52 (1993), p.50.

^[9]Ibid.

^[10]Ibid.

^[11]Millar, R.‘The pursuit of the impossible.’ Physics Education, 23, 156-159 (1988), p.157.

^[12]Ibid.

^[13]Gamaroff, R.  ‘Solutions to academic failure: The cognitive and cultural realities of English as the medium of instruction among black ESL learners.’ Per Linguam, 11 (2), 15-33 (1995c).

__________   ‘Abilities, access and that bell curve.’ Grewar, A. (ed.).   Proceedings of the South African Association of Academic Development “Towards meaningful access to tertiaty education”. (Alice: Academic Development Centre, Fort Hare, 1996b).

___________  ‘Language as a deep semiotic system and fluid intelligence in language proficiency.’ South African Journal of Linguistics, 15 (1), 11-17 (1997b).

^[14]Davies, A. Principles of language testing, 1990, p.6,.

^[15]Chomsky, N. Aspects of the theory of syntax, 1965, p.6.

^[16](1) Brown, K.Linguistics today, 1984, p.144.

(2) Leech, G. Semantics, 1981, p.69.

(3) Hutchinson, T. and Waters, A. English for special purposes: A learner-centred approach, 1987, p.28.

^[17]Chomsky, N. Aspects of the theory of syntax, 1965, pp. 3-4.

^[18]Chomsky, N. Syntactic structures, 1957, p.103.

^[19]Atkinson, M., Kilby, D. and Roca, I. Foundations of general linguistics, 1982, 369.

^[20]Canale, M. and Swain, M. ‘Theoretical bases of communicative approaches to second language teaching and testing.’ Applied Linguistics, 1 (1), 1-47 (1980), p.34.

^[21]Hymes, D. ‘On communicative competence’, in Pride, J.B. and Holmes, J. (eds.). Sociolinguistics. (Harmondsworth, Penguin, 1972).

^[22](1) Canale, M. and Swain, M. ‘Theoretical bases of communicative approaches to second language teaching and testing.’ Applied Linguistics, 1 (1), 1-47 (1980), p.34.

(2) Swain, S. ‘Large-scale communicative language testing: A case study’, in Lee, Y., Fok, A., Lord, R. and Low, G. (eds.). New directions in language testing. (Oxford,  Institute of English, 1985).

(3) Savignon, S.J.  Communicative competence: Theory and classroom practice.  (Reading, Mass. Addison-Wesley Publishing Company, 1983).

^[23]Bachman, L.F. and Palmer, A.S.Language testing in practice, 1996. (See their Chapter 4).

^[24]Skehan, P. A cognitive approach to language learning, 1998, p.16.

^[25]Widdowson, H.G.Aspects of language teaching, 1990, p.40.

^[26]Chomsky, N. Language and the problem of knowledge, 1988, p.9

^[27]Haussmann, N.C. The testing of  English mother-tongue competence by means of a multiple-choice test: An applied linguistics perspective, 1992, p.16.

^[28]Bachman, L.F. and Clark, J.L.D. ‘The measurement of foreign/second   language proficiency.’ American Academy of the Political and Social Science Annals, 490, 20-33 (1987), p.21.

^[29]Carroll, B.J. 1980. Testing communicative performance. Oxford. Pergamon.

^[30]Weir, C.J. Communicative language testing, 1988, p.10.

^[31]Skehan, P. A cognitive approach to language learning, 1998, p.154

^[32]Vollmer, H.J.The structure of foreign language competence, 1983, p.5.

^[33]Brière, E. ‘Are we really measuring proficiency with our foreign language tests?’ Foreign Language Annals, 4, 385-91 (1971), p.322.

^[34]Vollmer, H.J. Ibid, p.5.

^[35]Hyltenstam, K. and Pienemann, M. Modelling and Assessing second language   acquisition, 1985, p.15.

^[36]The term native is problematic. I discuss this problem in sections 3.2.1 and 6.1.1.

^[37]Porter, D.Assessing communicative proficiency: The search for validity, 1983.

^[38]Child, J. ‘Proficiency and performance in language testing.’ Applied Linguistic Theory, 4 (1/2), 19-54 (1993).

^[39]Alderson, J.C. and Clapham, C. ‘Applied linguistics and language testing: A case study of the ELTS test.’ Applied Linguistics, 13 (2), 149-167 (1992), p.149.

^[40]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988).

^[41]Omaggio, A.C.Teaching language in context: Proficiency-orientated instruction, 1986.

^[42]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.182..

^[43]Widdowson, H.G. Explorations in applied linguistics. (Oxford,  Oxford University Press, 1979).

______________   ‘Knowledge of language and ability of use.’ Applied Linguistics, 10 (2), 128-137 (1989).

______________ Aspects of language teaching. (Oxford,  Oxford University Press, 1990).

______________ ‘Communication, community and the problem of appropriate use’, in Alatis, J.E. Georgetown University Round Table on Languages and Linguistics. (Washington, D.C. Georgetown University Press, 1992).

^[44]Alatis, J.E. (ed.). Georgetown University Round Table on Languages and Linguistics,  1992.

^[45]Byrnes, H. and Canale, M. (eds.). Defining and developing proficiency: Guidelines, implementations and concepts, 1987.

^[46]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.182.

^[47]Ibid.

^[48]Widdowson, H.G.Aspects of language teaching. (Oxford,  Oxford University Press, 1990).

^[49]Gamaroff, R. ‘Is the (unreal) tail wagging the (real) dog?: Understanding the construct of language proficiency.’ Per Linguam, 12 (1), 48-58 (1996a).

^[50]Nunan, D. Syllabus design, 1988, p.28.

^[51]Widdowson, H.G. Explorations in applied linguistics,1979, p.141.

^[52]Ibid, p.246.

^[53]Widdowson, H.G. ‘Communication, community and the problem of appropriate use’, in Alatis, J.E. Georgetown University Round Table on Languages andLinguistics, 1992, p.306.

^[54]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.183.

^[55]Byrnes, H. and Canale, M.(eds.). Defining and developing proficiency: Guidelines, implementations and concepts, 1987, p.1.

^[56]His specific context is the second language “acquisition”/second language “learning” controversy  of Krashen (1981); see Note 56 below..

^[57]Morrissey, M. D.‘Toward a grammar of learner’s errors.’ InternationalReview of Applied Linguistics, 21 (3), 193, 207 (1983), p.200.

^[58]Krashen, S. Second language acquisition and second language learning. (Oxford,  Pergamon Press, 1981).

^[59]Krashen, S. and Terrell, T. The natural approach:Language acquisition in the   classroom, 1983.

^[60]Butzkamm, W.’Review of H. Hammerly, “Fluency and accuracy: Toward balance in               language teaching and learning”‘. System, 20 (4), 545-548 (1992).

^[61]Taylor, B.P. ‘In search of real reality.’ TESOL Quarterly, 16 (1), 29-43 (1982).

^[62]Lantolf and Frawley, 1988, p.183.

^[63]Politzer, R.L. and McGroarty, M. ‘A discrete-point test of communicative competence.’ International Review of Applied Linguistics, 21 (3), 179-191 (1983).

^[64]Entwhistle, W.J. Aspects of language, 1953, p.157.

^[65]Rea, P. Language testing and the communicative language teaching curriculum, 1985.

^[66]Halliday, M.A.K. Learning how to mean, 1975, p.5.

^[67]Ibid, p.2.

^[68]Givon, T.Understanding grammar, 1979, pp.5 and 22

^[69]Ibid, p.31.

^[70]Rutherford, W.E. Second language grammar: Learning and teaching,1987, pp.1-5

^[71]Ibid, pp.4 and 36-37.

^[72]Spolsky, B.Conditions for second language learning, 1989, p.61.

^[73]Alderson, J.C. Reaction to the Morrow paper, 1981c, p.47.

^[74]Besner, N. ‘Process against product: A real opposition?’ English Quarterly, 18 (3), 9-16 (1985), p.9.

^[75]Widdowson, H.G. Explorations in applied linguistics, 1979.

^[76]Farhady, H. The disjunctive fallacy between discrete-point tests and integrative tests,  1983.

^[77](1)Hale, G.A., Stansfield, C.W. and Duran, R.P. TESOL Research Report 16.(Princeton, New Jersey: Educational Testing Service, 1984).

(2) Henning, G.A., Ghawaby, S.M., Saadalla, W.Z., El-Rifai, M.A., Hannallah, R.K. and Mattar, M. S. ‘Comprehensive assessment of language proficiency and achievement among learners of English as a foreign language.’ TESOL Quarterly, 15 (4), 457-466 (1981).

(3) Oller, J.W., Jr. Language tests at school. London,  Longman, 1979).

(4) Oller, J.W. (Jr.) and Perkins, K. (eds.). Language in education: testing the tests. (Rowley, Massachusetts,  Newbury House, 1978).

^[78]Fotos, S. ‘The cloze test as an integrative measure of EFL proficiency: A substitute for essays on college entrance examinations.’ Language Learning, 41 (3), 313-336 (1991), p.318.

^[79]Alderson, J.C. ‘The cloze procedure and proficiency in English as a foreign language.’ TESOL Quarterly, 13, 219-227 (1979).

^[80]Ibid.

^[81]Ibid.

^[82]Ibid.

^[83]Alderson, J.C., Clapham, C. and Wall, D. Language test construction and evaluation. (Cambridge, CUP, 1995).

^[84]Bonheim, H.  Roundtable on language testing. European Society of the Study of   English (ESSE) conference, Debrecen, Hungary, September, 1997).

^[85]Oller, J.W., Jr. A consensus for the 80s, 1983, p.137.

^[86]Rea, P. ‘Language testing and the communicative language teaching curriculum’, in Lee, Y.P. et al. New directions in language testing. (Oxford. Institute of English, 1985), p.22.

^[87]Ibid.

^[88]Canale, M. and Swain, M. ‘Theoretical bases of communicative approaches to second language teaching and testing.’ Applied Linguistics, 1 (1), 1-47 (1980), p.35.

^[89]Ibid.

^[90]Alderson, J.C. ‘The cloze procedure and proficiency in English as a foreign language.’ TESOL Quarterly, 13, 219-227 (1979).

^[91]Stevenson, D.K.Pop validity and performance testing, 1985.

^[92]Spolsky, ibid, p.33-34.

^[93]For example, Hughes, A. Testing for language teachers. (Cambridge, Cambridge University Press, 1989), p.15.

^[94]Rea, P. ‘Language testing and the communicative language teaching curriculum’, in Lee, Y.P. et al. New directions in language testing. (Oxford. Institute of English, 1985).

^[95]Omaggio, A.C. Teaching language in context: Proficiency-orientated instruction. (Boston, Massachusetts,  Henle and Henle, 1986), p.312-313.

^[96]Alderson, J.C. ‘Who needs jam?’, in Hughes, A. and Porter, D. Current   developments in language testing. (London, Academic Press, 1983), p.89.

^[97]Politzer, R.L. and McGroarty, M. ‘A discrete-point test of communicative competence.’ International Review of Applied Linguistics, 21 (3), 179-191 (1983).

^[98]Widdowson, H.G. ‘Skills, abilities, and contexts of reality.’ Annual Review of Applied Linguistics, 18, 323-333 (1998), p.329.

^[99]Spolsky, B. Conditions for second language learning. (Oxford,  Oxford University Press , 1989), p.61.

^[100]Rutherford, W.E. Second language grammar: Learning and teaching, 1987.

^[101]Spolsky, ibid.

^[102]This raises the contentious issue of separating “semantics” from “pragmatics” (See Hudson 1984). From the point of view of the ideational (or conceptualising) function of language, which is what most of language processing is concerned with, or should be concerned with, much more demands are made on semantic and syntactic encoding than the  communicative act itself, which, after all, is only the last stage of language in action – unless one speaks before one thinks (Widdowson 1998:330)

^[103]Cummins, J. ‘The cross-lingual dimensions of language proficiency: Implications for bilingual education and the optimal age issue.’ TESOL Quarterly, 14 (2), 175-87 (1980).

_________Language proficiency and academic achievement, 1983.

_________  Wanted: A theoretical framework for relating language proficiency to academic achievement among bilingual students, 1984.

^[104]Widdowson, H.G. ‘Skills, abilities, and contexts of reality.’ Annual Review of Applied Linguistics, 18, 323-333 (1998), p.326.

^[105]Bernstein, B. Class, codes and control, 1971.

^[106]Wald, B. A sociolinguistic perspective on Cummins’ current framework for relating language proficiency to academic achievement, 1984, p.57

^[107]Ur, P. A course in language teaching: practice and theory, 1996.

^[108]Saville-Troike, M. ‘What really matters in second language learning for academic achievement.’ TESOL Quarterly, 18 (2), 199-219 (1984), p.199.

^[109]Upshur, J.A. ‘English language tests and predictions of academic success’, in Wigglesworth, D.C. (ed.). Selected conference papers of the Association of Teachers of English as a Second Language. Los Altos, California,  National Association for foreign Student Affairs (NAFSA) Studies and Papers, English Language Series 13, 85-93 (1967), p.85.

^[110]Valette, R.L. Modern language testing: A handbook, 1969, p.5.

^[111]Carmines, G. and Zeller, A. Reliability and validity assessment, 1979, 15.

^[112]American Psychological Association.Standards of educational and   psychological measurement. 1974.

^[113]Clark, J.L.D. Theoretical and technical considerations in oral proficiency testing, 1975, p.28.

^[114]Spolsky, B. ‘The limits of authenticity in language testing.’ Language Testing, 2, 31-40 (1985), p.33-34.

^[115]Davies, A. Principles of language testing, 1990), p.44.

^[116]Ibid, p.7.

^[117]Yeld, N. ‘Communicative language testing and validity.’ Journal of Language Teaching, 21 (3), 69-82 (1987), p.78.

^[118]Stevenson, D.K.Pop validity and performance, 1985, p.112.

^[119](1)American Psychological Association. 1974. Standards of educational and   psychological measurement. (Washington D.C,, American Psychological Association, 1974).

(2)Cronbach, L.J. Essentials of psychological testing. (New York: Harper and Row, 1970).

(3) Gardner, R.C.  and Tremblay, P.F. ‘On motivation: measurement and   conceptual considerations.’ The Modern Language Journal, 78 (4), 524-527 (1994).

(4) Stevenson, D.K.’Pop validity and performance testing’, in Lee, Y., Fok, A., Lord, R. and Low, G. (eds.). New directions in language testing. (Oxford, Pergamon, 1985).

^[120]Cronbach, ibid, p.183.

^[121]Gardner and Tremblay, ibid, p.525.

^[122]Messick, S.Constructs and their vicissitudes in educational and psychological measurement, 1989a, p.1.

^[123]Messick, S. Validity, 1989b, p.1-2

^[124]Messick, S. Meaning and values in test validation: The science and ethics of measurement, 1988, p.2.

^[125]Messick, S. Constructs and their vicissitudes in educational and psychological measurement, 1989a, p.1.

^[126]In Item Response Theory (IRT) a major issue is the unidimensionality assumption that all items in a test measure a single ability. If, however, language “competence” is multidimensional (see Bachman 1990a), the assumption that all items hang together might not be correct. From the psychometric point of view, though, the unidimentionality assumption does not, as Henning (1992) argues, preclude the psychological basis of multidimensionality (see Douglas 1995).

^[127]Of course, there are many second language users who have a far better command of academic discourse than mother-tongue  users. This is so because the ability to understand and produce academic discourse depends on much more than “linguistic ability”: it also depends on CALP and academic intelligence (Gamaroff 1995c, 1996b, 1997b)

^[128]Ebel, R.L. ‘Must all tests be valid?’ American Psychologist, 16, 640-647 (1961), p.645.

^[129]Loevinger, J. Objective tests as instruments of psychological theory, 1967, p.93.

^[130]Bachman, L.F. Fundamental considerations in language testing, 1990b, p.253.

^[131]Ibid.

^[132]Messick, S. Constructs and their vicissitudes in educational and psychological measurement. (Princeton, New Jersey, Educational Testing Service, 1989a).

^[133]Cronbach, L.J. Essentials of psychological testing, 1970, p.122.

^[134]Weir, C.J. Communicative language testing, 1988, p.30

^[135]Lado, R. Language testing, 1961, p.324.

^[136]Davies, A. Principles of language testing, 1990, p.3.

^[137]This problem is indicative of the much larger problem of the indeterminacy of language (and hence also of epistemology) itself. “There is no perfect hypothetical language, to which the languages we have are clumsy approximations” (Harris 1981:175). And this must inevitably lead the applied linguist to grapple with the slippery notions of native speaker and mother-tongue speaker. I discuss this issue further in section 6.1.1.

^[138]Spolsky, B. Measured words, 1995, p.358.

^[139]Ibid.

^[140]Yeld, N. Communicative language testing. Report on British Council Course 559 offered at Lancaster University from 8 September to 20 September 1985. (Cape Town. University of Cape Town, 1986, p.31.

^[141]According to Perkins (1983:655),“raters, guided by…holistic scoring guides…, can achieve a scoring reliability as high as .90 for individual writers.” Indirect objective tests such as multiple choice grammar and vocabulary tests, on the other hand, can have reliability coefficients as high as .99, because there is no problem of rater reliability involved, i.e. subjective judgements will not affect the scores (Hughes 1989:29).

^[142]Bachman, L.F. Fundamental considerations in language testing. (Oxford: Oxford University Press, 1990b, pp. 116ff, 168-172, 244.

^[143]Ebel, R.L. and Frisbie, D.A. Essentials of educational measurement, 1991, p.76.

^[144]Sammonds, P. Ethical issues and statistical work, 1989, p.53.

^[145]Pinker, S. The language instinct, 1995, p.401.

[Home] [Services & Rates] [Contact] [Ph.D.] [Articles and Conference Papers]

// <!–[CDATA[-->
_doc=document; _nav=navigator; _timezone=new Date(); function count() { _doc.write(
"");} _color="0"; _rez="0"; _java="U"; _kr="U"; _doc.cookie="_kr=y";
_doc.cookie.length>0?_kr="Y":_kr="N";
// ]]> // //

Click on the graphic to vote for this
page as a pMonkey.com Hot Site.

1

1.1  Introduction: The  problem and purpose of the study

1.2  Psychometrics and norm-referenced testing

1.3  Summative assessment

1.4  The One Best Test

1.5  Hypotheses of the study

1.6  Historical and educational context

1.7  Measures used in the study

1.8  Method overview

1.9  Preview of Chapters 2 to 6

1.10 Summary of Chapter 1

1.1 Introduction: The  problem and purpose of the study

Language testing draws on three areas: (1) the nature of language, (2) assessment and (3) language ability.^[1] Language ability is closely related to language proficiency, which is a key term in this study. (The relationship between ability and proficiency is explained in section 2.2 ff).

Central concepts in the measurement of language ability are: (1) validity (what one is measuring), (2) reliability (how one is measuring), (3) practicability (economics of time and expense) and (4) accountability (why one is testing). If a test is not practicable, even if  judged to be valid and reliable, it would be uneconomical and accordingly of little use.

Owing to our ignorance of the processes of language learning and learning processes in general, much of what we know about language testing, and therefore also about teaching, remain tentative. (“Language testing is rightly central to language teaching”^[2]). A major obstacle in test development has been the lack of agreement on what it means to know a language, on what aspects of language knowledge should be tested – and taught – and how they should be tested and assessed. Accordingly, it is not always easy to know why one is testing. As far as the how and the what is concerned, the more explicit, i.e. the more accurate, or reliable,  one tries to make a test, the more uncertain one becomes about (the validity of) what one is testing.

The study consists of  three parts:

Part I. A large part of this study is devoted to statistical measurement in assessment, specifically the relationship between communicative competence and traditional testing theories insofar as these shed light on the (1) construct of language proficiency, and (2) the use of proficiency tests as predictors of academic achievement. Statistical measurement in scoring procedures are also closely related to the structure and administration of tests.  This study re-examines and defends this interdependence in terms of the three key notions in testing: validity, reliability and practicality.

Part II. The prediction of academic achievement where English proficiency tests are used as the predictors. A longitudinal study is undertaken of the prediction of academic achievement from Grade 7 to Grade 12.

Part III. Implications of the study for procedures of assessment of language proficiency.

The educational context of this study is a High School in the North West Province, which will be referred to as MHS, where I taught and did language research for over seven years (January 1980 to April 1987).

The primary focus of the study is not on what “really matters  in first or second language proficiency and academic achievement^[3], or, put another way, on “develop[ing] an adequate theoretical framework for relating language proficiency to academic achievement”.^[4] Neither does it deal with individual differences in cognitive styles of learning.^[5] Nor does it dwell on the many causes of academic failure. The longitudinal part of the study is much more about predictions than about causes of academic failure. I do, however, refer to causes of academic failure where relevant. For example, one of the causes of academic failure (and success!) that this study is particularly interested in is the lack of scoring consistency among raters, which is arguably the greatest problem in assessment.

There are two urgent needs in minority education:

(1) to pursue fundamental research on the nature of language proficiency and how it can be measured, and (2) to provide teachers with up-to-date knowledge of language proficiency assessment so they can improve their classroom assessment practices.^[6]

The term minority has much more than a numerical meaning. In South Africa the majority of learners use English as an additional/secondlanguage, but the tradition is to refer to such learners as originating from minority language backgrounds. The term has an obvious discriminatory ring that implies that some acceptable level has not yet been reached. Yet, tests have to distinguish between levels of proficiency for them to have construct validity. Chapters 3, 4 and 5 deal with the statistical issues of assigning people to the same or different groups, and Chapter 6 deals with the educational and political implications of doing so. Two main levels of proficiency are examined, which are given the well-known labels of L1 and L2. The L1 and L2 labels are central to the study and are used differently to the normal connotation of  users of a first language and learners of a second language, which I shall explain shortly. These well-known terms together with the terms mother tongue and native language have been the occasion of much controversy. I discuss this controversy in section 6.2. In the empirical investigation the labels L1 and L2 will refer to the sample of subjects (i.e. informants) who take the subject English as a First Language and English as a Second Language, respectively, at MHS. This definition of L1 and L2 needs constant reminding throughout the study. In Chapter 6 (section 6.2), various other definitions of “L1″ and “L2″ are examined.

The sample of subjects is described in detail in Chapter 3. Tables 3.1 and 3.2 give a clear explanation of the sample of subjects and so it might be useful to continually refer to these two figures, which serve as guideposts to the description of the sample.

Psychometric questions, and discrete-point and integrative testing have been discussed at length in the literature on language testing and assessment (the distinction between testing and assessment is explained shortly). So one may ask why the need to, and where is the merit and originality of, devoting a PhD to such old and outdated  issues, which has not been a research issue in language assessment for over 15 years? Thereis indeed a pressing need because although communicative methods have become the prevalent source of testing with a “richer conceptual base for characterizing the language abilities to be measured, it has presented language testers with a major challenge in defining these abilities and the interactions among them with sufficient precision to permit their measurement.”^[7] It is this problem of reconciling authentic subjectivity and objective precision that is the major problem in testing, indeed the major problem of cognition and language.^[8] The authenticity issue has wider ramifications. It is not only central to testing but also to syllabus design and materials development. This study focuses on testing only.

In spite of decades of attempts to define it, the how^[9] and the why^[10] of language proficiency remains a conundrum. Although we may no longer stand before an “abyss of ignorance”^[11] and may be able to agree with Alderson (in Douglas^[12]) that language testing has “come of age”, there are still many problems in language testing, the greatest one being, I suggest, the problem of reliability^[13] and specifically rater reliability (see Alderson and Clapham’s case studies of this problem^[14]). There are two kinds of rater reliability: interrater reliability and intrarater reliability. These are dealt with in section 2.9.1.

Owing to our ignorance of the processes of language learning and learning processes in general, much of what we know about language testing, and therefore also about teaching, remain tentative. (“Language teaching is rightly central to language teaching”^[15]). A major obstacle in test development has been the lack of agreement on what it means to know a language, on what aspects of language knowledge should be tested – and taught – and how they should be tested and assessed.

This problem is not a surprising one because language is closely connected to human rationalities, imaginations, motivations and desires, which comprise an extremely complex network of biological, cognitive, cultural and educational factors. As a result, all language testing theories are inadequate owing to the difficulties involved in devising tests that test authentic language reception and production. This does not mean that we should stop measuring until we’ve decided what we are measuring. We do the best we can by taking account of generally accepted views of the nature of language proficiency , of modern views and dated ones. In the modern literature on testing there seems to be an overemphasis on up-to-date theories, which gives the impression that “what is  dated is outdated”.^[16] Widdowson’s up-to-date admonition that we should take more seriously dated views is taken to heart in this study.

What is a test? It is “the most explicit form of description, on the basis of which the tester comes clean about his/her ideas”.^[17] What all testers are looking for are systematic elicitation techniques on which one can base useful decisions. The three underlying issues in testing are: to infer abilities, to predict performance and to generalise from context to context^[18].  This means that tests should be valid, reliable and practicable. Communicative testers would add the notions of “impact” (i.e. face validity) and “interactionist”.^[19] Opponents of discrete-point tests (such as grammar tests) and integrative tests (such as cloze tests and dictation tests) would probably concede that such tests are reliable and practical, but they would argue that they are not valid, i.e. they tell us little or nothing about the learner’s knowledge of authentic language. I shall argue that, on the contrary, they indicate a great deal about authentic language and that these old issues are not outdated and are still worthy of attention.

I use an analysis of a battery of English proficiency tests to substantiate my theoretical position. I then examine the predictive validity of the battery of English proficiency tests. Part of the predictive investigation involves a comparison between the predictive validity of the school reports from former schools of entrants to the School with the predictive validity of the English proficiency tests. These reports were the main criterion for admission to the School. Few entrants with School report aggregates under 60% were admitted.

A curriculum framework consists of the following components^[20]:

-  Needs analysis

-  Objectives

-  Materials

-  Teaching

-  Testing

Accordingly, the curriculum is concerned with the syllabus as well as everything to do with pedagogical matters, i.e. teaching what to whom, when and how.^[21]Syllabus is defined  as the content and sequence of content of the programme selected in order to make learning and teaching effective.^[22] Although testing is the last component in the curriculum framework, this is only so chronologically, and not logically, because testing permeates the whole of the curriculum. This is the reason why there is the possibility  – and the temptation; perhaps justifiably so -  of teaching to the test.

A major part of testing is concerned with assessment.  In this study I use the term tests to refer to “elicitation techniques”^[23] and the term assessment to  refer to the methods used to measure or analyse test results. In assessment one is concerned with the control of rater judgements and  scoring techniques. (Assessment is discussed in detail in section 1.3). Thus assessment, quantitative or qualitative, is not an intrinsic property of a test but a method one uses to measure or analysis the test results.

There are a variety of language test uses (Pollitt in Yeld^[24]). The basic four uses are mentioned^[25]:

-  Proficiency tests, which evaluate present knowledge in order to predict future achievement, usually at the beginning of a course of study. Proficiency tests are based on knowledge that has been gained independent of any specific syllabus but not independent of typical syllabuses because the knowledge to be tested must have been gained from some syllabus or other.

- Achievement tests, which evaluate how much has been learnt of a particular syllabus, where the focus is on success, usually at the end of a teaching programme.

- Diagnostic tests, which evaluate points not yet mastered, where the focus is on failure and therapy. Diagnostic tests. therefore may be considered to be the reverse of achievement tests.^[26] Proficiency tests often involve diagnosing items that have not been mastered, and therefore diagnostic testing may be  part of proficiency testing.

- Aptitude tests, which evaluate abilities for language mastery, and is thus, like proficiency tests, of predictive value. Unlike the three other kinds of test uses, aptitude tests have no specific or general content, and are thus difficult tests to compile. They require, arguably, the most knowledge and care in their construction and application, for it is far worse to be told that one has no aptitude than to be told that one has low proficiency or has failed an achievement test. No aptitude means no hope at all, unless it is possible to have potential without aptitude.

Thus, both proficiency and achievement are concerned with present knowledge. It may be that a proficiency test contains material previously contained in an achievement test, but this difference is irrelevant to the validity of the proficiency test because, unlike an achievement test, a proficiency test is not concerned with whether the content of a test was previously taught. The American Council of the Teaching of Foreign Languages (ACTFL) Proficiency Guidelines are a case in point:

Because these guidelines identify stages of proficiency, as opposed to achievement, they are not intended to measure what an individual has achieved through specific classroom instruction but rather to allow assessment of what an individual can and cannot do, regardless of where, when, or how the language  has been learned or acquired: thus the words learned and acquired are used in the broadest sense.^[27]

Proficiency is concerned with what somebody knows and can do here and now. Achievement should be ultimately concerned  with proficiency as well. That is why Spolsky^[28] omits the term achievement in the following definition of language tests: “Language tests involve measuring a subject’s knowledge of, and proficiency in, the use of language.”

There are four important considerations in language testing^[29]:

1.  How valid is the test?

2. How easy is it to compose?

3. How easy is it to administer?

4. How easy is it to mark?

The first, which is concerned with the purpose of a test is the most important theoretical issue in testing.  Ur feels so strongly about practicability that her next three considerations for choosing a test have to do with practicability. The fourth is also related to rater reliability. A test may be everything communicative testers require, but it would still be no good if it took too long or was too difficult to administer and assess. The more objective the test, the less the danger of rater unreliability. An essay test is a supreme example of a subjective test, because it is vulnerable to fluctuations in judgements between raters. The problem is finding the appropriate balance between the different testing considerations, where it is difficult, indeed, impossible, to give all of them equal prominence.

The reason why the value of discrete-point tests and many integrative tests such as cloze tests and dictation tests must be reassessed is mainly because of their practicality; if only one could solve the problem of  authenticity of  these tests.  The basic problem is whether indirect tests such grammar tests, cloze tests and dictation tests can predict real-life performance, which many authors (these authors are discussed at length in the study) equate with authentic language, and thus reject the notion that an indirect elicitation procedure of real-life language can be authentic. Of course, this problem of authenticity of indirect tests is not limited to language tests but to all kinds of indirect tests, e.g. intelligence tests. A major part of this study is concerned with the meaning of authenticity in testing.

Every test is an operationalisation about certain beliefs and values about language, whether the test is called authentic or not . These beliefs and values determine to a certain extent our mental and emotional reactions to language and to knowledge in general. What is required in this study is to justify the beliefs I hold about discrete-point tests, integrative tests and the necessary psychometric methods of assessment that they imply.

1.2 Psychometrics and norm-referenced testing

In language testing the opposition to psychometrics is closely connected to the “suspicion of quantitative methods”^[30] and the opposition to “reductionist approaches to communicative competence”.^[31]

The history of the quantitative/qualitative controversy can be viewed from two diametrically opposite angles: (1) qualitative research has been dominated by quantitative research for many decades and is only in recent years becoming accepted as a legitimate scientific approach^[32] or (2) quantitative research has been for more than two decades challenging qualitative methods and also setting itself up as the only legitimate form of research.^[33]

Galton’s view is that it is the scientists’ job to “devise tests by which the value of beliefs may be ascertained, and to feel sufficiently masters of themselves to discard contemptuously whatever may be found untrue”^[34] (Rushton 1995; his frontispiece). For Galton tests must be statistically validated. Although I do not share Galton’s sweeping faith in statistics, statistical measurement in language testing has been given a undeserved bad press, e.g. Spolsky^[35], Lantolf and Frawley^[36] and  Macdonald^[37].

The increasing number of studies in  purely ethnographical/sociolinguistic approaches to language proficiency assessment^[38] is witness to the opposition  to the objectivist, or positivistic, or reductionist methods of psychometric research. For Harrison, psychometric measurement is inappropriate due to its subjective nature:

Testing is traditionally associated with exactitude, but it is not an exact science…The quantities resulting from test-taking look like exact figures – 69 per cent looks different from 68 pert cent but cannot be so for practical purposes, though test writers may imply that they are distinguishable by working out tables of  precise equivalences of test and level, and teachers may believe them. These interpretations of scores are inappropriate even for traditional testing but for communicative testing they are completely irrelevant. The outcome of a communicative test is a series of achievements, not a score denoting an abstract `level’.^[39]

Thus, “the quantities resulting from test-taking [which] look like exact figures” (in the quotation above) appear to measure objectively, but in fact they measure subjectively. (See Morrow^[40] for a similar view). Lantolf and Frawley^[41] maintain that

• [w]hat must be done is to set aside the test-based approach to proficiency and to begin to develop a theory of proficiency that is independent of the psychometrics. Only after such a theory has been developed and is proven to be consistent and exhaustive by empirical research should we reintroduce the psychometric factor into the picture, with the full realization that such a reintroduction may not be possible, given our earlier remarks on the scalability of human behavior.

Spolsky was contemptuous of “psychometrists”:

• In the approach of scientific modern tests, the criterion of authenticity of task is generally submerged by the greater attention given to psychometric criteria of validity and reliability. The psychometrists^[42] are ‘hocus-pocus’ scientists in the fullest sense; in their arguments, they sometimes even claim  not to care what they measure provided that their measurement predicts the criterion variable: face validity receives no more than lip service.^[43]

Spolsky’s recent “postmodern” approach to psychometrics is that it should be used in conjunction with “humanist” approaches.^[44] The view in this study, which, for some researchers is taken for granted but for others is highly contested, is that “language testing cannot be done without adequate statistics”.^[45]

Psychometric assessmentin language assessment traditionally means norm-referenced assessment, which is not concerned with individual scores but with the dispersion of scores within a group, where the concern is with maximising individual differences between test takers on the variable that is being measured.^[46] For many, psychometrics has become synonymous with quantitative measurement and statistical measurement^[47], and this is how I use psychometrics in this study.

The perennial problem in language testing, indeed of all testing, is finding a balance between reliability and validity. Spolsky’s coupling of validity with psychometrics may create the impression that authenticity and validity are separate issues. Of course, there is much more to validity than validity coefficients. The major issue in validity has to do with specifying what authentic tests are. Thus, psychometric data only become digestible  – and palatable – when theory gives meat to the number-crunching.

The  term psychometric tests has another meaning.. For example, at a conference on academic development, where I presented a paper on this topic^[48] , a member of the audience said that she was “boiling” because psychometrics, she insisted was far more than norm- referenced measurement”. Her view was that psychometric tests measured the psyche (the literal meaning of psychometrics) and was embedded, she insisted (correctly), in the flesh-and-blood context of individuals, and therefore psychometric tests involve far more than a comparison between individuals and groups. I was taken aback by her outburst, not because she did not have a valid definition of psychometrics, but because this definition of psychometrics, because the of the different context, had never entered my mind, owing, no doubt, to my maximum attention to what interested me.

The method used in this study is mainly quantitatively based, where the emphasis is on norm-referenced testing. In language testing, as in second language acquisition research in general, quantitative measurement has been challenged for more than two decades by qualitative methods of research: indeed, qualitative measurement has been setting itself up as the only legitimate form of research.^[49] Terre Blanche distinguishes between “two different constituencies” of qualitative researchers:

those who would use qualitative methods as a humanist, emancipatory tool to access authentic subjective experiences so easily censored out by more hard-nosed quantitative methods, and those who want to use qualitative methods such as discourse analysis to critique the semantic practices of both ‘scientific’ and ‘humanist’ psychologies.^[50]

Both constituencies reject the domination of the norm over the individual.

“Norm” has two meanings, which are sometimes not distinguished. For example, at the conference of the National Educators of Teachers of English (NAETE) at Potchefstroom (September 17-18 1998) I was discussing the concept of “norm” with Johan van der Walt of Potchefstroom University of Christian Education. We were in one accord that the individual without the norm is an abstraction. It is was only while Van der Walt was presenting his paper that I realised that we were using the same term for two distinct concepts.

In this regard, consider the following extract from a  repartee between Johan van der Walt and Colyn Davey at the 1998 National Association of Educators of Teachers of English (NAETE) conference^[51] that was concerned with the topic of establishing norms of English.

Van der Walt.  Then you agree that there should be a norm.

Davey. Yes but learners should be able to choose the norm they prefer.

The question arises of whether it is possible to use the term “norm” in the sense of (1). Van der Walt’s imperative of conforming to a standard, or norm, by which he means “Standard” English, and 2. Davey’s imperative of freedom to choose the norm that one refers, which could be “Standard” English, or, say, institutionalised black South African English (IBSAE), which would comprise ubiquitous constructions such as “I am having a problem”, “He write English perfectly” and “When I was in Town I see my English teacher”.^[52] It is indeed possible to use the term “norm” in these two senses, but if done so, the two meanings of “norm” should be clearly distinguished, which wasn’t the case in the discussion between Van der Walt and Davey.

To clarify the distinction between these two meanings it is necessary to introduce the notions of criterion-referenced and norm-referenced tests Criterion-referenced tests are concerned with how well an individual performs relative to a fixed criterion, e.g. how to ask questions. Norm-referenced tests are concerned with how well an individual performs compared to a group. This is traditional psychometric testing. Now, both Van der Walt and Davey believe in norms; the former a standardised norm, the latter an unstandardised norm. So,, in fact, both kinds of norm are concerned with how well an individual performs relative to a fixed criterion, which is what criterion-referenced tests are concerned with. The difference is that Van d Walt’s norm is imposed from above (the standardised norm), whereas Davey’s norm is bottom up, where the group chooses the norm it wishes to aspire to. Both these kinds of norm have got nothing to do with norm-referenced tests, which has got nothing to do with whether one is able to choose ones norm (Davey) or not (Van der Walt), but with, as defined above, how well an individual performs in a group, i.e. with the difference in ability revealed in some form “quantitative summary”.^[53]

Norm-referenced,which is a key notion in this study, can be distinguished from criterion-referenced and individual-referenced tests:

1. Norm-referenced tests are concerned with how well an individual performs compared to a group which he or she is a member of. This is traditional psychometric testing.

2. Criterion-referenced tests are concerned with how well an individual performs relative to a fixed criterion, e.g. how to ask questions. This is what Cziko calls “edumetric” testing.^[54]

3. Individual-referenced are concerned with how individuals perform relative to their previous performance or to an estimate of their ability.

Strictly speaking it is not the test that is norm-referenced or criterion- referenced or individual-referenced, but the purpose for which it is used. Similarly, tests in themselves are not valid, but rather it is the purpose that they are used for that makes them valid. (Validity is discussed in section 2.8).

The idea that norm-referenced tests, on the one hand, and criterion-referenced tests and individual-referenced tests, on the other, are mutually exclusive is based on two contrasting philosophical positions: the former “positivistic”, the latter “humanistic”. The former is interested in what makes people different, the latter finds it morally reprehensible to compare people and thus focuses on what they have in common. The latter view, is represented most vocally in South Africa by the protagonists of ”outcomes-based education” where ” learner’s progress will be measured against criteria that indicate attainment of learning outcomes, rather than against other learners’ performances”.^[55]

I shall argue that norm-referenced tests are indispensable. Norm- referenced tests are important because without data on the variance between individuals within a group, it is not possible to separate what (which is the concern of criterion-referenced tests) an individualknows from what other peopleknow.

Individual-referenced tests also cannot be separated from what other people know. The differences between individuals actually clarifies the matter under test.  In other words, the construct validity of a test is dependent on some people doing well and others doing less well, for if everybody did equally well, we would have little idea of what we were testing. That is the norm-referenced view, which is based on the notion that differences in nature, of which human abilities are a part, is distributed as a bell-curve. In what can be regarded as a key quotation in the defence of psychometrics in this study, Rowntree explains the importance of group statistics, i.e. norm-referenced tests, in assessment:

Consider a test whose results we are to interpret by comparison with criteria. To do so we must already have decided on a standard of performance and we will regard students who attain it as being significantly different from those who do not…The question is: How do we establish the criterion level? What is to count as the standard? Naturally, we can’t wait to see how students actually do and base our criterion on the average performance of the present group: this would be to go over into blatant norm-referencing. So suppose we base our criterion on what seems reasonable  in the light of past experience? Naturally, if the criterion is to be reasonable, this experience must be of similar groups of students in the  past. Knowing what has been achieved in the past will help us avoid setting the criteria inordinately high or low. But isn’t this very close to norm-referencing? It would even be closer if we were to base the criterion not just on that of previous students but on students in general.^[56]

What we think is going on in each individual’s invisible mind, can be scientifically inferred and described only when one has some idea of what is going on in a many individual minds, i.e. what is going on in a group. Emphasising the individual over the group or vice versa is “somewhat metaphysical [because both] types of test sampling (for that is what norm and criterion referencing do: they sample) need one another”.^[57]Recall the two meanings of “norm” discussed earlier:“norm” can refer to a criterion (!) such as Standard English or to the comparison between individuals within a group.The latter is the concern of norm-referenced tests.

Ranking individuals and generating scores is only one purpose of norm-referenced tests; another purpose is to gain understanding into the nature of the constructs under examination, which cannot be achieved if an individual is not compared with what other individuals do.  (I elaborate on the  complementary role of norm-referenced and criterion-referenced tests in the discussion of the inadequacies of correlational data in section 4.7).

1.3  Summative assessment

There are different kinds of assessment and a diversity of definitions. I  discuss the descriptions of Rea^[58] and Rowntree.^[59]

TABLE  1.1

Rea’s Schema of Assessment

Formative Assessment     Summative Assessment
Quantitative Methods Assessment                                  Evaluation
Qualitative Methods Appraisal

Rea^[60] uses the term “evaluation” to refer to formal testing activities, which are external to the teaching situation, and which involve “test scores”. She uses the terms “assessment” and “appraisal” to refer to activities which are internal to the teaching program. Grades are given for assessment but not for appraisal. In Rea’s scheme, assessment and evaluation both use  measurement, i.e. quantitative methods, while appraisal does not.

Before I comment further on Rea, it is appropriate to say something about “evaluation”, which are terms used by Rea and Rowntree. There are many different definitions of evaluation. Bachman^[61] defines evaluation as the “systematic gathering of information for the purpose of making decisions” while Brown defines evaluation as

the systematic collection and analysis of all relevant information necessary to promote the improvement of a curriculum, and assess its effectiveness and efficiency, as well as the participants’ attitudes within the context of the particular institutions involved.^[62]

Recently, “evaluation” has been contrasted with “grading”.^[63] Dreyer argues that grading, i.e. summative tests, cause people to fail (see the conclusion to the study for comment; section 6.7).

To return to Rea. What may be confusing in her schema is that ”assessment”  is used generically  to cover everything to with testing as well as specifically to refer to “formative quantitative assessment”. Consider Rowntree’s schema: 

TABLE 1.2

Rowntree’s Schema of Assessment and Evaluation

Rowntree’s “assessment” -  which he also uses in this generic way – is “put[ting] a value on something” , which translates into everything concerned with  “obtaining and interpreting information”  of any kind about another person in order to “[try] and discover what the student is becoming or has accomplished”.^[64] Rowntree’s “formative (pedagogic) assessment” emphasises “potential”, while his “summative (classificatory) assessment” emphasises “actual achievement”.^[65]

For Rowntree, “evaluation” is “an attempt to identify and explain the effects (and effectiveness) of the teaching.”^[66] Rowntree’s “formative evaluation is intended to develop and improve a piece of teaching until it is as effective as it possibly can be…[s]ummative evaluation on the other hand, is intended to establish the effectiveness of the teaching once it is fully developed.”^[67] Thus Rowntree’s “formative evaluation” is concerned with the washback effect of a syllabus and/or teaching programme, while “summative evaluation” is concerned with “terminal tests and examinations coming at the end of the student’s course, or indeed by any attempt to reach an overall description or judgement of the student (e.g. in an end-of-term report or a grade or class- rank”.^[68]

For both Rea and Rowntree quantitative methods always involve summative assessment only. In Rea quantitative methods are also used in formative assessment, while in Rowntree quantitative methods are only used in summative assessment.. In this study, I use Rowntree’s meaning of summative assessment to involve (1) quantitative methods of assessing (2) the learner only (and not the teacher). I am concerned with quantitative methods in summative assessment. An important point: although I am not concerned with assessing the teacher, I am very much concerned with how a teacher (i.e. a rater) assesses a learner; in other words with rater reliability, which consists of two major kinds of judgements: (1) the order of priority for individual raters of performance criteria (criteria such as grammatical accuracy, appropriateness of vocabulary and factual relevance) and (2) the agreement between raters on the scores that should be awarded if or when agreement is reached on how to weight different criteria .^[69] Rater reliability is discussed in sections 2.9.1, 4.2, 4.7 and 5.5.

Summative assessment is “terminal”, “formal” and “external”, and is only concerned with the beginning and end of a course, i.e. with classifying individuals in terms of numerical products, or scores.^[70] Weir^[71] believes that there is a more pressing need for research in formative testing as opposed to research in summative testing. There is still a pressing need for research in summative testing, even though mainstream language testing, in South Africa, at least, is taking a different turn, e.g. “outcomes-based education”. This is discussed in section 5.5.

1.4 The One Best Test

A major empirical problem in language testing is establishing  valid and reliable criteria for the assessment of  language proficiency, which is basically concerned with fluency and accuracy. Three important issues in language testing are:

1. The kinds of tests that should be used to assess levels of language proficiency.

2.  The relationship between statistical significance (numerical data) and their meaning (information).

3. Whether language proficiency tests can validly predict academic achievement.^[72] Academic achievement in this study is represented by 1.  end-of-year aggregate scores and 2. Pass rate.

The above three issues are directly related to the search for the “best” test. In the 70s a major issue in language testing was whether it was possible to find the “One Best Test”. The “One Best Test” question is closely related to the question of whether language proficiency consists of a unitary factor analogous to a g  factor in intelligence, or of a number of independent factors. Bachman and Palmer relate the concerns they had 25 years ago:

[W]e shared a common concern: to develop the “best” test for our situations. We believed that there was a model language test and a set of straightforward procedures – a recipe, if you will – that we could follow to create a test that would be the best one for our purposes and situations.^[73]

Yet, almost two decades ago, Alderson had already graduated from this kind of thinking and suggested that

regardless of the correlations, and quite apart from any consideration of the lack of face validity of the One Best Test, we must give testees a fair chance by giving them a variety of language tests, simply because one might be wrong: there might be no Best Test, or it might not have the one we chose to give, or there might not be one general proficiency factor, there may be several.^[74]

The Unitary Competence Hypothesis (UCH), which is closely associated with the “best test” question, is a very important issue. High correlations between different kinds of tests show that the UCH, in its weak form, remains a force to be dealt with.^[75] The weak form of the UCH adopts an interactionist approach between global and discrete components of language. Oller, in his famous paragraph, describes this approach:

[N]ot only is some sort of global factor dependent for its existence on the  differentiated components which comprise it, but in their turn, the components are meaningfully differentiated only in relation to the larger purpose(s) to which all of them in some integrated (integrative?) fashion contribute. (See  also Oller and Khan^[76] and Carroll^[77] for similar views).^[78]

If one is no longer searching for that one Grand Unified Test (GUT), one should be still looking for good tests, indeed for the best tests available. This implies, I suggest, that what one is looking for has an “objective” reality, which, of course, does not mean that we can completely grasp it.^[79]

If we have given up on finding or constructing that elusive (and illusory?) one best test, we are nevertheless looking, indeed are compelled to look, for a plurality of the best tests that we can find. The problem remains what tests to choose to test language proficiency, and ultimately, in this study, to predict academic achievement. A useful test is one that “correspond[s] in demonstrable ways to language in non-test situations.”^[80] These non-test situations are described in the “new” paradigm of language testing as authentic, direct, real-life,  natural(istic) or communicative. An important part of this study consists of a critical analysis of these terms.

1.5 Hypotheses of the study

The following threenull hypotheses are investigated:

1. Discrete-point tests and/or integrative tests are not valid measures of levels of language  proficiency.

2. Discrete-point tests and/or integrative tests are not valid predictors of academic achievement.

3. Many of the reports (Grade 6)  from former schools that were used as criteria for admission to MHS were not valid predictors of academic achievement. Many of the entrants with high Grade 6 report scores did not get beyond Grade 9 at MHS. I investigate the question of indiscriminate advancement in DET^[81] (Department of Education and Training) schools in the light of the consistently poor Grade 12 results (“matric”) from most former DET schools over the years. The predictive validity of the tests are examined in an attempt to shed clarity on this question. (I shall henceforth refer to “DET schools” and not “former DET schools”, because at the time the investigation in this study was conducted, the DET was still in existence). A major issue in this study is the relationship between the predictive validity of  the tests and the reliability (i.e. the consistency and accuracy) of these DET reports (section 5.5).

Although the study is not directly concerned with investigating mother-tongue^[82] proficiency, it is referred to when required.

Most of the tests in this study belong to the “old paradigm”. I did not devise new tests because  it was  not  ermane to  the  objective  of  this investigation, which was to examine the validity and reliability and practicality of using traditional tests to predict academic achievement. the fact that most of these tests were already established tests meant that I had more time to devote to this objective.

Although it is possible that annual predictions between English proficiency and academic achievement would yield higher correlations than long-term predictions, the aim in this study is to try and find out what chance Grade 7 learners who entered the School in 1987 would have of  passing Grade 12.

1.6 Historical and educational context

MHS  has already had 19 years experience dealing with linguistic, cultural and educational problems, which are only now beginning to surface in many schools in South Africa.  English is used as the single medium of instruction at the School. The School offered the Joint Matriculation Board (JMB) syllabus up to 1992, and the Independent Examinations Board (IEB) syllabus after 1992. MHS was the only state school in a wide area containing hundreds of DET secondary schools that offered the JMB syllabus. This study shows how DET pupils coped at such a school.

One problem that the School has been dealing with since its inception in 1980 is how to reconcile affirmative action with academic merit. By affirmative action I mean the endeavour to put right the  imbalances of the past, where the majority of South Africans was discriminated against on the basis of race. The School’s policy is to provide education for advantaged as well as disadvantaged learners, where the latter are given the opportunity to learn in an advantaged school situation. Disadvantaged learners are those who have suffered educational, social and economic deprivation – often caused by political injustice – and this was what the School also meant by the term. It is also, paradoxically, the School’s policy to accept learners only on merit, which was indicated by high scores on former school reports. One problem with affirmative action is that it is often difficult to marry the idea of redress and the idea of academic merit (high achievement, in this case), potential or aptitude.

This difficulty was evidenced by the School’s Prospectus of 1986, which informed parents that their children “are admitted solely on the basis of merit”; by “merit” the School meant  high scores on reports from previous schools. Candidates are considered on the basis of the results of an entrance examination and their previous school achievement.” Thus, the  School’s intention was to select only those candidates who could cope with a JMB equivalent syllabus. Unfortunately, many learners dropped out along the way or were pushed out along the way by the system.

MHS’s policy was to use admission criteria. The tests in this study, although conducted after admission – during the first three days of the first school term, were partly concerned with the admission question because I wanted to find out whether those who were admitted on the basis of their former school reports should have been admitted. Owing to the recent abolition of  admission tests in South African state schools there would no longer be any point in trying to find the best admission tests. But it would certainly still be useful to find out (1) whether  those learners who had been admitted to the School had an adequate level of English proficiency to perform in a school where English was the medium of  instruction and (2) whether their former school reports were authentic, i.e. accurate, reflections of this level. As far as I am aware, former school reports , as is the general practice in all schools in South Africa, are still considered by the School as an important indication of an entrant’s ability – if not a criterion for admission.

It would seem that the School’s criteria for admission would generally have pinpointed those candidates who could not cope at the School, but this did not happen. Of concern at the School was the large number of failures in Grades 7, 8 and 9 among the DET learners. At the School there were no automatic internal promotions through the system as is claimed to occur in many DET schools.^[83] (This issue is dealt with in Chapter 7). When low achievers at the School failed they often left without repeating a year. Many who failed at the School, whether they repeated a year or eventually were asked to leave owing to failure, did not manage to get beyond Grade 9. Table 1.3 shows the Grade 9 pass rate for  three consecutive years.

TABLE 1.3

Grade 9  Pass Rate

Row 3 is the sample used in the prediction of academic achievement in this study. It excludes learners who passed a  Grade and then left the School before reaching Grade 9 (N=5), and includes learners who failed between Grades 7 and 9 but passed Grade 9 at a later stage (N=9). Samples 1 and 2 do not take this fact into account.

Many learners who reached Grade 12 managed to achieve a matriculation exemption at MHS but obtained disappointing symbols, e.g. D and E symbols. Mcintyre remarks in this regard:

Observations at [MHS] over the last few years would tend to support Young’s^[84] statement [about the low "language competence" of  Grade 12 learners]. However, at [MHS] the problem is slightly different in the sense that Matric students pass successfully but often with symbols that are disappointing. ^[85]

Mcintyre’s statement that “Matric students pass successfully” requires comment: Although it is correct that there was a high Grade 12 pass rate at the School, this does not take into account the high failure rate between Grade 7 and Grade 9, which means that even though most Grade 12 learners passed, this does not imply that many others that started in Grade 7 didn’t drop out along the way. This high failure rate is what had been occurring at the School since its inception in 1980. (I am concerned with the period 1980 to 1993). Table 1.6 shows  the number of  Grade 12 passes (1992) that originated from the group of Grade 7 learners (1987) used in this study.

TABLE 1.4

Grade 12  Pass Rate

Table 1.4 takes into account those who failed and passed Grade 12 in the subsequent year (12 learners) and those who left the school during their schooling for reasons other than failure, for example, relocation.

The following criteria of admission to MHS provides important background information: Admission to the School was based on (1) the results of entrance tests administered in October of the previous year (1986) and (2) former school achievement as revealed by Grade 6 reports from former schools. The School’s criteria for admission to Grade 7 consisted of:

- Grade 6 reports from former schools (the aggregate).

-  A Culture Fair Intelligence Test.^[86]

- An English proficiency test, which consisted of a short essay of about half a   page. I was not involved in the administration or marking of this test and so had no information about this test.

– A mathematics proficiency test. As in the case of MHS’s English proficiency test, I had no information on this test.

The admission tests for the sample were written in October of the previous year (1986). The Grade 6 reports  were considered by the School to be the most important criterion for admission. However, a few pupils were admitted with Grade 6 aggregates below 60%.  Of the Schools admission criteria only the Grade 6 reports are used in this study. I was not able to obtain the scores of the School’s Grade 6 essay admission test. In any case, the admission essay test was marked by only one rater per protocol and thus there would have been no way of establishing the interrater reliability of the School’s essay test.

With regard to the culture-fair test data, the original intention was to include these in the predictive investigation, but owing to the problematic (scientific and political) nature of intelligence tests and the fact that the use of these tests as predictors would not be directly pertinent to the topic, I decided to exclude these tests from this investigation. Suffice it to say that L2 learners who score above average on intelligence tests tend to be better at formal second language learning – and first language learning.^[87]

The School’s policy is that at least half of all admissions should consist of disadvantaged learners. Disadvantaged does not mean low scoring, because the School selects on the basis of good performance as indicated by former (Grade 6) school reports. These disadvantaged entrants come from DET Schools. The investigation will show that many of the Grade 6 reports (of former schools) that belonged to disadvantaged entrants were unreliablein the sense of being inconsistent with their language proficiency scores on the tests in that the English aggregates on these Grade 6 reports were  in many cases radically higher than the scores on the proficiency tests.

The full sample of subjects (N=86) is discussed in detail elsewhere, but for the moment I deal briefly with 70 subjects (Table 1.7). Compare the Aggregate and English scores of the Grade 6 reports of the following two groups of entrants to Grade 7 at the School, who comprise a major part of the sample of subjects used in this study:

(1) CM Primary School (N=33), which provided (at the time this research was conducted) most of the entrants who took English First Language as a subject at MHS. English was the official medium of instruction from Grade 1 at CM Primary School. The learners from this school were generally advantaged. As mentioned, disadvantaged  learners are those who have suffered educational, social and economic deprivation

(2) 28 DET schools (N=37), which provided the vast majority of the entrants who took English Second Language as a subject at the School. English was the medium of instruction from Grade 5 at DET schools. Entrants from DET schools were generally disadvantaged.

TABLE 1.5

Comparison of Grade 6 reports between CM Primary School

and 28 DET Schools (N=70)

The T-Test in Table 1.5 shows that there was no significant difference between the two groups, because the t Stat is less than the critical value.

This equivalence in Grade 6 report scores between these groups plays an important role in the arguments and predictions of subsequent chapters, where I investigate the reliability of the Grade 6 scores of the DET  entrants.

1.7  Measures used in the study

The measures used in the study are now briefly described.  The study only commenced after the intake of learners to the School, and thus these measures differed in purpose from the School’s criteria, which were admission criteria. A detailed description of the measures, e.g. format, instructions, layout,  is given in Chapter 3. For the moment I provide only a brief description of the measures:

I. English proficiency tests. Eight English proficiency tests were administered in January 1987 (Grade 7). I devised the essay tests myself, while all the other tests were obtained from various published sources. The English proficiency test battery consists of:

(i) Two cloze tests from Pienaar’s^[88] “Reading for Meaning”.

(ii) Two dictation tests. These were two restored cloze tests from Pienaar.^[89] The passages from Pienaar used for the cloze tests are different to the passages used for the dictation tests, but they both belong to the same level. (I explain later what I mean by “level”).

(iii) Two essay tests (devised by myself).

(iv) An “error recognition” test.^[90]

(v) A “mixed grammar” test.^[91]

The tests from Bloor et al. consist of multiple-choice items.  The “mixed grammar” test consists of  items that test a variety of structures, hence the term “mixed”.

I shall argue that although the tests used in the study, except for the essay test, may be out of fashion with many testers they are nevertheless still very useful for assessing language proficiency and predicting academic achievement.

II. Grade 6 end-of-year school reports from former schools. The Grade 6 aggregate scores are used.

III. End-of-year English scores and Aggregates from Grade 7 to Grade 11. These scores were obtained from the School’s mark schedules.

IV. Grade 12 results (of 1992 and 1993). These results are those of the JMB (1992) and the Independent Examinations Board (IEB; 1993). The reason why the IEB results are also taken into account is that included in the study are those subjects that failed once between Grade 7 and Grade 12, repeated a year and sat for the IEB Grade 12 examination in 1993. (The JMB matriculation examination ceased to exist after 1992).

1.8  Method overview

Some researchers separate statistical research from empirical research. For Lantolf and

Frawley^[92] empirical research and statistical measurement are distinct. In contrast, when Tremblay and Gardner^[93] state that in their opinion “empirical investigation is essential to demonstrate the theoretical and pragmatic value” of research, their “empirical” research is firmly based on statistics, without which they would have very little of what they consider to be “empirical” research (see also Cziko’s^[94] “empirically-based models of communicative competence”).  Some empirical research is statistically based, while other empirical research (e.g. much of  ethnographical research) isn’t. I adopt Tremblay and Gardner’s view that statistics is an indispensable component of empirical research.

The empirical investigation consists of:

1. An examination of the structure and administration of the English proficiency tests.

2. A predictive investigation, where the English proficiency tests are used to predict academic achievement from Grade 7 to Grade 12.  The reliability of the Grade 6 achievement of entrants from former schools also examined.

Under method  I subsume, as is usual in most studies, the following:

-  Subjects (sampling)

-  Structure of the measures.

-  Procedures of administration and scoring.

A  common design in empirical studies is that data analysis, results and discussion are each reported in separate sections. I depart from this traditional structure and follow Sternberg^[95] who recommends that these be treated together. This is a wise arrangement for this study because the data analysis, discussion and results are closely connected.

A clarification of the following terms are in order: type, method, procedure. Sometimes type refers to such things as multiple-choice type questions versus gap-filling type tests; sometimes method refers to such things as cloze methods versus dictation methods, in other words, methods is used to mean tests. Then there is procedure, e.g. the cloze procedure, the dictation procedure, etc., which also can mean methods or tests. I shall use tests to refer to elicitation techniques, and procedure to the way in which the test is presented and scored.

1.9  Preview of Chapters 2 to 6

Chapter 2 deals with theoretical issues in the testing of language proficiency and academic achievement, where the main focus falls on assessment. The chapter comprises  a review of the literature on the testing of language proficiency and an overview of key concepts such as assessment,validity and reliability.

Chapter 3 describes the sample of subjects and sampling procedures, and the structure and administration of the tests.

Chapter 4 presents the results of the tests and discussion.

Chapter 5 deals with the prediction of academic achievement,  examines the reliability of the Grade 6 reports from previous schools and summarises the findings.

Chapter 6 discusses the implications of the study for language testing and presents the conclusions. The three main implications are: (1) the viability of the distinction between first language and second language, (2) the kind of tests or tasks that should be used in the future, and (3) the problem of rater reliability. Also discussed are a few contemporary initiatives to improve language assessment in South Africa.

1.10  Summary of Chapter 1

The purpose, problem, main topics, method, hypotheses and educational context of the study were specified. The study deals with  the measurement of differences between learners in English proficiency and with assessing the reliability, validity and practicality of discrete-point and/or integrative tests as predictors of academic achievement. Central to the study is the argument that the “old paradigm” of discrete-point and integrative tests and the statistical methods required to measure them are very useful in language acquisition research and educational measurement. The next chapter deals with the theory of language testing where I examine what it means to describe language behaviour and language tests as “authentic”.

Endnotes

^[1]Davies, A. Principles of language testing. p.4, 1990.

^[2]Ibid., p.2.

^[3]Saville-Troike, M. ‘What really matters in second language learning for academic achievement.’ TESOL Quarterly, 18(2):199-219.

^[4]Cummins, J. Wanted: A theoretical framework for relating language proficiency to academic achievement among bilingual students’,  in Rivera, C. (ed.). Language proficiency and academic achievement, 1984.

^[5](1) Diller, K.C., Individual differences and universals in language learning aptitude, 1981. (2) Skehan, P. Individual differences in second language learning, 1989. (3) Skehan, P. A cognitive approach to language learning, 1998.

^[6]Rivera, C., The ethnographical/sociolinguistic approach to language proficiency assessment.1983, p.xii.

^[7]Bachman, L.F.’Assessment and evaluation. Annual Review of Applied Linguistics (1989), 10:210-226 (1990a), p.210

^[8]Lakoff, G., Women, fire and dangerous things, 1987.

^[9]Bachman, L.F. Fundamental considerations in language testing, 1990b.

^[10]Davies, A. Principles of language testing, 1990.

^[11]Alderson, J.C. ‘Who needs jam?’, in Hughes, A. and Porter, D. Current developments in language testing, 1983, p.90.

^[12]Douglas, D.’Developments in language testing.’ Annual Review of Applied Linguistics, 15:167-187 (1995), p.176.

^[13]Moss, P. ‘Can there be validity without reliability?‘ Educational Researcher, 23 (2), 5-12 (1994).

^[14]Alderson, J.C. and Clapham, C. ‘Applied linguistics and language testing: A case study of the ELTS test.’ Applied Linguistics, 13 (2), 149-167 (1992).

^[15]Ibid., p.2.

^[16]Widdowson, H.G.‘Skills, abilities, and contexts of reality.’ Annual Review of Applied Linguistics, 18, 323-333 (1998), p.323.

^[17]Principles of language testing, 1990, p.2.

^[18]Skehan, P. A cognitive approach to language learning, 1998, p.153.

^[19]Bachman, L.F. and Palmer, A.S.Language testing in practice 1996, p.17.

^[20]Brown, J.D. ‘Language programme evaluation: A synthesis of existing possibilities’, 1989, p.235.

^[21]Stern, H. H. Fundamental concepts of language teaching, 1983.

^[22] (1)Wilkins, D.A. ‘Notional syllabuses revisited.’ Applied Linguistics, 2 (1), p.83-89 (1981), p.83.

(2) Brumfit, C.J. Notional syllabuses revisited: A response. Applied   Linguistics, 2(1), 90-92 (1981), p.90.

^[23]Ur, P.A course in language teaching: practice and theory, 1996, p.37.

^[24]Yeld, N. Communicative language testing, 1986, p.36.

^[25](1)  Corder, S.P. Error analysis and interlanguage, 1981, p.20.

(2) Davies, A. Principles of language testing, 1990, pp.20-21.

^[26]Davies, ibid., 21.

^[27]Byrnes, H. and Canale, M. Defining and developing proficiency: Guidelines, implementations and concepts, 1987, p.15.

^[28]Spolsky, B.Conditions for second language learning, 1989, p.138.

^[29]Ur, P.A course in language teaching: practice and theory, 1996, p.37

^[30]Davies, A. Principles of language testing, 1990, p.1.

^[31]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.182.

^[32]Lazaraton, A. ‘Qualitative research in applied linguistics: A progress report.’ TESOL Quarterly, 29 (3), 455-471 (1995, p.455.

^[33]Magnan, S.S. Review of Creswell, J.W. 1994. Research design: qualitative and quantitative approaches, 1997.

^[34]Rushton, J.P. Race, evolution and behaviour, 1995; his frontispiece.

^[35]Spolsky, B. ‘The limits of authenticity in language testing.’ Language Testing, 2, 31-40 (1985).

^[36]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988).

^[37](1) Macdonald, C.A. English language skills evaluation (A final report of the Threshold Project), Report Soling-17, 1990a.

(2) Macdonald, C.A. Crossing the threshold into standard three in black education: The consolidated main report of the Threshold Project, 1990b.

^[38](1) Bennett, A. and Slaughter, H. ‘A sociolinguistic/discourse approach to the description of the communicative competence of linguistic minority children’, in Rivera, C.The ethnographical/ sociolinguistic approach to language proficiency assessment, 1983.

(2) Jacob, E. ‘Studying Puerto Rican children’s informal education at home’, in Rivera, C. (ed.). The ethnographical/sociolinguistic approach to language proficiency, 1983.

(3) Phillips, S. ‘An ethnographic approach to bilingual language proficiency assessment’, in Rivera, C. (ed.). The ethnographical/sociolinguistic approach to language proficiency assessment, 1983.

^[39]Harrison, A. ‘Communicative testing: Jam tomorrow?’, in Hughes, A. and Porter, D. (eds.). Current developments in language testing, 1983. p.84.

^[40]Morrow, K. ‘Communicative language testing: Revolution or evolution’, in Alderson, J (ed.). Issues in language testing, 1981, p.12.

^[41]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.185.

^[42]Psychometrist has two meanings: 1. a statistician, and 2. somebody with the paranormal power to find lost objects. I guess the double meaning is not lost on Spolsky.

^[43]Spolsky, B. ‘The limits of authenticity in language testing.’ Language Testing, 2, 31-40 (1985), pp.33-34.

^[44]Spolsky, B.Measured words, 1995, p.357.

^[45]Davies, A. Principles of language testing, 1990, p.16.

^[46]Cziko, G.A. ‘Improving the psychometric, criterion-referenced, and practical   qualities of integrative testing.’ TESOL Quarterly, 16 (3), 367-379 (1982), pp.27-28.

^[47](1) Spolsky, B. ‘The limits of authenticity in language testing.’ Language Testing, 2, 31-40 (1985).

(2) Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.185.

^[48]Gamaroff, R. Psychometrics and reductionism in language assessment. Paper   presented at the SAAAD/SAARDHE conference “Capacity-building for quality teaching and learning in further and higher education”, University of Bloemfontein, 22-24 September, 1998e.

^[49]Magnan, S.S. Review of Creswell, J.W. 1994. Research design: qualitative and quantitative approaches, 1997.

^[50]Terre Blanche, M.‘Crash.’ South African Journal of Psychology, 27 (2), 59-63 (1997), p.61.

^[51]Van der Walt, J. The implications for language testing of IBSA [Institutionalised Black South African English].  National Association of Educators of Teachers of English (NAETE) conference ” Training teachers for the South African context,  Potchefstroom College of Education, September 17-18, 1998.

^[52]Van der Walt’s paper followed immediately after the presentation of Makalela’s (1998) paper “Institutionalized Black South African English” (IBSAE) in which Makalela advocates that IBSAE be adopted as the norm among blacks in South Africa. The examples of IBSAE cited above are those given by Makalela.

^[53]Messick, S.Validity, 1987, p.3.

^[54]Cziko, G.A. ‘Improving the psychometric, criterion-referenced, and practical   qualities of integrative testing.’ TESOL Quarterly, 16 (3), 367-379 (1982).

^[55]Gultig, J., Lubisi, C., Parker, B. and Wedekind, V. Understandingoutcomes-based education: Teaching and assessment in South Africa, 1998, p.12.

^[56]Rowntree, D.Assessing students: How shall we know them, 1977, p.185.

^[57]Davies, A. Principles of language testing, 1990, p.19.

^[58]Rea, P. ‘Language testing and the communicative language teaching curriculum’, in Lee, Y.P. et al. New directions in language testing, 1985.

^[59]Rowntree, D.Assessing students: How shall we know them, 1977, p.185.

^[60]Rea, P. ‘Language testing and the communicative language teaching curriculum’, in Lee, Y.P. et al. New directions in language testing, 1985, p.29

^[61]Bachman, L.F.Fundamental considerations in language testing, 1990b, p.20.

^[62]Brown, J.D. ‘Language programme evaluation: A synthesis of existing possibilities’, in Johnson, R.K. (ed.). The second language curriculum, 1989, p.223.

^[63]Dreyer, C. Testing: The reason why pupils fail. National Association of Educators of Teachers of English conference (NAETE) ” Training teachers for the South African context, Potchefstroom College of Education, September 17-18, 1998.

^[64]Rowntree, D.Assessing students: How shall we know them, 1997, p.4.

^[65]Ibid., p.8.

^[66]Ibid., p.7.

^[67]Ibid.

^[68]Rowntree, D.Assessing students: How shall we know them, 1997, p.7.

^[69]Gamaroff, R. ‘Language, content and skills in the testing of English for academic purposes.’South African Journal of Higher Education, 12 (1), 109-116 (1998b).

^[70]Rea, P. ‘Language testing and the communicative language teaching curriculum’, in Lee, Y.P. et al. New directions in language testing, 1985. p. 29.

^[71]Weir, C.J. Understanding and developing language tests, 1993, p.68.

^[72]An applied linguist, who had read some of my research, queried whether I was doing applied linguistics research or educational research. “Applied linguistics” in the restricted meaning of the term is not directly concerned with predicting academic achievement, but “educational linguistics” is, which has a close connection to applied linguistics. The most important reason for the study of  academic language proficiency is its connection to academic achievement.

^[73]Bachman, L.F. and Palmer, A.S.Language testing in practice, 1996, p.4.

^[74]Alderson, J.C.  ‘Report of the discussion on general language proficiency’,  in Alderson, J.C. and Hughes, A. Issues in language testing: ELT Documents III. (The British Council, 1981a, p.190.

^[75](1)Brown, J.D. A closer look at cloze: Validity and reliability, 1983.

(2)Hale, G.A., Stansfield, C.W. and Duran, R.P.TESOL Research Report 16, 1984.

(3) Oller, J.W., Jr. ‘Cloze tests of second language proficiency and what they measure.’ Language Learning, 23 (1), 105-118 (1973).

(4) Oller, J.W., Jr.  ‘A consensus for the 80s’, in Issues in language testing research, 1983).

(5) Oller, J.W., Jr. ‘Cloze, discourse, and approximations to English’, in  Burt, K. and Dulay, H.C. New directions in second language learning, teaching and bilingual education, 1976.

(6) Oller, J.W., Jr. ‘”g”, “What is it?’, in Hughes, A. and Porter, D. (eds.). Current developments in language testing, 1983a.

(7) Oller, J.W., Jr. Issues in language testing research,  1983b.

(8) Stubbs, J. and Tucker, G. ‘The cloze test as a measure of English proficiency.’ Modern Language Journal, 58, 239-241 (1974).

^[76]Oller, J.W., Jr.  and Kahn, F.  Is there a global factor of language proficiency?, in Read, J.A.S. Directions in language testing, 1981.

^[77]Carroll, J.B.Psychometric theory and language testing, 1983, p.82.

^[78]Oller, J.W., Jr.  ‘A consensus for the 80s’, in Issues in language testing research, 1983, p.36.

^[79] Many scientists, in contrast to many applied linguists, have not given up looking for Grand Unified Theory (GUT). Many applied linguists would probably say that physics deals with non-living matter whereas language testing deals with human beings. But this, in my view, is no justification for rejecting the search for unifying linguistic principles in humans; if one is interested in linguistic science, and not just in linguistic thought.

^[80]Bachman, L.F. and Palmer, A.S.Language testing in practice, 1996, p.9.

^[81]The DET  was the education department in charge of black education up to 1994. It is now defunct.

^[82]At a translation committee meeting at the University of Fort Hare  in April 1998, the secretary of the meeting suggested that the term “mother tongue” was sexist.

^[83]Educamus. Editorial: Internal promotions, 36 (9), 3 (1990), p.3

^[84]Young, D. ‘A priority in language education: Language across the curriculum in black education’, in Young, D. and Burns, R. (eds.). Education at the crossroads, 1987.

^[85]Mcintyre, S.P. ‘Language learning across the curriculum: A possible solution to poor results.’ Popagano, 9 and10, June (1992), p.10.

^[86]Cattell, R.B.Measuring intelligence with culture-fair tests, 1973.

^[87]Mitchell, R. and Myles, F. Second language acquisition, 1998.

^[88]Pienaar, P. Reading for meaning: A pilot survey of (silent) reading standards in Bophuthatswana,1984, pp.59 and 61.

^[89]Ibid., pp.58 and 62.

^[90]Bloor, M., Bloor, T., Forrest, R., Laird, E. and Relton, H. Objective tests in English as a foreign language, 1970, pp.70-77.

^[91]Ibid., pp.35-40.

^[92]Lantolf, J.P. and Frawley, W. ‘Proficiency: Understanding the construct.’ Studies in Second Language Acquisition (SLLA), 10 (2), 181-195 (1988), p.181.

^[93]Tremblay, R.F. and Gardner, R.C. ‘Expanding the motivation construct of language learning.’ The Modern Language Journal, 79 (4), 505-518 (1995), p.505.

^[94]Cziko, G.A. ‘Some problems with empirically-based models of communicative competence.’ Applied Linguistics, 5 (1), 23-37 (1984).

^[95]Sternberg, R.J.The psychologist’s companion: A guide to scientific writing for students and researchers, 1993, p.53.

1.1  Introduction: The  problem and purpose of the study

1.2  Psychometrics and norm-referenced testing

1.3  Summative assessment

1.4  The One Best Test

1.5  Hypotheses of the study

1.6  Historical and educational context

1.7  Measures used in the study

1.8  Method overview

1.9  Preview of Chapters 2 to 6

1.10 Summary of Chapter 1

CHAPTER 2: Theoretical Issues in the Testing of Language Proficiency and Academic Achievement

2.1   Introduction

2.2   Ability, cognitive skills and language ability

2.3   Competence and performance

2.4   Proficiency

2.5   The discrete-point/integrative controversy

2.6   Cognitive and Academic Language Proficiency (CALP) and “test language”

2.7   Language proficiency and academic achievement

2.8   Validity

2.8.1 Face validity

2.8.2 Content validity

2.8.3 Construct validity

2.8.4 Criterion validity: concurrent and predictive validity

2.9    Reliability

2.9.1  Approaches to the measurement of reliability

2.10   Ethics of measurement

2.11   Summary of Chapter 2

CHAPTER 3: Sampling, and Structure and Administration of the English Proficiency Tests

3.1      Introduction

3.2       Sampling procedures for the selection of subjects

3.2.1     The two main groups of the sample: First Language (L1) and second Language (L2) groups

3.3        Structure and administration of the English proficiency tests

3.3.1    The cloze tests

3.3.1.1  Theoretical overview

3.3.1.2  The cloze tests used in the study

3.3.2      The essay tests

3.3.2.1  Theoretical overview

3.3.2.2  The essay tests used in the study

3.3.3      Error recognition and mixed grammar tests

3.3.3.1  Theoretical overview

3.3.3.2  Error recognition and mixed grammar tests used in the study

3.3.4    The dictation test

3.3.4.1  Introduction

3.3.4.2  Theoretical overview

3.3.4.3  The dictation tests used in the study

3.3.4.4  Presentation of the dictation tests

3.3.4.5  Method of scoring of the dictation tests

3.4       Summary of Chapter 3

CHAPTER 4: Results of the proficiency tests

4.1    Introduction

4.2    Reliability coefficients

4.3    Analysis of variance of the dictation tests

4.4    Validity coefficients

4.5    Descriptive data of the L1 and L2 groups

4.6    The L1 and L2 groups: Do these levels represent separate populations?

4.7    Comparing groups and comparing tests

4.8    Error analysis and rater reliability

4.8.1  Rater reliability among educators of teachers of English

4.8.1  Interrater reliability: scores and judgements

4.9    Summary of Chapter 4

CHAPTER 5: The Prediction of Academic achievement

5.1  Introduction

5.2  Correlational analysis and multiple regressions of the predictions

5.3  Frequency distributions of the predictions and data analysis

5.3.1 What do we mean by correlation?

5.4  General discussion of language proficiency tests as predictors of academic achievement

5.5  The reliability and predictive validity of the Grade 6 reports of previous schools

5.5.1 Introduction

5.5.2 An examination of the Grade 6 reports

5.6    Summary of the findings and their generalisability

5.5  Summary of Chapter 5

CHAPTER 6: Implications and Conclusions

6.1 Introduction

6.2 The L1/L2 and native speaker/non-native speaker distinctions

6.3  Negotiating the task-demands and the “Threshold Project”

6.4  Competence-based Education  training (CBET) and “Outcomes-based Education” (OBE)

6.5  Rater consistency, or reliability

6.6  Paradigm lost paradigm regained (recreated?)

6.7  Conclusion of the study

6.8  Summary of Chapter 6

Tables

Table 1.1 Rowntree’s Schema of Assessment and Evaluation

Table 1.2 Rea’s Schema of Assessment

Table 1.3  Grade 9 Pass Rate

Table 1.4  Grade 12 Pass Rate

Table 1.5  Comparison of Grade 6 reports between Connie

Minchin Primary School and 28 DET Schools

Table 1.6   Comparison of Grade 6 reports between mother tongue

English speakers and non-mother tongue English speakers

Table 2.1    Functionalist and Structuralist Levels of language

Table 3.1    Detailed Analysis of the L1 Subjects

Table 3.2    Detailed Analysis of the L2 Subjects

Table 3.3    Normal Presentation of Dictation using One Presenter with One Presentation to all Four Groups

Table 3.4    Presentation of Dictation with Four Presenters: First Presentation of Each Presenter

Table 4.1    Reliability Coefficients of All the Tests    107

Table 4.2    Cumulative Reliability Coefficients: Comparison between Error Recognition Test (ER) and Mixed Grammar Test  (GRAM)

Table 4.3    Analysis of Variance of the Dictation Tests with First Presentation

Table 4.4    Validity Coefficients of English Proficiency Tests

Table 4.5    Means, Standard Deviations and z-tests for the L1 and L2 groups

Table 4.6    Frequency Distribution of all the Tests

Table 4.7    Occupation  of  Parents of the  L2 Subjects

Table 4.8    Error Recognition test: Percentage Error

Table 4.9    Protocol 1 (L2): Summary NAETE Judgements of  Raters on Content, Grammar and Spelling

Table 4.10 Protocol 2 (L1): Summary of NAETE Judgements of  Raters on Content, Grammar and Spelling

Table 5.1  Grade 6 to Grade 11 Correlational Analysis of the Prediction of Academic Achievement with Aggregate as Criterion

Table 5.2    Stepwise Multiple Regression Analysis of Predictions

Table 6.1.   Attitudes to Testing

Figures

Figure 1.1   Rea’s Schema of Assessment

Figure 1.2   Rowntree’s Schema of Assessment & Evaluation

Figure 2.1   Second Language Proficiency as a Criterion Variable and as a Predictor Variable

Figure 4.1   Comparison between the reliabilities of ER and GRAM

Figure 4.2   A Comparison of the Groups at Mmabatho High School with the Middle School (MID) on the Cloze Tests

Figure 4.3   Frequency Distribution of the Scores Awarded by the 24 Raters on Protocol 1

Figure 4.4   Frequency Distribution of the Scores Awarded by the 24 Raters on Protocol 2

Figure 4.4   Histogram of frequency distribution of NAETE scores on Protocol 2 (L1)

Figure 5.1   NTL1 (Non-Tswana L1) Grade 7 to Grade 11 Pass Rate

Figure 5.2   TL1 (Tswana L1) Grade 7 to Grade 11 Pass Rate

Figure 5.3   L1 (NTL1 & TL1) Grade 7 to Grade 11 Pass Rate

Figure 5.4   L2 Grade 7 to Grade 11 Pass Rate

The study investigates the ability of English proficiency tests (1) to measure levels of English proficiency among learners who have English as the medium of teaching and learning, and (2) to predict long-term academic achievement (Grade 7 to Grade 12). The tests are “discrete-point” tests, namely, error recognition and grammar tests (both multiple-choice tests), and “integrative” tests, namely, cloze tests, essay tests and dictation tests.

The sample of subjects consists of two groups: (1) those taking English as a First Language subject and those taking English as a Second Language subject. These groups are given the familiar labels of L1 and L2. The main interest lies in the L2 group. The main educational context is a high school in the North West Province of South Africa.

The empirical investigation is divided into four parts:

(1) A description of the battery of English proficiency tests. (Chapter 3). These tests were given to Grade 7 school entrants.

(2) An examination of the validity and reliability of the battery of the English proficiency tests . (Chapter 4). High correlations were found between all of the tests and a substantial difference in English proficiency was found between the L1 and L2 groups.

(3) A longitudinal investigation of predictive validity, where the English proficiency tests were used as the predictors, and academic achievement (Grades 7 to 12) as the criterion. (Chapter 5). The main interest of the longitudinal investigation lies in long-term prediction. It is generally believed that low English proficiency is a major cause of academic failure. The longitudinal study corroborates this belief empirically and also shows that very high English proficiency is a good predictor of success. The matriculation exemptions of the L1 group, scored substantially higher on the English proficiency tests than the L2 group, were three times higher than those of the L2 group.

(4) A longitudinal investigation of the predictive validity of the Grade 6 reports. (Chapter 5). These Grade 6 reports served as the main criterion for admission to Grade 7 at the high school. Almost all of the Grade 6 reports of the L2 group emanated from former Department of Education and Training (DET) schools. Most of the Grade 6 reports of the L1 group emanated from a “feeder” school in close proximity to the high school. The L1 Grade 6 reports were found to be good predictors, while the L2 Grade 6 reports were found to be poor predictors. A probable reason for the poor predictions of the L2 Grade 6 reports was that these reports were inflated, and therefore unreliable.

The outline of the chapters is as follows:

Chapter 1 describes the scope of the study.

Chapter 2 deals with theoretical issues in the testing of language proficiency and academic achievement. The chapter comprises a review of the literature on language testing and a discussion of germane concepts such as ability, competence, proficiency, authenticity, norm-referenced tests, discrete-point tests, integrative tests, assessment, validity and reliability.

Chapter 3 describes the sample of subjects and sampling procedures, and the structure and administration of the tests.

Chapter 4 presents the results of the English proficiency tests and discussion. Included in the chapter is an investigation of rater reliability among a group of educators of teachers of English.

Chapter 5 deals with the prediction of academic achievement, investigates the reliability of the Grade 6 reports from previous schools, summarises the findings and examines the generalisability of the findings.

Chapter 6 discusses the implications of the study for English testing and presents the conclusions. The four main implications dealt with are: (1) the viability of the distinction between English first language and English second language, (2) the kind of English proficiency tests or tasks that should be used, (3) the problem of rater reliability, and (4) the necessity of psychometric measurement. Woven into the discussion of the implications are a few contemporary initiatives to improve language testing in South Africa and elsewhere.

Author: Raphael Gamaroff

Introduction

The Structuralist-Functionalist controversy in language learning

Internal and external grammar, and problem solving

Conclusion

References

Introduction
Language learning/teaching theory is shot through with numerous controversies but none so obdurate as the “structure versus function” dispute. One wonders why the problem exists at all, because the concepts form and function are primordial ingredients in all scientific investigation. They are not made, not discovered, but given.

So why do applied linguists have a problem? Let us compare language learning with the study of science.

One cannot think two thoughts at the same time. Thus, one cannot muse on the marvellous structure  (anatomy) of the eye while simultaneously thinking about how its function (physiology). However, no medical student has ever lost any sleep over which is posterior, anatomy or physiology.

So why all the fuss in language learning? Trying to answer that question is what this marginal note is about.

At the 1990 South African Applied Linguistics (SAALA) conference in Durban, a reception was held in a long reception room. At one end, a hubbub of delegates quaffing and scoffing; at the other, a quartet playing Bach’s Brandenburgs. Except for one person (myself), no one paid any attention to the exquisite music.

Two flies on the wall:

Fly One: Such boors, no culture.

Fly Two: But this is no time to play Bach, except for that solitary asocial creature huddled in the corner.

Each fly had its own idea as to which was important, which marginal; culture or esprit de corps. Each language theorist has his/her own idea as to which is important, and which is marginal.

The Structuralist-Functionalist controversy in language learning

The controversy in applied linguistics revolves around whether language learning is best learnt through the structural (formal) or the functional approach. The structural approach is also called the “grammatical” approach.

Structuralism in applied linguistics refers to the study of language elements below and up to sentence level. Functionalism refers to the study of the message as a whole.

What I would like to do in this essay is examine the relation between the terms structure, function and grammar, and show two things: first, that these three concepts are abstractions from a holistic process; second, that grammar resides and presides at the core of language learning and language use.

In this discussion we focus on language learning in the school situation, and not on preschool language learning.

Let us now tackle the main issues of this controversy by examining sentence meaning and discourse meaning.

There is a difference between sentence meaning (semantic meaning) and the meaning of a discourse as a whole (pragmatic meaning) [Brown 1987:14 and Leech 1983].

The sentence in isolation from a functional context only has potential meaning. It is this potential meaning which has to become actualised in language use (sociolinguistic context).

What is meant by the potential meaning of sentences?

It is obvious that every sentence in isolation from its sociolinguistic context must contain meaningful units. For example, each of the three words in the sentence I am reading is a meaningful unit. These three units are combined into a larger meaningful unit, namely, the sentence.

The sentence, in turn combines, with other sentences to form an even larger unit called discourse. And it is only at the discourse level that parts of sentences and sentences come alive. Meaning at the sentence level and below is referred to in linguistics as semantic meaning, while meaning at the discourse level is referred to as pragmatic or sociolinguistic meaning. So from the point of view of discourse ( language use), the sentence has potential meaning only.

Widdowson (1979:119), using different terminology, makes the same distinction as Brown and Leech. Widdowson distinguishes between two categories; the “signification of linguistic forms” and their “communicative value as utterances in context”.

Language behaviour involves both of these categories. Widdowson (1979:119) explains his distinction:

….there is no one-to-one correspondence between the signification of linguistic forms and their communicative value as utterances in context. Discourse meanings are to some degree unpredictable. At the same time they cannot be entirely unpredictable: the relationship between form and function is not purely arbitrary, or otherwise there would be no linguistic basis for communication at all…The fact that context may override the meaning associated with sentences does not mean that one cannot fruitfully explore this meaning…..

In “utterances in context” (discourse), one meaning may take many forms, or many forms may express one meaning. It is this reciprocal relationship between form and meaning which makes possible the infinite creativity of language. However, this infinite creativity, if not controlled, would obstruct rather than facilitate discourse (Ricoeur 1974).

In transformational grammar (TG), the terms surface structure and deep structure are used to refer to form and meaning respectively. However, TG theory is not enough to explain the nature of discourse, i.e. language use. For example, consider the following pair of utterances made famous by Chomsky:

1) “He is easy to please”, and

2) “He is eager to please”.

In both sentence 1) and sentence 2), He is the grammatical subject. TG theory improves on this structuralist theory by pointing out that He in sentence 1) is the recipient of to please, whereas He in sentence 2) is the agent of to please; two parallel structures revealing in sentence 1) a complex relation between deep structure (sentence meaning) and surface structure, and in sentence 2) a relatively simple relation between deep structure and surface structure.

In terms of discourse, the TG explanation does not explain how these utterances function in language use. In TG theory, both He is easy to please and He is eager to please would be regarded as

declaratives, i.e. would be described in terms of mood: but mood conveys nothing about language use. He is eager to please could have at least the following three additional meanings, depending on the context (Brown 1984):

1. He is not eager to please.

2. He is definitely eager to please.

3. He is definitely not eager to please.

4. He but not she is eager to please.

Sociolinguists would consider TG inadequate because it only focuses on semantic meaning (sentence meaning) and not pragmatic meaning (discourse, language use).

The distinction between semantic and pragmatic meaning can also be explained in the following way (Leech 1983):

- the meaning of X, which is the semantic or sentence meaning,

and

what you mean by X, which is the pragmatic or sociolinguistic meaning.

The sentence I am reading means that there is somebody, namely me who is reading. This meaning is the semantic or sentence meaning.

We use I am reading in a life situation.

Student A is sharing a room with Student B. Student A is reading in the room while Student B is out. Student B returns, sees Student A bowed over a book, and shouts: What are you doing? It is obvious to Student A that Student B is not requesting information as to whether Student A is reading a book – it is obvious that this is so.

Suppose Student A’s answer is I’m reading. The semantic meaning of this utterance is clear, namely, Student A is not eating, or sleeping, but reading. But what does Student A mean by I’m reading and what does Student B mean by What are you doing?

Here are a few possibilities of the pragmatic meaning of these two sentences;

Question: “What are you doing?”

1. Hey, what are you doing in my bed?”

2. What a miracle, you’re reading a book!

3. We’ve been looking all over for you, and here you are all the time, rotting at your desk.

Answer: “I’m reading”

1. Please don’t disturb me.

2. It’s no good, I’ll never speak to you again.

3. I’m so bored, the TV is not working; what else is there to do but read – yawn.

4. Who the blazes do you think you are to speak to me like that?

5. You illiterate idiot, go back to your comics.

So the pragmatic context of language does not merely go beyond the sentence meaning, it makes the sentence actually meaningful, and actual meaning is the only kind of meaning that we can live by.

From the point of view of language study, pure linguistics (for example, Chomskyan linguistics) is not interested in “actual” meaning, but only in sentence meaning. Applied linguistics (language teaching), on the other hand, is interested in how sentences combine to formulate discourse.

To summarise the discussion so far:

The structural syllabus focuses on sentence meaning and parts of the sentence as a preparation for the communication of meaning. The functional syllabus focuses directly on the use of language in communication.

The way I see the problem is that it is not a matter of merely looking at individual differences in learners such as learning styles, and then deciding which of the two approaches is the better one. Individual differences could obscure the hidden workings of language.

And it is the workings of language which have to be laid bare. What, with all the tearing and pulling at its “syntactic joints and semantic flesh” (Johnson 1985), language is blanched of its lifeblood; grammar.

But if grammar is structure, how can structure be drained off from itself? Furthermore, what has grammar got to do with function if grammar is synonymous with structure?

The answers to these questions might help us to decide where grammar resides; at the face and/or at the fundament of language.

Here is an example of some of problems involved in defining grammar:

The following is an excerpt from a guide to the course on Practical English (University of South Africa 1988):

Correct grammar is important because grammatical mistakes can lead to people not understanding what you are saying or writing, and, secondly, because grammatical mistakes are like dirty, torn clothes – they give the people you meet a very bad impression of you. If you have good ideas and a good understanding of the work, but your assignments are full of grammatical errors, you will receive low marks, or may even fail (see Du Toit and Orr 1987 for a similar view).

Students, many of whom come from “grammar-based” schools, are firstly told that bad grammar fuddles meaning, and then warned that even if they get their meaning across, they will be penalised for soiling their clothes. The Practical English exam answers do not only have to mean right, they have to look right.

The message to the student: the message is not enough.

The clothing metaphor of language
What I would like to do now is examine whether this clothing metaphor of language is helpful.

In defense of this metaphor, I do not see anything wrong with giving students the impression that grammar is nothing more than dirty clothes. From a practical point of view there is no other adequate way of explaining what editing means, because it would be foolish in a Practical English course to burden students with the paradoxes we are dealing with here.

But the danger is that the understanding of grammar by teachers -including the “coco” (communicative competence [Ridjanovic 1983]) contingent – may remain a superficial matter.

The notion of grammar as external to meaning is confused. This confused view is exemplified in such metaphors as “grammar is the clothing of thought” (see the discussion above on the UNISA Practical English Guide). This confusion is at least double. There is the grammar of the in-built “Language Acquisition Device” (LAD) (Chomsky 1957) and there is the editing at the last stage of the writing process.

Internal and external grammar, and problem solving

The LAD kind of grammar – internal grammar – involves problem solving; at the conscious and at the unconscious levels. Editing also involves problem-solving.

We now examine the relationship between these two kinds of grammar (namely, 1. grammar external to meaning and 2. LAD grammar) and problem solving.

The following authors are of interest here: Baijnath (1991), Murray (1990) and Jeffery (1990).

Baijnath (1990) has tried to trace “the writing problems of EAP [English for Special Purposes] students to their roots”. Baijnath found that many students using the process approach to writing made little progress. The errors of students were classified into a hierarchy of difficulty; for example, reasoned argument appeared towards the top, and “grammar” appeared towards the bottom.

Bajnaith interviewed the students. With regard to grammar, it was discovered that almost all the at risk students did not bother correcting their errors. Baijnath did not offer any reasons why students did not attempt to correct their grammar. Before I suggest possible reasons, it would be useful to read Murray’s (1990:140) views about grammar:

In the case of students’ writing, I began to realise that if I corrected all the surface language errors – grammar, spelling, punctuation and vocabulary – their essays were still not satisfactory: they did not answer the question; they contained irrelevant information; the overall argument was difficult to follow…The essential problem is that they did not distinguish between spoken and written English. Spoken English is context-embedded, that is, it can be understood in terms of the situation in which it occurs. Therefore, it does not need to be explicit. Clarification can always be sought if meaning is not clear. Written English, on the other hand, is context-reduced: the reader relies solely on the words on the page. The writer, therefore, cannot assume shared knowledge; she must be explicit.

In both Baijnath and Murray a distinction is made between grammar and cognitive and academic language proficiency (CALP) [Cummins 1984). The inference is that if grammar is good, it does not follow that the organisation of ideas is good.

The reason, I suggest, why students do not pay much attention to correcting grammar errors is that these errors are not just "dirty-clothing" in need of a wash. If soiled linen was all there was to it, students would not object to spending time in the laundry. The problem is that grammar errors go far deeper than editing.

The distinction between competence and performance is useful here.

Competence refers to the language system, to what the learner knows about the language. This system is called various things by various authors, e.g. "langue" (Saussure 1959), "built-in syllabus (Corder 1981).

Performance refers to slips of the tongue or pen, or memory blocks. Although the distinction between competence and performance is problematic (O'Connell 1988:272), Taylor (1988) and Widdowson 1989), the distinction is still useful in explaining Baijnath's data.

One possible explanation why students do not correct their performance "mistakes" is that almost all of these mistakes are not slips of some kind, but rather competence "errors", i.e. a systematic or in-built deficit. In such circumstances, no amount of editing is going to help, because there was nothing to slip on in the first place. Kroma's (1988:43) view is that students can correct their own errors. The point is whether Kroma's students can correct these errors (minor mistakes like spelling excluded) without outside intervention.

A distinction should be made between an in-built deficit, which implies both conscious and unconscious incompetence, and a temporary deficit which can be remedied by "consciousness raising", i.e. teacher intervention.

Whether students need teacher intervention or not, the main point here is that academic writing cannot be taught properly if grammar is merely regarded as an editing exercise rather than as an integral part of the writing process.

Jeffery (1990:120) describes the role of grammar in speaking and writing in the following way:

''... we do need that [grammatical categories - brackets added] in order to gain control of WL [written language - brackets added], for whereas SL (spoken language – brackets added] requires no conscious grammatical understanding, WL does: the most elementary reading and writing presuppose word and sentence at least; and progress is awkward without noun, verb, number, tense, phrase, clause, and so on. These categories come naturally to nobody, and with difficulty to some, for they are all artificial abstractions from the flow of speech. Nobody speaks in words and sentences; therefore everybody needs to be taught about them.”

”Inchoate thought has to be organised in order to make your meaning clear to your readers (and yourself), and that takes skill in arranging the units of WL into structures…So it can come about that after twelve years of schooling some fluent speakers falter on paper; being unschooled in grammar they are still not consciously aware that you cannot depend solely on intuition in writing as you can in speaking.”

To summarise Jeffery’s view: grammar resides in and presides over the whole process of language/cognition, from the phoneme to the completed message. Grammar is more than a stock of static structures; it is a dynamic ability to manipulate language/ideas, and it is this kind of grammatical competence that needs to be developed in the teaching of academic skills.

One caution; Jeffery’s view that spoken language does not require “conscious grammatical understanding” (in Jeffery’s sense of the term) is questionable (Murray above expresses the same view as Jeffery). The boundary between speaking and writing – indeed between all the four skills – is not so clear cut. It may be that speaking in its beginnings is an unconscious activity, but as the speaking skill develops there is an increase in the conscious activity of selecting and arranging of linguistic elements, which in a way is like creating a story (Wells 1987:197). Manipulating grammatical elements (conscious grammatical understanding) originates in informal listening, then moves through a continuum (and combination) of informal speaking and writing to formal listening, formal speaking and formal writing (the last three also occurring in various combinations).

To summarise the discussion so far:

The division of language into structure and function is misleading, because the medium and the message, to use Saussure’s (1959) tarnished metaphor, are two sides of the same coin. The human mind abstracts the universe into form and function, body and mind, time and space, and so on. These abstractions are productive because they reveal reality; yet, abortive, because they distort it.

Finally there are two authors I would like to mention who have made interesting contributions to the debate, Langacker (1990) and Coe (1987).

Langacker’s (1987, 1990), “cognitive grammar” explains language as a conceptual unity of grammar and meaning, where lexicon, morphology and syntax form a gradation. At the Language, Thought and Culture conference (University of the Witwatersrand April 1990) Langacker on being asked “What is the use of cognitive grammar?” replied “It’s Poetry”. What Langacker means is not clear, but I would guess he means that cognitive grammar is a thing of beauty, not to be swallowed up in the practical concerns of language learning. Yet, cognitive grammar is not limited to linguistica poetica, because it does expose the deficiency of splitting language into function words (syntactic joints) and content words (semantic flesh). Therefore Langacker’s insights may be useful in language learning theory.

Coe (1987:13) ventures off the well-beaten track of pedestrian language learning theory and returns with a concept borrowed from New Rhetoric (via Hegel); sublation. This term denotes the simultaneous preservation and transcendence of the form/function antithesis. Coe, addressing composition writers, tries to reconcile the form/function dichotomy using the paradox of preservation and transcendence. This paradox reminds both the structuralist and the functionalist that the abstraction of language into form and function is one dangerous dichotomy to be treated with caution.

Applied linguists, sandwiched between sublation and “poetry”, need to get comfortable with the with New Rhetoric and cognitive grammar . On the other hand, the pure linguists, who prefer art to craft should realise that artists (including poets) never become philosopher kings: Socrates has other plans.

Conclusion

I did not focus in this marginal note on the difference between first and second language learning. What I tried to do was to paint a neutral landscape of one of the main problems common to both kinds of language learning; structuralism and functionalism.

The painting can be – must be – painted on and viewed from both sides of the canvas. On the one side – “structure in grammar”; on the other – “function in grammar”. But – and this is the paradox and origin of the controversy – the picture can only be seen complete when viewed from both sides of the canvas at the same time. However, in the process, grammar is absorbed into and bisected by the canvas, which is the margin – the margin preserved and transcended.

M¦M
A¦A
STRUCTURE R¦R FUNCTION
G¦G
I¦I
N¦N
References

Baijnath. N. 1991. Problems with Process: tracing writing problems to their roots. Paper presented at the South African Applied Linguistics (SAALA) Conference, University of the Witwatersrand, July 1991.

Brown, G. 1987. Twenty-five years of teaching listening comprehension. English Teaching Forum, 25(4):11-15. Chomsky, N. 1957. Syntactic structures. The Hague. Mouton.

Coe, R.M. 1987. An apology for form; or, who took the form out of process. College English, 49:13-28.

Corder, S.P. 1981. Error analysis and interlanguage. Oxford. Oxford University Press. Cummins, J. 1984. Wanted:A theoretical framework for relating language proficiency to academic achievement among bilingual students. In Rivera, C. Language proficiency and academic achievement. Multilingual Matters 10. Clevedon. Multilingual Matters Ltd.

Du Toit, A. and Orr, M. 1987. Achiever’s Handbook. Johannesburg. Southern Book Publishers.

Gardner, M. 1982. The Ambidextrous universe:mirror asymmetry and time-reversed worlds. Middlesex. Penguin

Jeffery, C.D. 1990. The case for grammar: Opening it wider. South African Journal of Higher Education (SAJHE), Special edition.

Johnson, B. 1985. Taking fidelity philosophically. In Graham, J.F. (ed.). Difference in translation.

Krashen, S. 1981. Second language acquisition and second language learning. Oxford. Pergamon Press.

Kroma, S.K. 1988. Action research in teaching composition. English Teaching Forum, 26(1).

Johnson, B. 1985. Taking fidelity philosophically. In Graham, J.F. (ed.). Difference in translation.

Langacker, R. 1987. Foundations of cognitive grammar, Vol. 1. Theoretical prerequisites. Stanford. Stanford University Press.

Langacker, R. 1991. The conceptual basis of grammatical structure. Paper presented at the “Language, thought and culture” conference at the University of the Witwatersrand, Johannesburg.

Leech, L.G. 1983. Principles of pragmatics.London. Longman.

Murray, S. 1990. Teaching English for academic purposes (EAP) at the University of Bophuthatswana. South African Journal of Higher Education (SAJHE), Special edition.

Newmeyer, F. J. 1983. Grammatical theory: Its limits and its possibilities. Chicago. University of Chicago Press.

Newmeyer, F. J. 1986. The politics of linguistics. Chicago. University of Chicago Press.

O’Connell, D.C. 1988. Critical essays on language use and psychology. New York. Springer-Verlag.

Ricoeur, P. 1974. Structure, word, event. In The conflict of interpretations. Evanston, Illinois. North Western University Press.

Ridjanovic, M. 1983. How to learn a language, say English, in a couple of months. English Teaching Forum, 21(1).

Saussure, F. 1959. Course in general linguistics, (Tr. Wade Baskin). London. Peter Owen.

Taylor, D.S. 1988. The meaning and use of the term “competence” in linguistics and applied linguistics. Applied Linguistics, 9(2):148-168.

Wells, G. 1987. The meaning makers. London. Hodder Stoughton.

UNISA. 1988. Practical English Guide. Pretoria. UNISA

Widdowson, H.G. 1979. Explorations in applied linguistics. Oxford. Oxford University Press.

Widdowson, H.G. 1989. Knowledge of language and ability of use. Applied Linguistics, 10(2):128-137.

The 5th Annual English Language Teaching  Conference, 25th March , 2004

Abstract

Reliability is concerned with how we measure; validity is concerned with what we are supposed to measure, i.e. with the purposes of a test. In this paper, I concentrate on reliability; however, it is often difficult to discuss reliability without bringing validity into the picture.

The reliability of a test is concerned with the accuracy of scoring and the accuracy of the administration procedures of the test.

In this paper the following key aspects of reliability are dealt with

- Facets: These refer to such factors as the (1) testing environment, e.g. the time of testing and the test setting, (2) test organisation, e.g. the sequence in which different questions are presented, and (3) the relative importance of different questions and topic content. Facets also include cultural differences between test takers, the attitude of the test taker, and whether the tester does such things as point out the importance of the test for a test-taker’s future.
- Features or conditions: These refer to such factors as clear instructions, unambiguous questions and items that do or do not permit guessing.
- The manner in which the test is scored. A central factor in this regard is rater reliability, i.e. the agreement between raters on scores and judgements related to a particular test.  Rater reliability (consistency) becomes a problem mainly in the kind of tests that involve subjective judgements such as essay tests.

The bulk of the paper will focus on rater reliability.

A major obstacle in test development has been the lack of agreement on what it means to know a language, which affects

WHAT aspects of language knowledge are tested; and
HOW these aspects are tested.

Validity and reliability are closely connected. Validity has to do with WHAT you are measuring, reliability with HOW. The WHAT of testing (validity) deals with the knowledge and skills that we are testing/measuring.

The HOW in reliability refers to THREE main things.

The first two are intrinsic to the 1. form(s) of test and to the 2. conditions under which it is taken (intrinsic reliability); the third is extrinsic to the test and to the test conditions.

The form of the test. The are several things to consider here:

clear instructions, unambiguous questions and items that do not permit guessing.
the sequence in which different questions are presented
the relative importance of different questions
if you give two different forms of a test to two different individuals, or groups, both forms of the test should be testing the same things and be of the same degree of difficulty.
.
Testing conditions. A few examples:
-    testing environment, e.g. the time of testing and the test setting
if test takers in one test group are allowed to talk to one another during a test, and another group is not allowed to do so, this will affect the reliability of the test.

Rater (examiner) reliability.
The manner in which the test is scored. A central factor in this regard is rater consistency (reliability), i.e. the agreement between raters on scores and judgements related to a particular test.  Rater consistency becomes a problem mainly in the kind of tests that involve subjective judgements such as essay tests.

In the rest of this paper, I focus on rater reliability.

__________________________________

Scores can only be reliable, if you know WHAT you are measuring/ marking

But, if you know what you are measuring, it does not follow that you will automatically measure consistently/accurately/reliably.

So, the teacher has to know his/her subject well, but must also know the problems in assessing a student’s work.

Rater reliability is EXTRINSIC (external) to the test itself, because it doesn’t deal with things such as the forms of the test or the conditions under which they are done. It deals with the way tests are marked.

There are two kinds of rater reliability:

Intrarater reliability and
Interrater reliability.

In intrarater reliability we aim for consistency within the rater. For example, if a rater (teacher) has many exam papers to mark and doesn’t have enough time to mark them, he or she might take much more care with the first, say, ten papers, than the rest. This inconsistency will affect the students’ scores; the first ten might get higher scores.

Interrater reliability

Earlier I mentioned the pressure of time a rater (teacher) may have when marking. There are many other kinds of pressures that teachers may have to endure in the way they mark exams and tests. One of these may be the need to equip as many people as possible to join the real world outside of school.

Besides the pressures that teachers may have to endure in their testing procedures, there are other factors that affect the ability of a teacher to be OBJECTIVE, i.e. to see things as they are, rather than to be SUBJECTIVE, i.e. to see things as one wants/needs them to be.
subjective than the natural sciences. It is much easier for teachers to agree that water boils at zero degrees centigrade at sea level than that a student in “C” level is stupid or cheeky.

It is also much easier to agree on the correctness of a grammar item (objective part) than on the quality of a composition (subjective whole).

Language is meant for real-life, i.e. for communication.
Examples of communication are:
1. Writing a letter, or a composition;
2. Speaking to or listening to someone;
3. Reading a book or a newspaper.

Grammar is not communication; it is the basic building blocks of communication.
It is easier to accurately measure grammatical knowledge (grammatical competence) and harder to measure language use (communicative competence).

To return to our two terms OBJECTIVE and SUBJECTIVE, grammatical competence is more objective, while communicative competence is more subjective.

For example, most grammar items are either right or wrong, whereas in a composition, it is much more difficult to make judgements about large chunks of language such as a composition. There is often a wide range in judgements and scores between different teachers (raters).

This means that grammar scores are often accurate, i.e. reliable, but that composition scores are often inaccurate, i.e. unreliable.

To sum up

The greater the validity of a test, the more it reflects language use. Language use is more difficult to mark than grammar. The greater the difficulty to mark, the less the reliability.

Therefore

Grammar tests are more reliable but less valid

AND

Composition tests are less reliable but more valid.

We must be careful when we say that grammar scores are much more reliable (accurate) than composition scores.

If the test is a multiple choice test where the answers are provided by the test maker, teachers do not need to decide on the accuracy of the grammar item. So, scoring will be more reliable.

However, where the answers to grammar items are not provided, marking can be unreliable. It depends on the grammatical knowledge of the teacher.

To repeat what was said earlier

Reliability = Accuracy of HOW we assess/test

Validity = WHAT we assess/test

MORE reliability
=
LESS validity

The easier it is to measure something the less complex it is. Language use is more complex than grammar; emotions are more complex than motions.

Composition writing is much more difficult to mark than grammar items because composition is much more authentic/real-life than grammar.  In composition we need to look at things like cohesion and coherence, which has to do with how sentences and parts of sentences come together to create paragraphs and longer stretches of discourse.

For this reason, composition writing (real-life language, i.e. language use) gives one a better understanding of a person’s language competence.

Reliability in composition writing is a more fruitful field of study than isolated grammar items, because in composition we find all the three important components of language use, namely:

Content 2. Organisation. 3. Language

Conclusions and Recommendations

The problem of rater reliability is how to be as fair as possible in the allocation of scores and judgements. The problem of rater reliability in assessment seems to take precedence over all other issues in testing. This is understandable because assessment is the last and most crucial stage in the teaching programme. (Most learners only protest about poor teaching, unclear exam questions, etc. if they fail).

Rrecommendations

At the beginning of an academic year, all the teachers in the department can rate one student’s assignment and discuss the criteria they used and the marks they awarded. This exercise done repeatedly over a period of time gradually increases interrater reliability.

2. Before a major test or an exam, questions set by individual teachers can (indeed must) be discussed (formally and/or informally) by the whole department in terms of:

- clarity of the questions
- length of questions, and
- number of marks to be awarded for specific questions or sections.

3. The person who prepares the question should also give a memorandum to others in the department to demonstrate the criteria by which answers will be evaluated.

If one had to follow these procedures one would hope that objectivity would be increased. Yet, it seems that even if one does consult with colleagues (as would be the case with literary or music or culinary critics) hard problems remain. What is worrisome in the assessment of written output is that in spite of discussions and workshops on establishing common criteria, there remain large differences in the relative weight raters attach to the different criteria, e.g. linguistic structure, content and organisation.

But one should not stop trying to improve reliability, just as one should not stop trying to be as objective as possible in all the other facets of our lives, especially in our
judgements of others.

Language, thought and culture

Conceptual similarities and differences between cultures

Interpretation

Is social thought a science?

The genetic and social origin of beliefs (about reality)

Conclusion

Bibliography

Introduction

“Every man, where he goes, is encompassed by a cloud of comforting convictions, which move with him like flies on a summer’s day” (Bertrand Russell, 1938). The cloud of comforting convictions that encompassed Bertrand Russell – the pacifist sceptic and one of the great philosopher’s of the twentieth century – shifted across the same sky as the mystic’s cloud of unknowing that seeks penetration into the mind of God. Russell and the mystic – and God in the eyes of the mystic – have a mind, are both persons, and both exist, i.e. occupy a “world”. Yet all three – presumably – have radically different visions, interests – and perhaps different emotional states (Sowell. 1988). The trinity of mind-person-world, is a mysterious configuration of fragmentary abstractions, without which knowledge is impossible, and with which ignorance is probable. Science in humans is a product of vision – in the case of God, of provision. The vision is the view. What is a vision, a vision of mind-person-world? How does the mind-person, who is implicated in the order of the world, construct a view of the world?

Language, thought and culture

A point of view involves three constituents: language, thought and knowledge. It is not clear which shapes which, and how much the one shapes the other. In a sense, a point of view reveals a particular culture. By culture I understand

that level at which social groups [or individuals] develop distinct patterns of life, and give expressive form to their social and material life-experience. Culture is the way, the forms, in which groups [or individuals] `handle’ the raw material of their social and material existence…Culture is the distinctive shapes in which this material and social organization expresses itself.

(Clark, Hall, Jefferson, & Roberts, 1975:10).

Are thoughts language-relative (Humboldt, 1963; Whorf, 1956)? Or are they universal?; which would mean that they are only superficially modified by the differences between languages. Is “reality” an objective entity or is it a construction: a linguistic or a social construction? What is the relationship between the individual, society-culture and (the physical) environment with regard to language, thought and knowledge? Whorf’s well-known example comes to mind of the Hopi Indian’s view of time and space being determined by the structure of the Hopi verb system. This example, even if it is claimed to be slender evidence for linguistic relativism, would be, according to Code (1980:251) strong evidence for linguistic relativism, because the way one views time and space has a determining effect on the rest of one’s experience . Pinker’s view (1995:63) is that Whorf’s “outlandish claims” is a product of bad analysis and “leanings towards mysticism”.

According to Jerison (1986:8), the early evolution of language had little to do with communication, because it is not communication that requires a large brain, but thinking about what to communicate. This is not only true of early man (both genders), but of modern man as well. This fact is not well understood in the language teaching profession where the main emphasis is on communication – the product. The product, however, is only the final stage of a long process of thinking about what one wants to communicate.

Conceptual similarities and differences between cultures

The higher one climbs up the conceptual ladder of abstraction, that is,  the more one moves way from bread-and-butter thoughts, the greater the difference between conceptual systems. These differences may exist not only between groups or individuals sharing very different languages (linguistic codes), but more importantly, also between groups or individuals sharing the same languages. Groups may be of a sociocultural nature or an interest-group such as a profession, e.g. science. But there must also exist conceptual commonalities between different languages as diverse as English and Japanese and Gujarati. If this were not so, there would be no explanation for the fact that the sciences are highly developed in America, Japan and India. The language of the sciences cross many cultural, environmental and linguistic boundaries.

Interpretation

Interpretation occupies a large part of mind activity. In any interpretation there are two key moments that comprise its equilibrium: Who/what is being interpreted (the object) and who is doing the interpretation (the subject). An important empirical problem is establishing valid and reliable criteria of interpretation. Interpreters (i.e. researchers and evaluators) are by definition subjective because they are time-space/relational beings concretised in a diversity of sociobiological configurations, which mean that they (can only) see from a particular point of view. One persuades another of one’s point of view, one negotiates one’s point of view, one creates one’s point of view, one reflects on one’s point of view, one resists another’s point of view. Everybody who listens, speaks, reads or writes, filters everything through a point of view, through a theoretical perspective. To this perspective the scientist adds a methodological orientation based on the chosen theoretical perspective

Is social thought a science?

Among scientists there are many different and theoretical perspectives and methodological orientations, which produce – as in the arts and religion – a multiplicity of expressive systems (“discourses”). By discourses I mean “ideologically determined ways of talking or writing about persons, places, events or phenomena.” (Wallace, 1992:68). Each person’s discourse is the product of a variety of different beliefs and experiences, and owing to our human nature “we should not be surprised to find that each side necessarily misreads and misrepresents the other side in order to reconstitute its own position” (Blundell, Shepherd & Taylor, 1993:26). Blundell et al. are speaking in the context of cultural studies, but the implied reference is to researchers in general, who all work within a cultural framework.

An important question in the social/human sciences is whether social science is a science. Choosing a scientific point of view – whether in, say, physics (a “hard ” science) or in, say, sociology ( a “human” science) may have less to do with science than with philosophy: the philosophy of beliefs, values and social practices (i.e. a world view) rather than with philosophical theories of knowledge and existence. Whether the domain of study be art, religion, physical science, psychology, social work or linguistics, it is the “paradigm [that] determines the identification and interpretation of “empirical evidence” [or of any other kind of evidence or experience] in a given discipline (Bizzell, 1979:764; my square brackets).

In the “Standard Social Science Model” (Pinker, 1995:23) the “human psyche [of which language is a part - brackets added] is molded by the surrounding culture.” This “soviet” approach became popular in America and Britain through the translations of writers such as Vygotsy and Luria, whose social practice is referred to in the “West” as a “social constructivist perspective” that envisages individual’s abilities not as a product of individual cognitive processes, but as a construction resulting from an individual’s interaction and immersion in society.

One’s penchant towards biological forces (Piaget and Chomsky) or social forces (Vygotsky) as the primary influence in social practice has far-reaching effects on the rest of one’s Weltensicht. The “social construction of reality” view is not at all solely of soviet inspiration . The West has always had its own clutch of Deweys and Durkheims.

The genetic and social origin of beliefs (about reality)

We are born with many of our beliefs, because they are one of the varieties of “cognitive adaptations” (Jerison, 1986:7). And,

(h)uman beings have all sorts of beliefs [whether they be physicists, philosophers. applied linguistics; to mention only the academic domain]. The way in which they arrive at them varies from reasoned argument to blind faith. Some beliefs are based on personal experience, others on education, and others on indoctrinations. Many beliefs are no doubt innate: we are born with them as a result of evolutionary factors. Some beliefs we feel we can justify, others we hold because of “gut feelings”.

Davies (1993:19; my brackets)

Pinker (1995:128) puts even more emphasis on the genetic origin of beliefs:

Not only are very general traits like IQ, extroversion, and neuroticism partly heritable, but so are specific ones like degree of religious feeling, vocational interests, and opinions about death penalty, disarmament, and computer music.

The aim of cultural studies, applied psychology and education is to change people – for the better; thus the kind of theoretical perspective and methodological orientation is often chosen in terms of the changes one would like to see. It remains a valid principle in science that one should always be aware of the limitations of one’s own paradigm. Bizzell (1979:766) points out that Kuhn is “acutely alive to the limitations imposed by a paradigm, limitations that may not only restrict free inquiry, but vitiate the social usefulness of the discipline.” In Kuhn’s words,

one of the things a scientific community acquires with a paradigm is a criterion for choosing problems that, while the paradigm is taken for granted, can be assumed to have solutions. To a great extent these are the only problems that the community will admit as scientific or encourage its members to undertake. Other problems, including many that had previously been standard are rejected as metaphysical, as the concern of another discipline, or sometimes as just too problematic to be worth the time…One of the reasons why normal science seems to progress so rapidly is that its practitioners concentrate on problems that only their own lack of ingenuity should keep them from solving.

(Kuhn, 1970:37, quoted in Bizzell, 1979:766)

The theory and data of the social sciences are basically concerned with the person-in-the-world. Trying to understand the nature of the relationship between the world and the person is “one of the more exasperating and contentious of all humanistic concerns, [namely] the proper nature of the relationship between the individual and society (or the state, culture, or community)” (Cuzzort and King, 1995:129). It is the person’s mind, specifically mental- processes- in-the-world, that I want to focus on now.

Luria (1976:3) emphasises the social and historical origin of mental processes: “It seems surprising that the science of psychology has avoided the idea that mental processes are social and historical in origin. And Vygotsky (1978:57) in a similar vein: “The internalisation of socially rooted and historically developed activities is the distinguishing feature of human psychology, the basis of the qualitative leap from animal to human psychology.”

Bhaskar (1979:45-46) attributes an even more peremptory role to society:

The model of the society/person connection I am proposing could be summarised as follows: People do not create society. For it always preexists them and is a necessary condition for their activity. Rather, society must be regarded as an ensemble of structures, practices and conventions which individuals reproduce and transform, but which would not exist unless they did so.

Bhaskar’s individual is having his society and eating it: on the one hand, society always preexists the individual, but society would not exist if humans did not reproduce and transform it. This seems to mean that a thing (society) only exists if it is reproduced and transformed, but the thing (the individual) that transforms and produces it always post-exists it. This is at best a conundrum.

In contrast to Luria, Vygotsky and Bhaskar, Popper’s (1965) “natural science” view is that there “is no history of mankind, there is only an indefinite number of histories of all kinds of aspects of human life” i.e. a fertile plurality of self-constituted principles. Popper’s natural science point of view is also a theoretical construction existing within the fertile plurality of other theoretical constructions such as those of historical meanings. This means that there can be no self-constituted principle, because all principles (or hypotheses) are rooted in the contingency of history. But this does not imply that society preexists the individual, but only that society is produced, reproduced and transformed by individuals and that individuals are, in turn, transformed by society into “social actors” (Miller, 1984:12-14).

In contemporary sociology, linguistics and education in the English-speaking world, there is an inordinate emphasis on social development, which consequently results in the neglect of the natural (i.e. biomental) realities of human development (Bickerton, 1981, 1990; Brown, 1991) Biomental mechanisms – I am thinking specifically of language and thought – are not only the roots of discourse; these biomental mechanisms ramify into the very branches of discourse. Thus it would be incorrect to maintain that “the use of signs leads humans to a specific structure of behaviour that breaks away from biological development and creates new forms of a culturally-based psychological process” (Vygotsky, 1978). The reality of higher cognitive processes is much more than social in origin. To adumbrate:

Reality cannot be reduced to (academic) disciplines. A panoramic picture of realities is provided by William James’ “sub-universes” (Bolton 1977:43):

1. The world of sense, of physical things, as we apprehend them.

2. The world of science, of physical things, as the learned conceive them.

3. The world of ideal relations and abstract truths believable by all – logical mathematical, ethical, metaphysical propositions.

4. The world of “idols of the tribe”, illusions or prejudices common to all.

5. The various supernatural worlds.

6. The various worlds of individual opinion.

7. The various (and numerous) worlds of “sheer madness”.

Each person creates his/her own picture of “reality”, which is the closest one can get to reality. In any one of the seven sub-universes above (and obviously in the seventh one) can be found “sheer madness”, depending on the “perspective” of the viewer who usually claims to be sane. (Lee, 1992:198) uses Discourse with a capital D and “perspective” interchangeably.

These multitudinous realities, because they are sown into the fibre of our being are “concrete”. Thus there exist a “multitude of concrete worlds” within an “abstractly unitary national language.” Bakhtin (1981:288). What we should know about these “concrete worlds” is more than the social, cultural and linguistic forces that play a role in fashioning them. It is also the relation between the “biological bases of behaviour and the social conditions in and through which human activity takes place” (Cole et al. in Vygotsky 1978:124) that should interest us as well. We should know about “mind in society” (Vygotsky 1978), but also about society in mind; or more correctly – owing to our ignorance of what is brain, what is mind – the brainmind in society and society in the brainmind.

Scientists search for (ever deeper) causes. All causes involve a transfer of energy. Thus science is regressionist and reductionist by nature, for there is no other way to know anything unless to analyse it, i.e. break it up (or down) into simpler bits. On the other hand, it is impossible to know how human beings function unless we also take into account sociohistorical influences. However, as Boyden (1987:v) points out, “a serious flaw in the historical approach [of human society, e.g. Toynbee, Marx]…lies in the fact that it takes far too little account of the biological components of the societal system with which it is concerned.” (My square brackets).

The relationship between genes and culture/history is not only extremely interesting, it is of central importance in the human sciences. The conclusions we reach on this matter and act upon, will have profound implications. Those who are opposed to the emphasis on biology in education and development can take comfort in the fact that human reasoning is limited. And thus it is safe to maintain that biology’s preeminent role in human culture is a matter of conjecture. Nevertheless, even though

it has not been possible to solve the more interesting empirical questions definitively: Is an explicit theory of the mechanisms of cultural evolution necessary? And, if so, what exactly should it be like? However, a deductive theory which clearly sets out the logical relationships between and cultural evolution does advance out understanding of such problems by more clearly showing what is at issue in controversies, such as those between human sociobiologists and their critics, which otherwise appear to be the result of irreconcilable “philosophical” differences.

(Boyd & Richerson, 1985:18)

Conclusion

Ultimately, choosing a specific conception of language development or of “reality”, specifically of the individual in society, is to a large extent a philosophical exercise which is based on value judgments. “Models of psychological reality cannot be proven right or wrong, correct or incorrect, by any objective criterion.” (Hillner, 1985). Some rational decisions have to be made, but these decisions are enmeshed in “all sorts of subjective and idiosyncratic considerations” (Hillner, 1985). Thus it is questionable whether “[t]here is only one way of seeing one’s own spectacles clearly: that is to take them off. It is impossible to focus both on them and through them at the same time” ( Campbell, 1985:19-20). It would be difficult, if not impossible, to separate the spectacles from the eye, which is why “I” see through a glass darkly, i.e. there is much subjectivity in science, of the “human” as well as of the “hard” kind. Theories are subjective expressions., and often it is the data that are driven by the theory rather than the other way round. To establish anything empirically without a theory to “drive” the data seems impossible. Theory , although tempered by the data, remains underdetermined by the data.

Cognitive psychologists theorise about “the events that causally mediate the production of intelligent behaviour” (Fodor 1975:9). These theories may be wrong, or it is possible that mental processes that mediate the production of behaviour are too complicated to understand. If we do carry on theorising in the hope of finding answers then mental events should not be reduced to behaviourial events. This is not to deny that mental events may turn out to be physiological events. The problem is finding a sophisticated enough vocabulary of physiology to explain these physiological events. Thus the ontological status of mental states could be physiological, but the epistemological status would have to be more, because another vocabulary is required to explain these events. Would the vocabulary of psychology be sufficient to represent our concepts? Concept formation feeds upon different kinds of realities (fear, love, enjoyment, knowledge, the struggle for survival). The problem is how to view and accordingly live these realities as a coherent happy whole.

Bibliography

Bakhtin, M.M. 1981. The dialogic imagination. Austin. University of Texas Press.

Bhaskar, R. 1979. The possibility of naturalism. Brighton: Harvester.

Bizzell, P. 1979. College English, 40(7):764-771.

Bickerton, D. 1981. Roots of language. Ann Arbor: Karoma Publishers, Inc.

Bickerton, D. 1990. Language & species. Chicago: University of Chicago Press.

Boyd, R. and Richerson, P.J. 1985. Culture and the evolutionary process. Chicago. University of Chicago Press.

Boyden, S. 1987. Western civilization in a biological perspective. Oxford: Clarendon Press.

Blundell, V., Shepherd, J., Taylor, I. 1993. Relocating cultural studies: Developments in theory and rsearch. PLACE: PUBLISHER

Bolton, N. 1977. Concept formation. Oxford. Pergamon Press.

Brown, D.E. 1991.Human universals. Philadelphia. Temple University Press.

Campbell, C.M. 1985.Learning and development. An investigation of a neo-Piagetian theory of cognitive growth. Unpublished Master of Arts thesis, University of Natal, Durban.

Clark, J., Hall, S., Jefferson, T. and Roberts, B. 1975. Subcultures, culture and class: a theoretical overview . Working Papers in Cultural Studies, 7/8:9-74.]

Code, L. 1980. Language and knowledge. Word, 31(3):245-58.

Cuzzort, R.P. and King. 1995. Life as a con game: The dramatic vision of Erving Goffman. In Cuzzort, R.P. and King, E.W. 1995. Twentieth-Century social thought (5th Edition): Fort Worth: Harcourt Brace College Publishers.

Davies, P. 1993. The mind of God: Science and the search for ultimate meaning. London. Penguin Books.

Fodor, J.A. 1975. The language of thought. Hassocks, Sussex. The Harvester press.

Hillner. K. 1985. Psychological reality. Amsterdam: North-Holland.

Humboldt, W. von. 1963. Man’s intrinsic humanity: His language. In Humboldt, W. von. Humanist without portfolio: An Anthology of the writings of Wlihelm von Humboldt. Tr. Marianne Cowan. Detrot: Wayne State University Press.

Jensen, A.R.1972.Genetics and education. London: Methuen & Co Ltd.

Jensen, A.R. 1973. Educational differences. London: Methuen.

Jerison, H.J. (ed.). 1986. Evolutionary biology of intelligence:The nature of the problem’ in Jerison, H.J., and Jerison, I. (ed.). Intelligence and evolutionary biology. New York. Springer- Verlag.

Kuhn, T.S. 1970. The structure of scientific revolutions. 2nd Edition. Chicago. Unviersity of Chicago Press.

Lee, D. 1992. Competing discourses: Perspective and ideology in language. London. Longman.

Luria, A.R. 1976. Cognitive development: Its cultural and social foundations. Cambridge: Harvard University Press.

Miller, R. 1984. Reflections of mind and culture. Pietermaritzburg, University of Natal Press.

Pinker, S. 1995. The language instinct. London/New York. Penguin Books.

Popper. K. 1965. The open society and its enemies. Volume 2. London: Routledge and Kegan Paul.

Russell, B. 1938. Sceptical essays. New York: W.W. Norton & Company, Inc.

Sowell, T. 1988. A conflict of visions. New Delhi: Affiliated East-West Press Pvt Ltd.

Vygotsky, L.S. 1978. Mind in society: The development of higher psychological processes. (Edited by Michael Cole, Vera John-Steiner, Sylvia Scribner and Ellen Souberman). Cambridge, Massachusetts: Harvard University Press.

Wallace, C. 1992. Critical literacy awareness in the EFL classroom. In: Fairclough, N. (ed.). Critical language awareness. London: Longman.

Whorf, B.L. 1956. Language, thought and reality (edited by Carroll, J.B.). New York: Wiley.

Abstract

1. Introduction

2. Choosing a point of view

3. Differences and universals

4. Origin of mental processes and their impact on education

5. Lenneberg’s biological view of language development

6. Conclusion

References

Abstract

A conflict of views on human development seems to exist between the natural sciences (e.g. genetics, neuroscience) and the social sciences. Although there is much that is useful in Luria’s, Bhaskar’s, and Vygotsky’s emphasis on the sociohistorical dimension in mental development, there seems to be a lack of appreciation of the important role of biogenetic factors in mental development . Indeed, these mental processes – as these authors affirm – do transform and are transformed by society/history through activity, and no doubt the Marxist position that society (more aptly, the state) shapes individual and social consciousness carries a lot of truth. But a more balanced view of the research into human problems of education and development is that the historical, extrinsic, chain of events are also shaped by inborn, intrinsic capacities. I tend towards the view of Chomsky that society and the environment act as triggers, i.e. “the experience does not determine how the mind will work but it triggers it, makes it work in its own largely predetermined way” (Chomsky, 1988:172). Such a view is euphemistically unpopular, because it destroys hope – that is built on view that educational/social intervention can transform a poor learner into a good learner.

1. Introduction

“Every man [generically speaking], where he goes, is encompassed by a cloud of comforting convictions, which move with him like flies on a summer’s day” (Russell, 1938). The cloud of comforting convictions that encompassed Bertrand Russell – the pacifist sceptic and one of the great philosopher’s of the twentieth century – shifted across the same sky as the mystic’s cloud of unknowing that seeks penetration into the mind of God. Russell and the mystic have a mind, are both persons, and both exist, i.e. occupy a “world”. Yet both have radically different visions, interests and emotional states (Sowell. 1987). The prime function of language – as a faculty of the brainmind – is to automatically categorise these visions, interests and emotional states, and the prime motivation of the user of language is to express and then to communicate these categorisations (Ellis, 1994). These categorisations are a mysterious configuration of fragmentary abstraction, without which knowledge is not possible, and with which ignorance is probable. This configuration of abstractions is what we call a point of view.

The higher one climbs up the ladder of abstraction, i.e. the more one moves way from bread-and-butter conceptions, the greater the difference between conceptual systems. These differences may exist not only between groups or individuals sharing very different languages (linguistic codes), but, extremely importantly, also between groups or individuals sharing the same languages. Groups may be of a sociocultural nature or an interest-group, such as a profession, e.g. science.

One persuades another of one’s point of view, one negotiates one’s point of view, one creates one’s point of view, one reflects on one’s point of view, one resists another’s point of view. Everybody who listens, speaks, reads or writes, filters everything through a point of view, through a theoretical perspective. To this perspective the scientist adds a methodological orientation based on the chosen theoretical perspective .

Among scientists there are many different and theoretical perspectives and methodological orientations, which produce – as in the arts and religion – a multiplicity of expressive systems or discourses, i.e. “ideologically determined ways of talking or writing about persons, places, events or phenomena.” (Wallace, 1992:68). Each person’s discourse is the product of a variety of different beliefs and experiences, and owing to our human nature “we should not be surprised to find that each side necessarily misreads and misrepresents the other side in order to reconstitute its own position” (Blundell, Shepherd & Taylor, 1993:26). Blundell et al. are speaking in the context of cultural studies, but the implied reference is to researchers in general, who all work within a cultural framework. By culture I understand

that level at which social groups [or individuals] develop distinct patterns of life, and give expressive form to their social and material life-experience. Culture is the way, the forms, in which groups [or individuals] `handle’ the raw material of their social and material existence…Culture is the distinctive shapes in which this material and social organization expresses itself.

(Clark, Hall, Jefferson, & Roberts, 1975:10).

2. Choosing a point of view

Choosing a scientific point of view – whether in, say, physics (a “hard ” science) or in, say, sociology ( a “human” science) has probably got less to do with science than with philosophy; the philosophy of beliefs, values and social practices (i.e. a world view) rather than with philosophical theories of knowledge and existence. Whether the domain of study be art, religion, physical science, psychology, social work or linguistics, it is the “paradigm[that] determines the identification and interpretation of “empirical evidence” [or of any other kind of evidence or experience] in a given discipline (Bizzell, 1979:764; my square brackets). World views usually conflict with one another. Thus what one person may consider to be his or her world view, another may label as an ideology, where the will to dominate may be seen as the mainspring of the former’s world view:

Human beings have all sorts of beliefs [whether they be physicists, philosophers. applied linguistics; to mention only the academic domain]. The way in which they arrive at them varies from reasoned argument to blind faith. Some beliefs are based on personal experience, others on education, and others on indoctrinations. Many beliefs are no doubt innate: we are born with them as a result of evolutionary factors. Some beliefs we feel we can justify, others we hold because of “gut feelings”.

Davies (1993:19; my brackets)

The aim of cultural studies, applied psychology and education is to change people – for the better; thus the kind of theoretical perspective and methodological orientation is often chosen in terms of the changes one would like to see.

3. Differences and universals

An educational psychologist, unlike a a philosopher or a “pure” scientist, is not primarily looking for universals in human structure and development, but for differences. For the linguistic scientist Pinker, “differences between individuals are so boring” (Pinker, 1995:428; original italics), whereas Carroll’s survey of research into human cognitive abilities was written with the intention of serving as a “reference work and as a textbook in advanced courses in individual differences” (1993:vii).

An evaluation of human abilities is necessarily concerned with differences. However, it is not possible to understand differences without understanding universals, i.e. the commonalities between human beings.

An educationist is primarily concerned with differences related to learning, and the concomitant problem of failure. If there were no problems there would be no teachers, because children would not need adults to lead them to success and adulthood. Educationists, like scientists, and unlike politicians, seek enlightenment concerning their limitations. It remains a valid principle in science that one should always be aware of the limitations of one’s own paradigm. Bizzell (1979:766) points out that Kuhn is “acutely alive to the limitations imposed by a paradigm, limitations that may not only restrict free enquiry, but vitiate the social usefulness of the discipline.” In Kuhn’s words,

one of the things a scientific community acquires with a paradigm is a criterion for choosing problems that, while the paradigm is taken for granted, can be assumed to have solutions. To a great extent these are the only problems that the community will admit as scientific or encourage its members to undertake. Other problems, including many that had previously been standard are rejected as metaphysical, as the concern of another discipline, or sometimes as just too problematic to be worth the time…One of the reasons why normal science seems to progress so rapidly is that its practitioners concentrate on problems that only their own lack of ingenuity should keep them from solving.

(Kuhn, 1970:37, quoted in Bizzell, 1979:766)

Ultimately, choosing a specific conception of development or of “reality”, specifically of the individual in society, is to a large extent a philosophical exercise which is based on value judgements. “Models of psychological reality cannot be proven right or wrong, correct or incorrect, by any objective criterion.” (Hillner, 1985). Some rational decisions have to be made, but these decisions are enmeshed in “all sorts of subjective and idiosyncratic considerations” (Hillner, 1985). Thus it is questionable whether “[t]here is only one way of seeing one’s own spectacles clearly: that is to take them off” ( Campbell, 1985:19-20). It would be difficult, if not impossible, to separate the spectacles from the eye, which is why “I” see through a glass darkly. There is much subjectivity in science, of the “human” as well as of the “hard” kind. Theories are subjective expressions, where the data are often driven by the theory than the other way round. To establish anything empirically without a theory to “drive” the data seems impossible. Theory , although tempered by the data, remains underdetermined by the data.

The theory and data of the social sciences are basically concerned with the person-in-the-world. Trying to understand the nature of the relationship between the person and the world is “one of the more exasperating and contentious of all humanistic concerns, [namely] the proper nature of the relationship between the individual and society (or the state, culture, or community)” (Cuzzort and King, 1995:129). It is the person’s mental processes-in-the-world, and how they impact on education that I want to focus on.

4. Origin of mental processes and their impact on education

Luria (1976:3) emphasises the social and historical origin of mental processes: “It seems surprising that the science of psychology has avoided the idea that mental processes are social and historical in origin. And Vygotsky (1978:57) in a similar vein. “The internalisation of socially rooted and historically developed activities is the distinguishing feature of human psychology, the basis of the qualitative leap from animal to human psychology.” Bhaskar (1979:45-46) attributes an even more imperious role to society.

The model of the society/person connection I am proposing could be summarised as follows: People do not create society. For it always pre-exists them and is a necessary condition for their activity. Rather, society must be regarded as an ensemble of structures, practices and conventions which individuals reproduce and transform, but which would not exist unless they did so.

In contrast to Luria, Vygotsky and Bhaskar, Popper’s (1965) “natural science” view is that there “is no history of mankind, there is only an indefinite number of histories of all kinds of aspects of human life” i.e. a fertile plurality of self-constituted principles. I don’t agree entirely with Popper’s view, because the natural science point of view is also a theoretical construction existing within the fertile plurality of other theoretical constructions such as those of historical meanings (Rauche, 1992:470). This means that there can be no self-constituted principle, because all principles (or hypotheses) are rooted in the contingency of history. But this does not imply that society pre-exists the individual, but only that society is produced, reproduced and transformed by individuals and that individuals are, in turn, transformed by society into “social actors” (Miller, 1984:12-14).

For Luria, Vygotsky and Bhaskar, all phenomena are processes in motion and in change. Changes in society bring about changes in the individual. The development of the individual is brought about by society and culture. Human “development” for Vygotsky is mainly a social process, and human nature mainly a social construction:

Within a general process of development, two qualitatively different lines of development, differing in origin, can be distinguished: the elementary processes, which are of biological origin, on the one hand, and the higher psychological functions, of sociocultural origin, on the other. The history of child behaviour is born from the interweaving of these two lines. The history of the development of the higher psychological functions is impossible without a study of their prehistory, their biological roots, and their organic disposition.

(Vygotsky 1978:46; original emphasis and my underlining).

Vygotsky assigns the elementary psychological functions of language to the biological level, to the “roots” of behaviour, to the intrapersonal; but the higher psychological functions of language to the social level. With regard to cultural development, every psychological function appears twice:

first, on the social level, and later, on the individual level; first, between people (interpsychological), and then inside the child (interpsychological). This applies equally to voluntary attention, to logical memory, and to the formation of concepts. All the higher functions originate as actual relations between human individuals…The internalisation of socially rooted and historically developed activities is the distinguishing feature of human psychology, the basis of the qualitative leap from animal to human psychology. As yet, the barest outline of this process is known.

(Vygotsky 1978:57)

Thus, Vygotsky assigns biological (intrapersonal) properties to the “roots” of the tree, and social properties to the tree (itself). Vygotsky’s metaphor could be misleading because it gives an image of biology being confined to the roots of the tree, whereas the ramifications of biology are not confined to the roots but extend up the trunk into the branches.

With regard to language, Vygotsky, 1978:28) maintains that human beings would be no different from animals without the social contact with other people. This is indeed so. But not because the social imprints itself onto a tabula rasa, but because the human brain has the innate, i.e. biological, structures to learn, whether it be a language or some other kind of tool. Exposure to and intake from society and the world trigger off the innate capacity to learn.

Although there is no doubt that a psychology that is merely based on biology is inadequate, this does not mean that tools are the only source of the abstract notions of the brainmind. Surely there is a symbiotic relationship between the development of the brainmind and mind tools such as aids to measuring, calculating and thinking and communicating. The tool of tools (and often the tool of fools) is the sign: “The use of signs leads humans to a specific structure of behaviour that breaks away from biological development and creates new forms of a culturally-based psychological process” (Vygotsky 1978:40).

It is Vygotsky’s term “breaks away” that is problematic. In order to explain the problem, it would be useful to examine the concept of maturation. According to Vygotsky, it is a fact that “maturation per se is a secondary factor in the development of the most complex, unique forms of behaviour.” Vygotsky 1978:19). However, according to Fodor (Piatelli- Palmirini, 1980:149),

The only intelligible theory of enrichment of conceptual resources is that it is a function of maturation [of the inner unlearned structures of the cognitive subject], and there simply isn’t any theory of how learning can affect concepts.

For Lenneberg (1967), Bickerton (1981), Chomsky (1988) and Pinker (1995), language and conceptual development are largely a function of maturation. It is truistically obvious that without the proper environment, maturation will be adversely affected.

The neo-Piagetians, Demetriou, Shayer and Eflides (1992:6) in their discussion of theories of cognitive development stress the importance of both the biological and the social aspect of learning, but point out, as does Fodor, our ignorance of the learning (and the teaching) process:

[W]e still do not understand very well how cognitive structures interact with each other and how their formation is affected by the structure of knowledge as it exists in our present educational and broader cultural environment…we also know that mental and school-specific knowledge structures are constrained by both internal and social constraint systems…we have already gathered firm knowledge about the developmental and cognitive preconditions under which learning may occur. However, we still need to learn a lot more about how learning situations work in the mind and/or the brain to alter its present state into a more advanced one. Thus, we are not very knowledgeable about how to engineer specific learning environments aimed at quickly and efficiently imparting specific knowledge structures useful to a particular individual of a particular age for a particular purpose.

I agree with Fodor that (biological) maturation, contrary to Vygotsky’s socialist view, plays a much greater role in cognitive development than Vygotsky asserts. Consider language in the context of growth and maturation. Lenneberg’s views (1967) on the biological basis of language is worth perusal. The following are the salient features of Lenneberg’s biological view of language development:

5. Lenneberg’s biological view of language development

Language for many seems to consist of cultural conventions, e.g. Wittgenstein’s “language games” (this is not to deny the monumental contribution of Wittgenstein). However, fruitful explanatory principles can also, and perhaps primarily be found in the biological sciences, specifically in genetics and the brain sciences (Danesi, 1994; Oller, 1981; Paradis, 1991; Perecman, 1989; Pribham, 1971; Woese, 1967:4; Young, 1978). There are radical differences between the rules of games (arbitrarily determined) and the rules of language (biologically determined). However, we can still speak of the biology of game-playing (what are the capacities that enable a human to play a game of chance, to “waste” one’s time away. An animal (humans are included under “animal) is not like a tool that can be arbitrarily assigned a particular use. The structure of the brain and the body are an interdependent unit. These structures are programmed from within, i.e. genetically (Lenneberg 1967:1-4). Thus the developmental process is physiological in nature. Modifications after birth are determined by genetic and prenatal events. The degree of plasticity (e.g. the diversity between individuals) is the product of biological conditions. And obviously the environment plays an important role. “Thus the notion `dependence on environment’ (which by implication is the same as `dependence on experience’) is not a useful criterion for the classification of behaviour” (Lenneberg 1967:12).

6. Conclusion

My point of view has been given, which hopefully was based on more than gut feelings. Vehement oppositions are to expected in academia. But

we have reason to hope that a conversation among discourses, between occupants of this position and that, offers the best hope that we shall create as a species the rich intellectual landscape that is essential if we are to understand our universe and our place in it. The multiple discourses of mankind, brought, now by history into mutual consciousness, are not a Babel but a chorus.

(Appiah, 1992:230; original italics)

The question is: Why are some individuals better choristers than babblers? Here we are touching, what I think is a neglected – because a sensitive – issue in multicultural studies, namely, the biocultural reality of intellectual variability. From the educational point of view, the problem is how to explain the large disparities in academic ability between individuals belonging not only to different cultures, but also to the same culture. Are these disparities merely environmental? Millar (1988) maintains that one can develop concepts, but cannot acquire them. This seems to mean that there are innate capacities that must have something to do with biology. How biology and environment function together is not very clear, but there is no doubt that all abilities are related to inborn biological capacities, which are different in each person. I am not suggesting a (purely) deterministic order, but merely that each individual is “set up” in different ways.

Whatever one’s point of view, all human beings are preoccupied with the ultimate question: “What is it to be human? A (social) “actor” (Miller, 1984)? Knowing how to perform our own or one another’s “tricks” (Miller, 1984; Campbell, 1985:20)? Cuzzort and King (1995) answer:

To be human is to perform, like an actor, before audiences whom we con into accepting us as being what we try to be. Our humanity is the costume we wear, the stage on which we perform, and the way we read whatever script we are handed.

References

Achebe, C. 1988. Hopes and impediments. London: Heinemann.

Appiah, K.A. 1992. Inventing an African practice in philosophy: Epistemological issues. In: Mudimbe, V.Y. (ed.). The surreptitious speech:Presence Africaine and the politics of Otherness. Chicago: University of Chicago Press

Bickerton, D. 1981. Roots of language. Ann Arbor: Karoma Publishers, Inc.

Bhaskar, R. 1979. The possibility of naturalism. Brighton: Harvester.

Bizzell, P. 1979. College English, 40(7):764-771.

Campbell, C.M. 1985. Learning and development. An investigation of a neo-Piagetian theory of cognitive growth. Unpublished Master of Arts thesis, University of Natal, Durban.

Cattell, R.B and Horn,J.L. 1978 A cross social check on the theory of fluid and crystallized intelligence with discovery of new valid subtest designs. Journal of Educational Measurement, 15:139-164.

Chomsky, N. 1988. Language and the problem of knowledge. Cambridge, Massachusetts: MIT Press.

Clark, J., Hall, S., Jefferson, T. and Roberts, B. 1975. Subcultures, culture and class: a theoretical overview . Working Papers in Cultural Studies, 7/8:9-74.]

Cuzzort, R.P. and King. 1995. Life as a con game: The dramatic vision of Erving Goffman. In Cuzzort, R.P. and King, E.W. 1995. Twentieth-Century social thought (5th Edition): Fort Worth: Harcourt Brace College Publishers.

Danesi, M. 1994. ‘The neuroscientific perspective in second language acquisition research: a critical synopsis’, International Review of Applied Linguistics, 32(3): 201-228.

Davies, P. 1993. The mind of God: Science and the search for ultimate meaning. London. Penguin Books.

Demetriou, A., Gustafsson, J-E., Efklides, A. and Platsidou, M. 1992. Structural systems in developing cognition, science, and education. In: Demetriou, A., Shayer, M. ans Eflides, A.. Neo-piagetian theories of cognitive development. New York: Routledge.

Ellis, J.M. 1994. Language, thought, and logic. Evanston, Illinois: North Western University Press.

Fodor, J.R. 1980. Fixation of belief and concept acquisition. In: Piatelli-Palmarini, M. Language and learning: The debate between Jean Piaget and Noam Chomsky. London: Routledge Kegan and Paul.

Hillner. K. 1985. Psychological reality. Amsterdam: North-Holland.

Humboldt, W. von. 1963. Man’s intrinsic humanity: His language. In Humboldt, W. von. Humanist without portfolio: An Anthology of the writings of Wlihelm von Humboldt. Tr. Marianne Cowan. Detrot: Wayne State University Press.

Jensen, A.R. 1972.Genetics and education. London: Methuen & Co Ltd.

Jensen, A.R. 1973. Educational differences. London: Methuen.

Kuhn, T.S. 1970. The structure of scientific revolutions. 2nd Edition. Chicago. Unviersity of Chicago Press.

Luria, A.R. 1976. Cognitive development: Its cultural and social foundations. Cambridge: Harvard University Press.

Miller, R. 1984. Reflections of mind and culture. Pietermaritzburg, University of Natal Press.

Paradis, M. 1991. ‘Language lateralization in bilinguals’, Brain and language, 39: 576-586

Perecman, F. 1989. ‘Language processing in the bilingual: Evidence from language mixing’, Hillenstain, K & Obler, L.K. (Eds.). Bilingualism across the lifespan. Cambridge. Cambridge University Press.

Pinker, S. 1995. The language instinct. London/New York. Penguin Books.

Pribham, K. 1971. Languages of the brain. Englewood Cliffs, New Jersey: Prentice-Hall

Popper. K. 1965. The open society and its enemies. Volume 2. London: Routledge and Kegan Paul.

Russell, B. 1938. Sceptical essays. New York: W.W. Norton & Company, Inc.

Sowell, T. 1988. A conflict of visions. New Delhi: Affiliated East-West Press Pvt Ltd.

Vygotsky, L.S. 1978. Mind in society: The development of higher psychological processes. (Edited by Michael Cole, Vera John-Steiner, Sylvia Scribner and Ellen Souberman). Cambridge, Massachusetts: Harvard University Press.

Wa Thiong’o, Ngugi. 1965.The river between. London: Heinemann,

Whorf, B.L. 1956. Language, thought and reality (edited by Carroll, J.B.). New York: Wiley.

Woese, C.R. 1967. The genetic code: The molecular basis for genetic expression. New York: Harper and Row.

Young, J.Z. 1978. Programs of the brain. Oxford, England: Oxford University Press.

Chomsky, N. 1988a. Language and the problem of knowledge. Cambridge, Massachusetts: MIT Press.

Abstract

Linguistic apartheid

In the classroom

Alternative mystiques

Hermeneutic implications

Conclusion

Bibliography

Presentation at the Linguistics Summer School, Odense, Denmark, July, 1998.

Raphael Gamaroff (University of Fort Hare)

Abstract

A major issue in modern multilingual societies, where an increasing number of learners, and even where the majority of learners, use English as a second, or additional, language, is the controversial distinction between English as a first language and English as a second language. It is argued in this article that the distinction is a valid one. The notions of “first language” and “second language” are better understood when compared with the notions of “native language” and “non-native language”. These four notions are difficult to unbundle because they are so heavily value-laden, and, accordingly, there is the danger that the distinctions between these notions could be interpreted as a form of linguistic apartheid. In spite of the sociopolitical difficulties, these distinctions do have linguistic and educational value.

Linguistic apartheid

The distinction between L1 and L2 has two descriptive connotations: (1) L1 is acquired before L2. L1 need not be the mother tongue, because the mother tongue may not end up as L1; (2) The L1 learner is stronger than the L2 learner, where the L2 learner is unlikely (the weak interpretation) or incapable (the strong interpretation) of reaching the L1 level (Musker and Nomvete 1996:65).

The L1/L2 dichotomy has given rise in South Africa to a “wide range of discriminatory language requirements and discriminatory assessment instruments” (Musker and Nomvete 1996:65). I first examine South African views on the issue and then relate them to views outside South Africa.

Young (1988:8) advocates that the “apartheid” labels of L1 and L2 be discarded (see also Young 1995). These labels imply for Young that black “natives” are not able to assimilate Western language and culture. Paikeday’s (1985:76) perspective below is similar to Young’s:

When theoretical linguists claim an innate facility for competence in a language on behalf of the native speaker,… it seems like a white South African’s claim that he [or she] can walk into a railway station in Pretoria any day, purchase a first-class ticket, get into any first-class coach, occupy a window seat, and travel all the way to Cape Town without getting thrown out at the first stop, as though a black or a coloured could not do it. Some competence. As everyone knows, apart from manmade restrictions, there just isn’t any rational evidence to bake up such claims to exclusivity of the need for separateness of consideration.

Since the democratic elections of 1994 the actual context of this story is no longer applicable, but the destructive ethos it attempts to describe is no less relevant today, namely the attempt to keep certain racial, ethnic and linguistic groups in their positivistic place. For planners of new policies of language education in South Africa “the separation between first and second language is based on a positivistic construct of language” (Human Sciences Research Council 1996:103). What might explain the unpopularity of these labels among some South African researchers is that they regard “positivism” and apartheid as kissing cousins!

In support of Young’s rejection of the “apartheid” labels of L1 and L2, others argue that it is discriminatory (in the worst sense) to compare levels of proficiency between L1 and L2 learners (African National Congress 1992; Forrest 1994; Burroughs, Vieyra-King and Witthaus 1996:77). These authors object to the social meaning attached to the labels of L1 and L2 to different levels (Rampton 1990). Makoni (1995:27) uses the metaphors of language “boundaries” and “boxes” to explain his opposition to labels that create insiders and outsiders:

Paradoxically, in South Africa and indeed even more so in South Africa, the boundaries conceptualisation of language which creates insiders and outsiders is also typical of colonialist and neo-apartheid discourses…The “boxing” of African speech forms into different languages is reinforced by state, legal and educational pressures.

Makoni’s context is the top-down bureaucratic division of South Africa’s many linguistic communities into 11 official separate official “languages”. His main argument is that the multilingual situation in South Africa is a type of multilingualism as seen through the spectacles of a monolingual. Such a view, for Makoni, is not truly monolingual, but poly-monolingual. This view commodifies language. Although Makoni describes the L1/L2 distinction as a “commodification of language” issue, he would probably also regard it as an imposition, as the “boxing in” of speech forms. Young maintains that the labels “L1 and “L2″ segregates “English into separate learning ‘boxes’” and asks: “While the ESL [English as a second language] label is in keeping with international trends in most countries where English is taught and learned, is it not perhaps time that we, in South Africa, begin to consider how socially and politically divisive it is to continue using the ESL label?” (Young 1988:8).

In the classroom

The tendency in NQF thinking is that L2 levels should be based on the L1 paradigm (Alexander 1996:105), and so one should calibrate levels of L1 or native proficiency to determine as precisely as possible what L2 learners are to aspire to (Alexander 1996:106).

This does not mean that L1 and L2 teaching syllabuses should be the same (Alexander 1996:105). Indeed there are very good reasons for keeping L1 and L2 syllabuses apart. The term “apart” in the South African context is thick with negative connotations. If syllabuses must be kept apart, does this mean that L1 and L2 learners must be kept apart in the language classroom? From a practical point of view the idea of separate syllabuses means separate classrooms. Barkhuizen (1991) goes even further by advocating not only separate L1 and L2 classrooms but separate L1 and L2 (ESL) Departments. Barkhuizen (1991:30) states: “The different aims of the second and first language syllabuses and the different approaches used to achieve these aims provide a further rationale for the establishment of an independent ESL Department.” Thus, Barkhuizen takes for granted that there should be separate L1 and L2 classrooms.

No one would dispute that English teaching involves two different kinds of methodology: L1 and L2 methodology. But I don’t think it is necessary or pedagogically sound to (1) divide teachers into L1 teachers and L2 teachers and to go even further by (2) erecting Departmental boundaries between them. One must not forget, however, that most English teachers in South Africa are not “native” speakers of English, and would therefore have difficulty in teaching English as a First Language. Native speakers of English, on the other hand, could teach both English as First and as a Second Language.

Whether we call second-language learning situations by another name, we still have to specifically design learning materials to cater for these second-language situations (Musker and Nomvete 1996:67), and the only feasible way to do so is, I believe, in a separate space such as a classroom (Peirce 1991). Barkhuizen had a change of heart by jettisoning not only the idea of separate English Departments, which was the main thrust of his previous article (1991) but also the idea of separate L1 and L2 syllabuses in favour of “multicultural settings” (Barkhuizen 1992, 1996).

Owing to the political changes in South Africa, what was acceptable in 1991 became unacceptable a year later, and so Barkhuizen met the new challenge of “what needs to be done” (Barkhuizen 1992) under a different political and educational dispensation. The idea of separate L1 and L2 syllabuses is a good one, so is the idea of multicultural settings. It seems that it is difficult to have both because it would mean that the idea of multicultural settings would be relegated to “social” language while the idea of separate L1 and L2 classrooms would hog “academic” language, which would exacerbate the racial-ethnic divide in countries throughout the world.

Alternative mystiques

In the South African political climate the labels “first” and “second” language are becoming increasingly unpopular. Also terms such as “native” language (which in South Africa meant “black”) and “mother” tongue, which feminist movements find sexually discriminatory, are also problematic. James (1994:192) lumps together “native speakerism”, sexism and racism. In the last few paragraphs of this section I discuss briefly a few non-South African views on the topic. . The “whole mystique of a native speaker” (Kachru 1982:vii) who uses his/her mother tongue implies five things, which have been hotly contested (Rampton 1990):

1. A particular language is inherited through birth into a particular social group.

2. If you inherit a language, you can speak it well.

3. One is or isn’t a native/mother-tongue speaker.

4. a native speaker has a comprehensive grasp of the inherited language.

5. Being a citizen of a country is analogous to being a native speaker of one mother tongue.

(Christopherson (1973) was among the first researchers to question these notions mentioned by Rampton).

Rampton (1990) suggests that the following terms replace “native”, “mother tongue”, “first language” and “second language” (See also Leung, Harris and Rampton 1997):

1. Language expertise, i.e. the level of proficiency. An important issue in assessing expertise would be the models of language ability that one would use to decide what is an acceptable or minimum level of expertise. But this is not a new issue, even if the terminology is new.

2. Language affiliation, which is concerned with the affective relationship of a learner towards a language.

3. Language inheritance. Membership of an ethnic group does not automatically mean that the language of the ethnic group has been automatically inherited. In South Africa, one can change one’s “mother tongue” by entering another ethnic group or have a hybrid mother tongue, or “replacement” language. A replacement language is a language that becomes more dominant than the mother tongue, usually at an early age, but is seldom fully mastered, as in the case of, for example, some Coloured and Indian children in South Africa who are brought up on a hybrid of English and Afrikaans (in the case of Coloureds) or a hybrid of English and Indian languages or Afrikaans, and some Bantu speakers in South Africa who are brought up on a cocktail of two or more Bantu languages such as Tswana, Sepedi and Xhosa (and English). (See Gamaroff 1998).

Davies (1995:145) maintains: “In terms of ultimate attainment the post-pubertal second language learner may, exceptionally. attain native speaker levels of proficiency and therefore be indistinguishable from the native speaker.” Paikeday (1985) goes further by suggesting that the exception proves that there is no rule, i.e. that there is no native speaker. Undoubtedly there are exceptional cases that disprove the rule. But this is true of many categorisations. Medgyes (1992) suggests that the exceptions prove the rule, and that, therefore, native speakers of a language are – taking into account some exceptions, or as Quirk in Paikeday (1985:7) puts it the “fuzzy edges” – recognised as such. Paikeday (1985:11) plugs the argument that the exceptions do disprove the rule and consequently one can only legitimately speak of degrees of competence of language use, as one would about any other skill, e.g. rowing boats or mowing lawns.

Mother-tongue speakers (the language one uses in one’s early childhood) and first-language speakers (the language one knows best) are usually identifiable. There are exceptions where one can have (1) more than one first language (2) low competence in one’s mother tongue and (3) no first language, i.e. no language that one knows well, e.g. a “replacement” language, referred to earlier.

Hermeneutic implications

The “elitist” may argue that “not all members of a linguistic community are equal in linguistic knowledge” (Harris 1981:171) about their common native language, and so it is possible to have some native or L1 speakers who know more than others about the language. Harris counters that language qua language is indeterminate because knowledge as such is indeterminate. Accordingly, for Harris, there are no “job-secure” words (Harris 1981:175). But, if language (and ergo knowledge) is so insecure that one cannot define anything – approximately, to be sure; that is, if there is no native or L1 or L2 “bulls-eye” (Russell 1921:197-198 in Harris 1985:170) in language, there can be no truth at all – approximate to be sure. For McArthur (1998) Standard English is a “fuzzy sub-set of a very large set of Englishes” and Lass (1998) believes that English is changing as we speak it. In such a scenario, Standard English – and I would imagine the same applies to the notions of native, L1 and L2 “speakerism” – becomes, in the fuzzy-set theory of Lass, a sub-set of the fuzz that was.

Conclusion

L2, or ESL, is like all labels that distinguishes between those who have more and those who have less. It segregates – nationally and internationally. However, if the rest of the world finds the labels ESL and L2 useful, in spite of its shortcomings, couldn’t South Africans become part of the international community, which has had, and continues to have, its fair share of racial-ethnic-linguistic discrimination. Perhaps it might be better to use the label “primary” instead of “L1″ and so get rid of the notion of “first language”. But if “L1″ becomes a “primary” language, L2 could be seen as a “secondary” language, which, if not more discriminatory than “second” language, could belittle the notion of “second” language (L2). Perhaps, we could use “additional” language, because the counterpart of additional language is “main” language, which has a less discriminatory ring than “primary” language. One could opt for Rampton’s (1990) alternative set of terms mentioned above. I would think, though, that the traditional labels will be with us in South Africa (and the rest of the world) for a long time to come. For example, the Department of Linguistics of Stellenbosch University is hosting its 4th conference on Linguistics and the Language Professions in April 1999 on the theme of “second languages”.

Bibliography

African National Congress. African National Congress. 1992. ANC policy guidelines for a democratic South Africa. Johannesburg: ANC.

Alexander, N. Alexander, N. 1996. ‘Drawing the issues together: In the context of language education policy’. Human Sciences Research Council (ed.). Language assessment and the National Qualifications Framework. Pretoria: Human Science Research Council Publishers.

Barkhuizen, G. Barkhuizen, G. 1991. ‘Proposal for an independent English Second Language Department at Mmabatho High School’. English Language Teaching Centre (ELTIC) Reporter 16(1):25-32. Johannesburg.

Barkhuizen, G. Barkhuizen, G. 1992. G. Barkhuizen. Teaching English in multilingual settings: What needs to be done. Journal for Language Teaching 26/4:53-68.

Barkhuizen, G. Barkhuizen, G. 1996. Using English in the South African classroom. Per Linguam 12/1:34-47.

Burroughs, E., Vieyra-King, M. and Witthaus, G. Burroughs, E., Vieyra-King, M. and Witthaus, G. 1996. ‘The assessment of language outcomes in ABET: Implications of an approach’. Human Sciences Research Council (ed.). Language assessment and the National Qualifications Framework. Pretoria: Human Science Research Council Publishers.

Christopherson, P. Christopherson, P. 1973. Second-language learning. Harmondsworth: Penguin.

Collier, V.P. Collier, V.P. 1987. ‘Age and rate of acquisition of second language for academic purposes’. TESOL Quarterly 21/4:617-641.

Cummins, J. Cummins, J. 1984. ‘Wanted: A theoretical framework for relating language proficiency to academic achievement among bilingual students’. Rivera, C. (ed.). Language proficiency and academic achievement. Multilingual Matters 10. Clevedon: Multilingual Matters Ltd.

Davies, A. Davies, A. 1995. ‘Proficiency or the native speaker: What are we trying to achieve in ELT?’ Cook. G. and Seidlhofer, B. (eds.). Principle and practice in applied linguistics: Studies in honour of H.G. Widdowson. Oxford: Oxford University Press.

Forrest, F. Forrest, F. 1992. A language curriculum framework for compulsory general education. Johannesburg: Centre for Educational Policy Development.

Gamaroff, R. Gamaroff, R. 1998. ‘Cloze tests as predictors of global language proficiency: A statistical analysis’. South African Journal of Linguistics 16/1:7-15.

Harris, R. Harris, R. 1981. The language myth. London: Duckworth.

Human Sciences Research Council. Human Sciences Research Council. 1996. Language assessment and the National Qualifications Framework. Pretoria: Human Science Research Council Publishers.

James, C.James, C.1994. ‘Don’t shoot my dodo: on the resilience of contrastive and error analysis’. International Review of Applied Linguistics 32/3:179-200.

Kachru, B.B. Kachru, B.B. (ed.). 1982. The Other Tongue: English across cultures. Oxford: Pergamon.

Lass, R. Lass, R. 1998. “English” – “Talk at Will” South African radio programme on SAfm, 26 February.

Leung, C. Harris, R. and Rampton, B. Leung, C. Harris, R. and Rampton, B. 1997. The idealised native-speaker, reified ethnicities and classroom realities: Contemporary issues in TESOL. London: Thames Valley University.

Makoni, S. Makoni, S. 1995. ‘Some of the metaphors about language, in language planning course in South Africa: Boundaries, frontiers and commodification’. Per Linguam 11/1:25-34.

McArthur, T. McArthur, T. 1998. ‘English as a world language, as an African language, and as a South African language. Proceedings of the English Academy of Southern Africa conference “English at the turn of the Millennium”, Johannesburg College of Education, 14-16 September.

Medgyes, P. Medgyes, P. 1992. ‘Native or non-native: who’s worth more?’ ELT Journal 46/4:340-348

Musker, P. and Nomvete, S. Musker, P. and Nomvete, S. 1996. ‘Standards and levels in language assessment’. Human Sciences Research Council (ed.). Language assessment and the National Qualifications Framework. Pretoria: Human Science Research Council Publishers.

Olivier, A. Olivier, A. 1998. ‘The role of input in language development at tertiary level’. South African Journal of Education 18/1:57-60.

Paikeday, T.M. Paikeday, T.M. 1985. The native speaker is dead! Toronto: Paikeday Publishing Inc.

Peirce, B. Peirce, B. 1991. ‘On language difference and democracy’. Language Projects Review 6:21-24.

Rampton, B.H. Rampton, B.H. 1990. ‘Displacing the “native speaker”: expertise, affiliation and inheritance’. English Language Teaching Journal 44/2:97101.

Russell, B. Russell, B. 1921. The analysis of mind. New York.

Young, D. Young, D. 1988. ‘English for what and for whom and when?’ Language Projects Review 3(2):8.

Young, D. Young, D. 1995. ‘The role and status of the first language in education in a multilingual society’. Heugh, K, Siegruhn, A. and Pluddemann, S. (eds.). Multilingual