Skip to main content

Full text of "ERIC ED358830: Quality in Education and Training. Aspects of Educational and Training Technology. Volume XXVI."

See other formats


DOCUMENT RESUME - 
ED 358 830 IR 016 111 



AUTHOR 
TITLE 

INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 



AVAILABLE FROM 



PUB TYPE 



Shaw, Malcolm, Ed. ; Roper, Eric, Ed. 
Quality in Education and Training* Aspects of 
Educational and Training Technology. Volume XXVI. 
Association for Educational and Training Technology, 
London (England) ♦ 

ISBN-0-7494-0897-9; ISBN-0-89397-385-8 
92 

268p.; Proceedings of the Association for Educational 
and Training Technology's International Conference 
(1992) . 

Kogan Page Limited, 120 Pentonville Road, London Nl 
9JN, England; Nichols Publishing, P.O. Box 6036, East 
Brunswick, NJ 08816. 

Books (010) — Collected Works - Conference 
Proceedings (021) — Reports - Evaluative/Feasibility 
(142) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF01/PC11 Plus Postage. 

Computer Software Development; Curriculum 
Devel opment ; Educat i onal Assessment ; *Educat i onal 
Quality; *Educat i onal Technology; Elementary 
Secondary Education; Instructional Design; 
Interactive Video; Organizational Change ; 
Postsecondary Education; ^Quality Control; Teaching 
Methods ; ^Technological Advancement ; ^Training 
Association for Educational and Training Tech; 
Stakeholder Evaluation; Total Quality Management 



ABSTRACT 

The 38 conference papers in this volume were chosen 
to exemplify different definitions of, and approaches to, quality, as 
they are applied in a wide range of educational and training 
contexts. The papers are: "Designing Organisations That Learn" (D. J. 
Dicks); "Quality Assurance in a European Context" (D. Alexander, J. 
Morgan); " f What's in It for Me?'" (C. Hart, M. Shoolbred); "Quality 
and the Academic Administrator" (C. D. Payne); "Total Quality 
Management in an Education and Training Context" (B. Ellis); 
"Performance Management in a Competence Framework" (E. Sauve) ; 
"Quality in Further Education: An Unchanging Agenda" (D. 
Shepherds on) ; "Defining Quality in Higher Education" (A. Burrows, L. 
Harvey); "Total Quality Implementation — Cultural Issues and Training" 
(B. Hurley); "Developing the European Quality Model" (J. S. Oakland, 
L. J. Porter); 'Strategic Quality Management" (R. J. Newton); 
"Profits and Pitfalls in Establishing a Qualytechnic" (J. Heap, H. 
Solomon); "Student Satisfaction and Perceptions of Quality" (P. M. 
Mazelan and others); "Experiences in the Design and Conduct of 
Enterprise Audits" (D. Edgar); "Monitoring the Quality of Quality 
Control Systems" (H. I. Ellington, G. T. N. Ross); "Improving the 
Quality of a National Curriculum" (I. Musallam, M. Brophy, M. 
Schilling); "Developing Quality in Education" (R. J. D. Rutherford); 
"Quality in Course Design" (E. Roper); "Quality Horses for Quality 
Courses: Matching Students with Courses in Music" (L. Gibbs) ; 
"Recognising Quality in Engineering Education" (D. C. Hughes, R. G. 
S. Matthew); "Exploring Learners 1 Perceptions of Quality" (D. Miller, 
P. Funnell); "Concept Mapping, Post-Questioning and Feedback in 
Distance Education" (R. M. Bernard, S. Naidu, K. Lundgre-Cayrol^ ; 
"Comparing Chalk and Cheese — Quality in Assessing Worx-Based 



Learning" (C. Bucklow) ; "Using Learning Contracts To Enhance the 
Quality of Work-Based Learning" (I. S. Marshall, M. L. Mill); 
"Quality of Assessment" (P. Race); "BS5750 for Assessment" (P. Race); 
"Eating Frogs and Bridging Gaps — Post-Warnock Conditions for Teaching 
Quality" (C. Colling); "Teaching Quality" (D. Jones, J* Hanson); 
"Upward Appraisal and Its Implications for Higher Education" (G. 
McElwee and others); "A Total Quality Approach to Managing CBT 
Development" (S. G. Shaw, D. R. Shaw); "Towards Quality Management in 
Training Design" (M. Williams, G. Carr) ; "Achieving Quality in 
Networking Interactive Video" (P. Willis, J. Early); "Quality in 
Modernising Educational Technology" (D. Hawkridge) ; "Trends and 
Issues in Educational Technology" (D. P, Ely); "Training Quality and 
New Technology" (A. R. Bartolome) ; "Connecting Lectures and 
Laboratory Learning with CAI" (Y. Araki) ; "Graphical Routes to 
Quality Courseware" (P. Barker, C. Lamont) ; and "Enhancing the 
Quality of Student-Centered Mathematics" (D. Bowers, R. Burrell) . 
Most of the papers include references. (MES) 



it it jfc it it it it it it it it it it it it it it it it it it it it it it it A A Vc it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 

i: Reproductions supplied by EDRS are the best that can be made * 
* from the original document. * 

it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 



Assets ®$ EdidS&ifonal and 
Training Technology XXVi 



^ rr^ .^i j' rro " r ^ ? r^? ^Ti 
Li L vlh^J b U u L b u ii Li ^vj 



Edited by 

Malcolm Shaw and Eric flop or 




U.S. DEPARTMENT Of EOUCATtON 

Office ot Educational At March and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

C This document has been reproduced at 
'«»ceived from the person or organization 
originating it 

C Minor changes have oeen made to improve 

'eoroduction Quality 

e Pomts ot view o< opinions stated <n this docu-- 
-^ent do not necessarily represent official 

OERl position or policy 





"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

N.R. Winterburn 



ERJC 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER {ERIC)." 



Aspects of Educational and 
Training Technology 

VolumeXXW 



Aspects of Educational and 
Training Technology 

VolumeXXW 

Quality in Education and Training 

Edited for the Association for Educational and 
Training Technology by 

Malcolm Shaw and Eric Roper 



TNICHOLS/GP 
[PUBLISHING' 

Kogan Page, London 

Nichols Publishing Company, New Jersey 




First published in 1993 

Apart from am fair dealing for the purposes of research or private study, or criticism or 
review, as permitted under the Copyright, Designs and Patents Act. 1988, this 
publication may onlv be reproduced /stored or transmitted, in any form or by any 
means, with the prior permission in writing of the publishers, or in the case of 
reprographic reproduction in accordance with the terms of licences issued by the 
Copyright Licensing Agency. Enquiries concerning reproduction outside those terms 
should be sent to the publishers at the undermentioned address : 

Kogan Page Limited 
120 Pentonville Road 
London N19JN 



t The Association for Educational and Training 'technology. 1993 



British Library Cataloguing in Publication Data 

A CI P record for this book is available from the British Library 
ISBNO 7494 0807 9 



Published in the United States of America by Nichols Publishing, 
P.O. Boxf)Q36. East Brunswick, New Jersey 08816 

A C I P record for this book is available from the Library of Congress 

ISBN 0-89397-385-8 



Printed and hound in Great Britain by 
Biddies Ltd, Guildford and King's Lynn 



Contents 



List of contributors 1X 
Editorial x 
Section 1: Perspectives on Quality in Organisations 1 



11 Strategic quality management: if I pull it up the flagpole will anyone 
salute? 

Robert James Newton 

12 Profits and pitfalls in establishing a qualytechnic 
John Heap and Hazel Solomon 



3 
10 



1 Designing organisations that leam 
Dennis J Dicks 

2 Quality assurance in a European context 
David Alexander and Jim Morgan 

3 "What's in it for me?" Organisational culture, rewards and quality 16 
Christopher Hart and Michael Shoolbred 

4 Quality and the academic administrator 
Catherine D Payne 

5 Total quality management in an education and training context 29 
Barbara Ellis 

6 Performance management in a competence framework 35 
Elaine Sauve 

7 Quality in further education: an unchanging agenda 4 0 
David Shepherdson 

8 Defining quality in higher education - the stakeholder approach 44 
Alison Burrows and Lee Harvey 

Section 2: Implementing Quality in Organisations 51 

9 Total quality implementation - cultural issues and training 53 
Brian Hurley 

10 Developing the European quality model 59 
John S Oakland and Leslie J Porter 



68 



72 



13 Student satisfaction and perceptions of quality 76 
P M Mazelan, D M Green, C R Brannigan and P F Tormey 

14 Experiences in the design and conduct of enterprise audits 82 
Douglas Edgar 



15 Monitoring the quality of quality control systems 
Henry I Ellington and Gavin T N Ross 



88 



Section 3: Quality in Courses 93 

16 Improving the quality of a national curriculum: the role of textbooks 95 
/ Musallam, M Brophy and M Schilling 

17 Developing quality in education: a way forward 102 
RJD Rutherford 

18 Quality in course design: empowering students through course processes 107 
and structures in a professional development scheme 

Eric Roper 

19 Quality horses for quality courses: matching students with courses in music 1 14 
Louise Gibbs 

20 Recognising quality in engineering education 120 
D C Hughes and RG S Matthew 

Section 4: Quality in Learning and Assessment 123 

21 Exploring learners' perceptions of quality 125 
Dave Miiller and Peter Funnell 

22 Concept mapping, post questioning and feedback in distance education 132 
Robert M Bernard, Som Naidu and Karin Lundgren-Cayrol 

23 Comparing chalk and cheese - quality in assessing work-based learning 140 
Caroline Bucklow 

24 Using learning contracts to enhance the quality of work-based learning 144 
Iain S Marshall and Margaret L Mill 

25 Quality of assessment 149 
Phil Race 

26 BS5750 for assessment 162 
Phil Race 



Section 5: Quality in Teaching and Training 173 

27 Eating frogs and bridging gaps - post-Wamock conditions for teaching quality 175 
Clive Colling 

28 Teaching quality - "I know it when I see it!" 180 
David Jones and Janet Hanson 

29 Upward appraisal and its implications for higher education 186 
Gerard McElwee, Tom Redman, David Edson and Adrian Evans 

30 A total quality approach to managing CBT development 192 
Steven G Shaw and David R Shaw 

31 Towards quality management in training design 214 
Meurig Williams and Graham Carr 

32 Achieving quality in networked interactive video 222 
Peter Willis and John Eary 



Section 6: Quality in Technology 229 

33 Quality in modernising educational technology 231 
David Hawkridge 

34 Trends and issues in educational technology 237 
Donald P Ely 

35 Training quality and new technology 241 
Antonio R Bartolome 

36 Connecting lectures and laboratory learning with CAI 246 
Yoshikazu Araki 

37 Graphical routes to quality courseware 250 
Philip Barker and Charles Lament 

38 Enhancing the quality of student-centred mathematics 255 
David Bowers and Rod Burrell 



Index of contributors 



Alexander D 10 


McElwee G 186 


Araki Y 246 


Marshall I S 144 




Matthew R G S 120 


Barker P 250 


Maze lan P M 76 


Bartolome A R 241 


Mill M L 144 


Bernard R M 132 


Morgan J 10 


Bowers D 255 


Muller D 125 


Brannigan C R 76 


Musallam I 95 


Brophy M 95 




Bucklow C 140 


Naidu S 132 


Burrell R 255 


Newton R J 68 


Rtiirnw^ A 44 






Oakland J S 59 


Carr G 214 




Colline C 175 


Payne C D 22 


Porter L J 59 


Dicks D J 3 






Race P 149, 162 


Eary J 222 


Redman T 186 


Edgar D 82 


Roper E 107 


Edson D 186 


Ross G T N 88 


Ellington H I 88 


Rutherford R J D 102 


Ellis B 29 




Ely D P 237 


Sauve E 35 


Evans A 186 


O _t_ 'II' „ \ m AC 

Schilling M vd 




Shaw D R 192 


Funnell P 125 


Shaw S G 192 




Shepherdson D 40 


Gibbs L 114 


Shoolbred M 16 


Green D M 76 


Solomon H 72 


Hanson J 180 


Tormey P F 76 


Hart C 16 




Harvey L 44 


Williams M 214 


Hawkridge D 231 


Willis P 222 


Heap J 72 




Hughes D C 120 




Hurley B 53 




Jones D 180 




Lamont C 250 




Lundgren-Cayrol K 132 





Editorial 



This volume constitutes the Proceedings of the Association for Educational and Training 
Technology's 1992 International Conference, which explored "Quality in education and 
training". The theme is topical in view of the growing Government initiated climate of 
accountability, performance indicators and monitoring of quality, and in the context of 
the developing activities of sector-specific agencies such as the Higher Education Fu *ding 
Council through its quality audit activities and its quality assessment initiatives. 

One of the origins of the quality "movement" was the attempt to re-build Japanese 
industry after World War II. Initially many of the ideas were American, but the Japanese 
subsequently developed their own approaches to quality. Nowadays the movement is 
worldwide, and the approaches increasingly differentiated and sophisticated. However, 
the extent to which ideas and procedures developed in industrial and business applications 
and in different cultures can be applied to specific education and training contexts is 
problematic. To take some examples, is it possible and/or desirable in education and 
training to design "error-free" processes, to get it "right first time", to isolate variables 
and represent them statistically, to conform to requirements? Isn't education and training 
a more humane process, in which both teacher and learner are reaching out into what is, 
for each of them, the unknown? In doing so, aren't they extending the limits of their 
capabilities and surprising themselves - a process in which error or failure, if not feared 
or disguised (or eliminated!), can generate powerful insights and contribute to learning? 

Whilst acknowledging its problematic nature, much is being done to develop ways of 
enhancing and assuring quality in education and training. The papers in this volume have 
been chosen to exemplify different definitions of and approaches to quality, as they are 
applied in a wide range of educational and training contexts, and to explore some of the 
issues raised. Although the total quality management perspective is strongly represented, 
so too are others such as British Standard 5750, fitness for purpose, identifying and 
meeting the needs of different stakeholders, and empowering the customer or client. Also 
included are examples of applications of some of these perspectives outside education and 
training - for example, in commercial CBT development (Chapter 30) - which may (or 
may not) inspire us to adopt similar procedures in education and training, but which 
exemplify the methodology. Reflecting the international nature of the conference, several 
of the contributions are from outside the UK. 

The eclecticism of the contributions is inevitable. They have been written by 
practitioners wrestling with the problems of identifying and delivering quality in specific 
contexts, sometimes drawing explicitly on theoretical perspectives or attempting to adapt 
industrial applications of quality to educational settings, but often developing pragmatic 
solutions in response to institutional and external demands for quality assurance or in 
addressing day-to-day problems. The contributors demonstrate first-hand knowledge, 
expertise, originality and familiarity with relevant literature. In presenting their work in 
ways which vividly capture the richness of their experiences - the problems and struggles 
as well as the solutions - they help to demystify the search for quality and allow others 
to make connections with their own experiences and concerns. 

The organisation of these Proceedings reflects that of the conference. The organisers and 
editors struggled with alternative arrangements, but settled for these rather arbitrary 
divisions. We are aware that many papers could have been included in alternative 
sections. This is particularly the case with those illustrating applications of technology. 
We apologise to those contributors who feel that their work has been misplaced! Each 




of the sections has an introduction which provides an overview of the papers included. 

The timeliness of the conference theme was demonstrated by the number of contributions 
offered (over 70 were delivered as papers, seminars and workshops, together with poster 
and exhibition sessions). Such was the response that the editors have been unable to 
include in these Proceedings all of the conference contributions. They wish to 
acknowledge here the part played by all conference contributors in stimulating and 
engaging in what was a high quality, detailed and constructive debate. All conference 
participants were able to find sessions from which they were able to take new knowledge 
c : perspectives, together with practical ideas. The keynote addresses, delivered by Geoff 
Stanton of the Further Education Unit, John Oakland, Brian Hurley and Dave Miiller and 
Peter Funnell, all provided ideas and stimulated thinking which participants related to and 
developed in other conference sessions. The editors* thanks are extended to these 
keynote speakers for providing the "warp" in the "weft" of the conference! 

Organising a major Conference, like AETT 1992, requires the help and commitment of 
a considerable number of people. Whilst it would be absolutely impossible here to name 
all those who contributed to AETT 1992. it is certainly an appropriate place to single out 
the following friends and groups for a special mention in view of the extensive and vital 
contribution that they made to the success of our venture ... 

Members of the Planning Group: Graham Cheetham. Katherine Hayes, Bob Matthew, 
David Pratt and Hazel Solomon. 

Our sponsors: Cartwright Brice Office Supplies (Leeds), Kodak Photocopiers Division, 
Nestle Rowntree Division (York), York City Council Economic Development Unit, York 
Wines of Sheriff Hutton, Leeds Metropolitan University (Computing Services Unit, 
Media Services Unit, Enterprise in HE Unit, Communications and External Relations). 

AETT stalwarts, but in particular: Henry Ellington. Fred Percival and Roy Winterburn. 

Our Conference Team: students (Philippa Allott, Sarah Howson, Caroline D* Attorre and 
Melissa Labram) and technician - Bruce Knights. 

Special thanks are also due to our colleague, Stuart Rawnsley, for his unflagging and 
always enthusiastic achievements not least with regard to the social programme and 
sponsorship. 

We are sure that all would wish to join us, once more, in saying a re Ay big thank you 
to the one person on our Team who, above all. seemed able to remain absolutely calm, 
whilst the rest of us were quietly panicking, and who has done much to organise your 
editors through the task of producing these Proceedings * Jane Thomas, the conference 
secretary. 

Finally our thanks go to all the contributors and delegates, without whose support we 
could not have had such a memorable, enjoyable and QUALITY conference! 

Eric Roper 
Malcolm Shaw 

Leeds Metropolitan University 



Vjl 



Section 1: Perspectives on Quality in 
Organisations 



Contributors here have focussed on a number of theoretical issues and matters of 
principle that are suggested, in many cases, as necessary conditions for the application 
of quality approaches in organisations. 

The contexts vary widely, beginning with the global concerns outlined in "Designing 
organisations that learn" by Dennis Dicks, who questions the validity of trying to transfer 
essentially Japanese notions of quality into a British context and culture. This cultural 
theme is echoed in "Quality assurance in a European context" by David Alexander and 
Jim Morgan, where the context is Europe and the paradoxes and dilemmas inherent in 
the transfer of some quality indicators across higher education institutions in its Member 
States. 

In "What's in it for me?", Chris Hart and Michael Shoolbred focus on the individual 
organisation and the cultural features that might inhibit or nurture the growth of quality. 
Similarly, the focus is firmly located in the organisation in Catherine Payne's 
consideration of "Quality and the academic administrator", where factors influencing the 
implementation of TQM in higher education are viewed from the perspective of the 
educational administrator. Barbara Ellis also describes the necessary conditions for 
application of TQM in higher education and uses a case study approach by means of 
illustration. 

The role of effective management training in the push for TQM is the theme explored by 
Elaine Sauve in "Performance management in a competence framework" and this is 
linked to some contemporary training initiatives. The importance of training is also 
echoed by David Shepherdson, who goes on to describe a large scale training and 
development initiative across a Local Education Authority. 

Finally, ihe way in which the views on quality of the different stakeholders in education 
are to be compared and contrasted is the subject of work described by Alison Burrows 
and Lee Harvey in their multi-perspective approach - "Defining quality in higher 
education: the stakeholder approach". 



1 2 



1 Designing organisations that learn 

Dennis J Dicks, Concordia University, Montreal, Canada 
INTRODUCTION 

There is much debate in the U.K. and in North America about the way we organise work. 
A central issue is the impact of Japanese practices: can they be transplanted? Will they 
converge towards our methods or not? 

This debate arises in the context of a broader argument about the nature of industrial 
production. Some theorists suggest that the Japanese system heralds a break with the 
"Fordist" model of mass production: work teams, job rotation and innovation replacing 
centralised control, task fragmentation, and regimentation or small-firm networks 
replacing large corporations (Piore and Sabel 1984). Others see it as a transitory 
phenomenon (Gertler 1988) or even a refined form of Fordism (Dohse et al 1986). 

In any case, the current industrial imperative demands flexible response to changes in 
the business environment, innovation, short design to production cycles, tight scheduling, 
lean operation and quality control. The Japanese have demonstrated that they can achieve 
these things; and they have done so by creating organisations that learn, organisations in 
which comparative advantage is based upon the development and deployment of workers' 
skills and intelligence (see for example Dore et al 1989; Koike 1988). 

How is this done? The evidence indicates that the Japanese approach is systemic, in 
so far as it integrates values and practices developed in several institutions: the family, 
school, university, industry and government. 

DESIGNING A LEARNING ORGANISATION IN JAPAN 
Human Capital: Investing in People 

Much has been written about the superiority of Japanese management techniques. 
However, one could argue that managing in Japan is easier than in North America 
because human resources are belter suited to the tasks at hand. This is because skills 
development more closely follows the idealized "human capital" model; in short, people 
are given the information, incentives, and tools to invest in themselves with the promise 
of future payoffs. These investments are exploited, for mutual benefit, in the ways these 
skills are developed and deployed in the workplace. Japanese firms thus display what has 
been called "organisational capital". 

Ln the Japanese context, there arc five key elements contributing to the development of 
human and organizational capital: formal education, recruitment, deployment, 
remuneration, information transfer. 

Formal Education 

We have all heard about the rigours of the educational system in Japan. Students spend 
more hours per day, more days per week, and more weeks per year at school. On 
average, they are more dedicated than their North American peers to learning. One 
reason for this is the extreme competitiveness of a large society crowded onto a small 
island, with virtually no opportunity to move horizontally to other countries. Another 
reason is the complexities of the language, which require enormous concentration and 
effort to reach the compulsory level of mastery at the end of high school. After that 




4 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



point, with entry to a good technical college or university, demands on the student in fact 
become less rigorous. Students (and parents) are willing to invest their time and money 
in this process because they know it generally pays off in the trappings of success - 
money, status, the means to raise a family. 

Recruitment 

This effort would be lost if the skills produced by the educational system were not 
recruited into productive work. Graduates are carefully matched to employment 
opportunities in a highly organised recruitment system working on a national scale. 
TTiose from technical colleges flow into the openings for skilled labour, those from 
universities into the openings for office workers. As part of the effort to have employees 
identify with the firm rather than with economic classes, distinctions are minimized, so 
that the technical jobs which we would characterize as "blue collar" have relatively high 
status in Japan. 

Graduates are not recruited for specific positions, but for their potential to contribute 
to the organization in the long term. Accordingly, the level of recruitment is not tightly 
bound to economic conditions; and the criteria for selection focus more on attitude, basic 
ability and willingness to learn rather than precocious performance. 

Deployment 

Deployment has three key facets: probation, job rotation/working group, job-related 
training. 

Graduates fresh from school enter a period of probation, lasting perhaps three years. 
This entails a program of activities intended to develop skills and build identification with 
the group: "boot camps", orientations, and rotating appointments to several relevant parts 
of the organization. By the end of this period, the new employee and the firm will have 
assessed the potential for a fruitful relationship and substantial portion of the entrants will 
have left or been sent on their way by this point. 

This pattern of developing skills continues throughout the employee's life in the 
organization. Employees hired from other firms follow a similar pattern. Training may 
be organised within the firm, or the employee may be strongly encouraged to pursue 
courses outside working hours. Job rotation and working in groups not only develop a 
broad range of operational skills but also instill a clearer understanding of the firm's 
activities. 

Remuneration 

One of the glues holding these practices together is the system of remuneration, which 
blends a number of factors into the payment schedule for each individual: seniority, 
individual and group performance, and special needs such as family situation, 
geographical location, commuting distance. Each firm has its own way of weighting 
these factors, but the basic principles of the so-called "seniority-merit" system are the 
same: salary profiles follow a roughly s-shaped function (the familiar learning curve) in 
a sense underpaying the recruit during the probationary period, and overpaying during the 
period when the demands of raising a family are greatest. The system thus rewards 
loyalty as well as performance. It also conforms to "human capital theory" (Becker 
1964) in so far as payments are lower when the employee is less productive and is in fact 
benefitting from the skills being instilled at the organisation's expense. Japanese firms 



DESIGNING ORGANISATIONS THAT LEARN 



5 



can thus invest in "general 11 skills because the prospect of higher payments later on serves 
as the incentive for the employee to remain with the firm. 

Information Transfer 

If remuneration patterns are the glue that holds the system together, the lubricant that 
keeps it going is information transfer. One of the most significant features of the 
Japanese form of organisation is the decentralisation of control over information. 
Essentially, responsibility for dissemination of job-related information is pushed down to 
the lowest level possible. The work -team leader, and even team members, are expected 
to share their knowledge with co-workers. In effect, the day-to-day training function 
becomes the job of ordinary workers. 

Sharing information is one of the most important ways of raising the status of people 
performing technical jobs. In Japan, "blue collar" work takes on many of the decisional 
and innovative aspects of "white collar" work. 

Organisational Capital 

Together, the measures taken in Japan to develop human capital also contribute to an 
analogous form of investment, "organisational capital". In this case, investment is in 
organisational form, such that the organisation develops practices which enable it to adapt 
more easily to changing conditions (Prescott and Visscher 1980). Firms create 
organisational capital in three ways: by optimising the match between people and jobs, 
by optimising the match between co-workers, by optimising feedback so that the 
organisation can learn from its own experience. 

As outlined above, the Japanese system tends to optimise the match between people and 
jobs through careful recruitment, job rotation, and training. Secondly, it tends to optimise 
the match between co-workers by organising them in working groups, where peer 
processes are more important than external control; flows of information are decentralised 
to optimize this form of organisation. Thirdly, peer processes, the remuneration system 
and other factors favouring a focus on long-term goals facilitate the development, 
adoption and rapid diffusion of "best available practice" through the firm. All these 
practices encourage flexibility and adaptiveness. 

A knowledgeable reader would point out that the practices which I have attributed to 
the Japanese model are those of the major firms, those capable of sustaining the ideal of 
"lifetime employment". Different estimates would suggest th 4 the model might cover 
from 30% to 70% of the Japanese workforce, almost exclusively males. However, the 
point is that the practices I have described do in fact serve as the ideal which informs the 
decisions of those who invest in human capital: students, parents, governments and 
employers. In an earlier paper (Dicks 1986), I suggested that this provides a fourth way 
of developing organisational capital: when most of the firms competing for human 
resources optimise in the ways listed above, then organisational capital is created on a 
societal or national scale. 

Their overall goal is to minimise risk by sharing it over all employees (Nagatani 1991). 
Information critical to the enterprise's success is readily developed, quickly diffused and 
turned into comparative advantage. In this sense, Japanese organisations not only 
facilitate the learning of individual workers, but can be said to leam collectively from 
past experience. 

Though the details vary from firm to firm, and the pattern becomes fainter in smaller 
firms, the Japanese approach to building organisations that learn is rational and systemic: 



6 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



all the parts - formal education, recruitment, deployment, remuneration and information 
flows - fit together and are even mutually supportive. Given the interdependence of these 
parts, the system cannot be transferred easily to different organisational contexts, such as 
we find in the Anglo-Saxon parts of the world. 

THE ANGLO-SAXON EMPLOYMENT MODEL 

In contrast, "Fordist" methods of organising work seem to be increasingly counter- 
productive. I lump them together as the Anglo-Saxon model, for they have arisen in and 
dominated English-speaking cultures, at least those on the Atlantic rim. The model is 
based on free- market concents such as McMillan's "Type A" (McMillan 1979). That is 
to say, individuals are expected to develop their skills and sell them to the highest bidder. 
Emphasis falls on quick profit (minimum wage for serving fast food) rather than long 
term planning (finishing secondary school). Skills are seen to be the property of 
individuals, rather than the attributes of a working group. Similarly, organisations tend 
to be viewed from a structural rather than a dynamic perspective. Their overall goal is 
profit-maximisation, and so they tend to cost-reduction by minimising the quantity and 
quality of labour, thus pushing control cf information and profit-taking ever upward in 
the hierarchy. The free-market model is generally antithetical to the human capital model 
discussed above. 

The free-market model probably suffices when production makes minimal demands for 
skills (eg the factory system), when labour is plentiful, when other resource costs are low, 
when there is little competition for markets. It served North America well during most 
of this century, particularly after the Second World War. However, there is every 
indication that it is outmoded: we have high unemployment and skill shortages at the 
same time; state of the art has moved elsewhere; education and training are expensive but 
undervalued. Further, the basic flaws in the free-market model are being exposed: skill 
prices do not reflect uniformly the cost of their production, and do not move down as fast 
they move up. Even more fundamentally, we are learning that individual expertise is not 
necessarily the best unit for constructing a competitive organisation. 

DESIGNING A LEARNING ORGANISATION IN THE ANGLO- 
SAXON CONTEXT 

Obviously, organisations in North America or the U.K. cannot create a new educational 
infrastructure, nor change all the rules governing recruitment, deployment and 
remuneration in an environment shared with competitors, labour unions and governments. 
Consequently, their approach to creating organisations that learn must be very different 
from the highly orchestrated one which has dominated Japan since 1945. For the most 
part, they must work within the boundaries imposed by our current system of developing 
and deploying human resources. Several different trends arc detectable. 

One is to literally transplant Japanese-type operations. Most of the transplants seem to 
have successfully transferred many Japanese practices eg. the prize-winning Toyota plant 
in Cambridge (Romain 1991). But these cases have several features which suggest that 
their success may not be easily generalised: they are largely restricted to assembly 
operations, in the auto industry; these operations tend to be managed from Japan, by 
Japanese, as branches of large corporations; they are put together by very careful 
selection of workers, on the basis of education, attitude and union-affiliation. They have 
succeeded because they have minimised the impact of our established practices, 
effectively side-stepping the mainstream of our organisational culture. However, they 



DESIGNING ORGANISATIONS THAT LEARN 



7 



may have a broader impact by increasing competition, and demonstrating that alternative 
methods of organising work are not only possible, but sensible. 

Another trend has been to change the nature of the organisation indirectly, by examining 
all jobs, redefining their informational requirements, and then providing more or less 
traditional training, albeit on a massive scale. During the 1980*5, Motorola changed from 
an organisation that "...hired people to perform set tasks and didn't ask them to do a lot 
of thinking" to one which emphasises basic skills, problem solving in teams, and a focus 
on quality (Wiggenhorn 1990). This was accomplished by getting everyone involved in 
training programs, from elementary to university levels, working with educators to 
overcome shortcomings in the school environment (Wiggenhorn 1990). In short, 
Motorola became a learning organisation by breaking the monopoly of managers on job- 
related information and the authority to act on it. This approach was indirect in the sense 
that it created a corporate culture conducive to learning on the part of individual workers. 
Success is therefore measured in terms of their performance. Better performance of the 
firm as a whole is of course desired, but might be said to be an emergent outcome. 

On the other hand, a more direct approach is to change the organisation by renovating 
specific structures and procedures. In this case, success is measured by the collective 
learning of whole groups of people. One example of the direct approach is an outgrowth 
of the vast literature on "learning" or "experience curves" in production management, 
where it has long been known that manufacturing units improve the speed and quality of 
their work over time. Only recently, however, has anyone attempted to explore the 
factors determining whether and how learning actually occurs in these circumstances 
(Adler 1990). 

Adler conducted a case-study of the evolution of the design and manufacture of a high- 
tech product in a multi-national firm - in effect, an application of "concurrent 
engineering". He examined how changes in the flows of critical information among the 
firm's functional units affected the quality of processes and product. Broadly speaking, 
the firm had to replace some of its formal rules, which defined its structure in hierarchical 
terms, with more informal guidelines prescribing a more timely flow of information. This 
was accomplished by changing patterns of communication: by creating new sub-units for 
liaison, by creating "flying squads" to carry vital information to branch plants, and by 
job- rotation on a large scale. 

In Adler's terms, these changes were based upon a clearer perception of the degree to 
which these units performed differentiated functions, and the degree to which they were 
interdependent. For example, the firm created "centres of competence" to recognise and 
reinforce creativity at branch plants. These centres had a lot of autonomy, with rich 
internal communications, but little contact with other branches. On the other hand, the 
firm recognised the interdependence of the design and manufacture phases of production 
by rotating engineers between these two functions. 

Adler* s work is significant in two ways. First, it provides some empirical evidence of 
"shared learning", of deliberately created organisational capital. Secondly, it opens a new 
realm of research on organisational design, adding a higher level of analysis to the study 
of how training and other interventions improve an organisation's performance. 

IMPLICATIONS FOR QUALITY IN THE ORGANISATION 

In a broad sense, the key features in the cases described by Wiggenhorn and Adler 
replicate those in the Japanese approach: appropriate skills, deployed across traditional 
job boundaries, linked by optimal communication. We might call this "just-in-time skills 
delivery". 



17 



8 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



Can these approaches be adopted for large-scale changes in the way we organise work? 
In the Motorola case, it took ten years and an annual investment of the order of $50 
million to change corporate culture, work with schools to improve labour input, and 
develop a systemic program of job-related training. These issues concern some 
newsworthy companies, but one wonders whether the majority will respond. In Canada 
for example, private sector investment in training is extremely low by international 
standards, and the results are generally seen as inadequate (Quarterly Labour Market and 
Productivity Review 1990). 

In Adler's case, the changes in corporate structure are piecemeal, not necessarily rooted 
in cultural change, and so risk an abrupt end. As Motorola has learned, significant and 
sustainable change requires pervasive changes in attitude and behaviour. This must start 
at the top of the organisation and cascade down, so that managers understand and support 
change. In one major Canadian company which invests heavily in human resource 
development, upper managers initiated a program of decentralising training to teams, but 
suddenly changed direction when they felt control slipping away. Such are the risks 
which must be taken if our work culture is to change. 

My experience with joint- ventures where Japanese and Anglo-Saxon work cultures come 
into contact indicates that the most difficult changes involve re-examining the way in 
which we organise the flow of information and responsibility. To generalise, sharing 
knowledge and responsibility is first-nature in organisations built upon group processes, 
team-work, job-rotation. In fact, as indicated earlier, for the Japanese the point in 
organising teams is to minimise the risks of labour shortage and inflexibility. On the 
other hand, organisations founded upon the principle of expertise, that knowledge is a 
private, compensatable property of the individual, tend to create what have been called 
"chimneys", knowledge hierarchies wherein each level is privy to only that amount of 
knowledge required for a task prescribed by the chimney. The other side of this is that 
responsibility for the quality of performance is similarly limited: in short, everyone covers 
their own tail. 

These characteristics of the way we organize work thus mitigate against the very factors 
upon which quality of performance can be seen to depend: common experience, 
decentralised skill development, timely sharing of knowledge. If we are to begin to move 
towards the greater sharing of knowledge that the current interest in team -building 
implies, then significant changes in the way we organise work will be required. One is 
to push the knowledge and responsibility required to meet corporate goals down to the 
shop-floor. Another is to make sharing information and informal training everyone* s 
responsibility, not just that of the official training department. Japanese successes with 
a systemic approach suggest that these changes must lead to many others. 

References 

Adler P S (1990) Shared Learning Management in Science 36(8) August 

Becker G (1964) Human Capital: theoretical and empirical analysis with 
special reference to education Nat. Bur. Econ. Res. New York 

Dicks D J (1986) From Ships to Chips: manpower policy for industrial change in Japan 
Aspects of Educational Technology XIX Kogan Page London 

Dohse K, Jurgens U and Malsch T (1986) From Fordism to Toyotism? The social 
organization of the labor force in the Japanese automobile industry in Politics and 
Society 14 pp 45-66 



13 



DESIGNING ORGANISATIONS THAr LEARN 9 

Dore R, Bounine-Cabale J and Tapiola K (1989) Japan at Work: markets, management 
and flexibility OECD Paris 

Florida R and Kenney M (1991) Transplanted Organize r. the transfer of Japanese 
industrial organization to the US in American Sociological Review 56 pp 381-398 

Gertler M (1988) The Limits to Flexibility: comments on the post-fordist vision of 
production and its geography in Trans. Inst. British Geog. 13 pp 419-32 

Koike K (1988) Understanding Industrial Relations in Myiern Japan MacMillan 
London 

McMillan C (1979) Human Resource Policies, Labour Markets and Unions: Canada- 
Japan comparisons in Hay K A J (ed) Canadian Perspectives on Economic Relations 
with Japan IRPP Montreal 

Nagatani K (1991) Panel Participant Japanese Studies Association of Canada 
Conference Winnipeg October 

Nakamura Atsushi (1975) Intrafirm Wage Differentials reprinted in Nishikawa S (ed) 
(1980) The Labour Market in Japan University of Tokyo Tokyo 

Piore M and Sabel C (1984) The Second Industrial Divide Basic New York 

Prescott E C and Visscher M (1980) Organizational Capital in J Political Economy 88 
(3) 

Quarterly Labour Market and Productivity Review (1990) The Linkage between 
Education and Training and Canada's Economic Performance Winter Ottawa 

Romain K (1991) Toeing the Quality Line Globe and Mail 9 September p Bl 

Wiggenhorn W (1990) Motorola U: when training becomes an education in Harvard 
Business Review July - August 



JO 



2 Quality assurance in a European 
context 



David Alexander and Jim Morgan. Leeds Metropolitan University, Leeds, LSI 
3HE, UK 

INTRODUCTION 

"In the European Course Credit Transfer System (ECTS), a problem that is only latent in 
Erasmus becomes prominent, viz, that of calling all institutions of higher education 
universities, and allowing them to be in the same transfer system. The European Council 
Decision of 15 June 1987 stipulates that the term university shall be used to cover all 
types of post-secondary education and training establishments which offer, where 
appropriate within the framework of advanced naining, qualifications or diplomas of that 
level, whatever such establishments may be called in the Member States. Admittedly, 
there are non-universitary institutions of higher education in each EEC Member State that 
deserve a quality label as solid as that of the universities in the strict sense of the word. 
But as the Leuven Department of Mechanical Engineering has found out, this equal 
fooling can have its dangers, and may lead to legitimate reactions of guarding the 
indigenous quality. And this time* there is in the word quality an implication of better or 
worse, higher or lower because, within ECTS, the university does not have a choice as to 
which partners to accept or reject." Latre and Del Martino (1991) p6. 

"This paper has identified some of the specific items that are held to be indicators - they 
are often little more than surrogates - of quality. Though they are all deemed to assess 
quality in different national circumstances, they themselves vary considerably. There is 
no agreement on the purpose of quality assurance, save only as a resource allocation 
device or perhaps as a resource withdrawal device. Consequently, it is those features of 
an institution's performance which lend themselves to cross-institutional comparison which 
are tending to emerge. Often these are of a rudimentary nature, in comparison with the 
complexities of the inner life of institutions." CNAA (1991) 

MEASURING QUALITY - THE NUMBERS BIT 

This ought to be the easy part. Student results in one educational system need to be 
translated into ihe parlance of other educational systems. In pass/fail terms this may be 
fairly straightforward. But as soon as any form of classification or grading becomes 
involved - as in Higher Education it surely must - then considerable difficulties arise. 

The BA (Hons) European Finance and Accounting Degree at Leeds Business School 
is at present a 3-legged Erasmus supported ICP, involving Leeds, Hochschule Bremen, 
and ESC Normandie. Students spend 3 Semesters at "home", 2 semesters at one of the 
other institutions in full-time assessed study, and the final semester at home. Successful 
students receive 2 separate qualifications* 1 from each relevant country, The formal 
mechanism for this is that assessments are taken in Semesters 4 and 5 in the "foreign" 
language and cultural context, and also in Semester 6 in the "home" language and cultural 
context. This produces a single set of marks, which are then fed separately into 2 legally 
different sets of examination regulations. These regulations can have their own 
differences. For example, a student who has 2 attempts at an assessment, receiving marks 
of 30 and 32, would be regarded as failing the module for UK purposes. For German 
purposes the student would have the right of a further attempt. Conversely a student 
receiving marks of 30 and 38 could well be regarded as achieving a 40 mark for UK 



QUALITY ASSURANCE IN A EUROPEAN CONTEXT 



11 



purposes and therefore a pass, but such subjective adjustment could not happen at a 
Gentian examination board. 

More fundamentally, however, where do the numbers themselves come from? Table 
2.1 gives the agreed conversion scales used for students enrolling in 1989 and 1990 on 
the same course in the same circumstances. The relationships between the two years are 
significantly non -linear. We must remember that these are actual scales producing 
numbers which determine the degree classifications of actual individuals. It is clear that 
the decision as to which scale to use could influence degree classification or even 
pass/fail. 

1989 1990 



French 


CNAA 


German 


French 


British 


German 


Mark 


Mark % 


Note 


Mark 


Mark % 


Note 


20 


100 


1.0 


20 


85-100 


1.0 


19 


90-99 


1.0 


19 


80-84 


1.0 


18 


80-89 


1.0 


18 


75-79 


1.3 


17 


75-79 


1.0 


17 


70-74 


1.7 


16 


70-74 


1.3 


16 


67-69 


2.0 


15 


67-69 


1.7 


15 


63-66 


2.3 


14 


64-66 


2.0 


14 


60-62 


2.7 


13 


60-63 


2.3 


13 


55-59 


3.0 


12 


57-59 


2.7 


12 


50-54 


3.3 


11 


54-56 


3.0 


11 


45-49 


3.7 


10 


50-53 


3.3 


10 


40-44 


4.0 


9 


46-49 


3.7 


9 


38-39 


4.3 


8 


40-45 


4.0 


8 


36-37 


4.3 


7 


38-39 


4.3 


7 


34-35 


5.0 


6 


35-37 


5.0 


6 


32-33 


5.0 


5 


30-34 


5.0 


5 


30-32 


5.0 


4 


2029 


5.0 


4 


25-29 


5.0 


3 


15-19 


5.0 


3 


20-24 


5.0 


2 


10-14 


5.0 


2 


10-19 


5.0 


1 


0-9 


5.0 


1 


0-9 


5.0 



Table 2.1 Examination mark conversion scales 

How arc these scales prepared? The intention is that, taking one cohort of students with 
another, there should be the same number of firsts, upper seconds, etc from each 
country's batch of students. Or, to put it the other way round, that a student should have 
the same chance of a first whichever his home and foreign institutions. 

Given the intention, the detail follows on a rough and ready statistical basis, supported 
by discussion and subjective judgement between colleagues from the institutions. The 
crucial difference between the two scales, of course, is the minimum pass mark for each 
module in France and the UK. There was a widespread view amongst staff that the 1989 
scale was producing wrong results. Notice incidentally that the long tails in the German 
Note present obvious difficulty. A 5.0 has to be translated into a specific CNAA number 
for condonement and averaging considerations. Our German colleagues have no such 
problems, as condonement is not possible under their exam regulations. 

Leeds Business School is about to extend this programme to a fourth partner, at Pescara 



12 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



in Italy. The first stage in the process of producing a fourth column for kaly in the 
scales is to establish typical Italian thinking in first-degree assessment and grading. 

In the Italian Laurea System results are based on an overall mark out of 110. Each 
course is given a mark out of 30 if it has been passed (the assessments are nearly always 
orals). A bare pass is given 18, a failure is not given a mark. Resits are possible without 
limit (but not to raise a passed mark). The marks from as many as 30 courses are then 
averaged, with equal weighting. This average is then divided by 3 and multiplied by 11, 
to give a mark out of 110. The thesis is then presented and defended in public, and 
marked by a jury of 1 1 , out of ten. This mark is then added to the mark out of 1 1 0, and 
the total is regarded as a mark out of 110. The theoretical maximum mark out of 110 
is therefore 120, but a total over 110 is regarded as 110. It is this final mark out of 110 
which gives the Laurea its "classification". 

The task of moving from this starting point, to an extended fair and equitable 
Conversion Table, is one we approach with some interest! 

BEHIND THE NUMBERS - WHAT ARE WE TRYING TO 
MEASURE? 

It might appear from the discussion so far that the major problem is to create the best 
possible numerical conversion scale. In other words it might seem that there is a 
theoretical "right" version of the scales, and that the problem is that we can never more 
than approximate to it. This conclusion would be grossly misleading and optimistic 
however. A "right" version of scales only exists even conceptually or in theory if there 
is a commonality of view as to what it is that we are trying to measure. There is 
absolutely no such thing. 

Education, educational process and educational measurement are all subjective 
processes. They are defined and monitored by human agents. Tutors are entitled to be 
as individualistic as learners are. But that is not the point here. 

In any particular community or any particular culture the majority of members will tend 
to concentrate round the norm as regards any particular defined dimension. In the 
educational sphere we might hypothesise, for example, that the majority of an educational 
community will tend towards a broad agreement on the issue of the importance of theory 
and practice, and of the desirable balance between the two elements. The key issue, 
however, is what is meant by the phrase "an educational community". It is suggested 
here that, bearing in mind we are talking in a pan-European context, "an educational 
community" will often need to be defined fairly narrowly for the above hypothesis to be 
valid. This narrowness may well apply in both geographical and discipline focus terms. 
"German higher education" is too broad. "Business higher education" is too broad. 
"German business higher education" may be narrowly enough defined to justify 
generalisations. 

We are led to the hypotheses that: 

1 different discipline focus areas may, in general, have significantly different 
higher education aims and philosophies, and therefore practices; 

2 different geographical/cultural regions, within a single discipline focus, may 
have different higher education aims and philosophies, and therefore practices. 

In the context of student mobility, the latter issue is probably the more significant one in 
its effects. By way of illustration we briefly discuss here some perceptions of business 
education, and some particular experience with dissertations. 



2~ 



QUALITY ASSURANCE IN A EUROPEAN CONTEXT 



13 



The major French Business Schools are some of the most high profile higher education 
institutions in Europe. They are characterised above all else by close integration with the 
real business world and by a strong emphasis on practical application. This is 
encapsulated in the commitment to the case study as being the major learning mechanism. 
Work-placements are crucial. All this has a negative as well as a positive side. The 
emphasis can sometimes be on practice, rather than on theory and its application to 
practice. There is a strong interest in group work, personal skills, and synthesis of 
practices. There is less interest in reading, in theory, and in personal intellectual 
development. The French Business Schools and their products (they should not be 
confused with Economics-related departments at French Universities), have enormous 
importance in French business activity. 

Germany has no such tradition, there is no longstanding German equivalent. But it is 
difficult to argue on empirical grounds that the German economy and business community 
have been disadvantaged thereby. The approach to business education in Germany is 
much more "academic". In our experience this approach is characterised by emphasis on 
libraries, o?> reading, on lecture or lecture and exercise, rather than on seminar or case 
study. Work placements may not be involved. 

UK practice is probably somewhere in the middle, with named "Business Studies" 
degrees attempting (usually very broadly) to adopt the work-related French practices, 
whilst more specialist degrees may tend to a more library-based approach. 

These generalised differences are well illustrated and demonstrated in the approach to 
dissertations we have discovered on the Leeds Business School European Finance and 
Accounting Degree. In summary, we can suggest the following: 

• Germany - library based, long, detailed, ("Fussnoten Fetischismus"), personal 
opinion neither encouraged nor expected; 

• France - real-life, empirical base, little interest in theory; 

• UK - library or library-empirical, personal opinions and appraisal regarded as 
essential; 

• Italy - extremely rigorous, theoretical, assessed on a public presentation 
(usually called a "defence" which is perhaps indicative). 

We might suggest there is very little commonality here as regards educational criteria. 
So how can we monitor quality across the whole multi-country programme? 

The numbers conversion problem discussed in the previous section can in theory be 
eliminated by agreeing a single common marking scale across all partner institutions. But 
this does not get us very far unless we can also create a single coherent pan-programme 
statement of the desirable characteristics (and their relative importance) of a "good" 
dissertation. This means, in other words, doing away with national (ie geographical or 
cultural) differences. 

The other possibility is to say that these differences are expressed in different 
dimensions and are therefore not ultimately comparable. No one dimension can be 
translated into the measurement units of the other. (This room has the better shape, that 
room has the bene lighting. Which is the better room?) We simply accept - or reject - 
that these very different skills, objectives and attainments are of similar quality. (Both 
are good rooms.) This can be done by measurement of inputs, the use of surrogates, or 
by overt subjectivity. This last may not be at all a bad idea. In other words we are 
recognising the inevitability (and perhaps the desirability) of "national" (geographical and 
cultural) differences. 

There can be no fudging of the issue, no half-hearted compromise. We cannot, with 
any semblance of objectivity, cross-monitor quality by an output measure unless we adopt 
a single European Education Currency Unit. 



14 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



CURRICULUM DESIGN 

The characteristics of the system for curriculum design in England may be summarised 
as follows. 

a) There is no overt state regulation, though the government exercises influence, 
or attempts to, through its funding and student maintenance policies. Even 
here it is acting through intermediaries - PCFC, UFC, Local Authorities. 

b) Responsibility for approval of the curriculum rests with individual universities and 
for a little longer with CNAA, BTEC. These bodies provide only a loose 
regulatory framework - loosest in the universities and tightest with BTEC. 

c) Curriculum design rests with a team of academic staff who normally act 
within the assumptions and traditions of their subject or professional area. 

d) Approval involves an element of peer, and normally, external peer, review. 

By contrast continental Europe is characterised by state regulation of the curriculum and 
a lack of external peer review. 

In Finland the basic degree (equivalent to a UK Masters) is regulated by Council of 
State decrees which lay down the objectives, structures and conditions for awards in 19 
fields of interdisciplinary and vocationally relevant study. It is a credit based system with 
the basic units of study classified into general, subject and advanced. 

However, within this framework the syllabi and assessment of units are a matter for 
professors, with the approval of a representative Departmental Council. It is very unusual 
for this Council to question, let alone reject, a Professor's proposals. 

In Italy the basic degree of Laurea lasts four years. The Ministry of Education approves 
the examinations which constitute a particular Laurea. When a new Laurea is proposed 
it receives the approval of a university and then has to be enacted by the Ministry. 

However, this central control is more apparent than real. The Ministry names the 
examinations but individual professors have great autonomy ir determining the content 
of the curriculum and assessment. In Engineering, however, there is a national system 
in which the first two years are largely common to all universities as a result of pressure 
from industry and the professions. 

One might conclude from this brief survey that the difference between the devolved 
approval systems of the UK and the state regulations of continental Europe is more 
apparent than real. Paradoxically, the real contrast is between the practical professorial 
autonomy in Finland and Italy and the practice of peer review in England. 

COURSE MONITORING 

In England this is normally conducted through some form of regular monitoring meeting 
by staff (and sometimes students) and through the external examiner/moderator system. 

Germany and Italy do not have any formal, regular monitoring. Germany, however, 
conducts a formalistic system of assessment with the moderation of examination papers, 
double marking and final determination by an elected Examinations Committee. Each 
subject is required to produce an annual report on student performance to the elected 
Departmental Council which includes student representation. However, there is no 
external moderation. 

In Italy much assessment is still oral and conducted in theory by three professors, one 
of whom is a specialist. In the early years of the Laurea this seems to be increasingly 
compromised because of the pressure of student numbers, leaving decisions to the 



V 



QUALITY ASSURANCE IN A EUROPEAN CONTEXT 



15 



individual professor's judgement. There is no monitoring system. Informal student 
pressure is exercised through their choice of examinations to study - by avoiding the 
poorly taught or severely examined - or, at the final resort, through protests and strikes. 
Again there is no external moderation element. 

Once again the obvious contrast is between the autonomy of German and Italian 
professors and external peer moderation in England. 

CONCLUDING ISSUES 

These brief excursions into only some of the realities that lie behind the generalities of 
quality assurance systems illustrate the practical issues that have to be addressed in the 
context of pan-European higher education. Behind these practical issues lies a more 
fundamental difference between the English use of external peer review and the 
continental reliance on the professional autonomy and integrity of the professor. Perhaps 
even moar fundamentally England has a determination to assure an equivalence between 
institutions at variance with continental practice, which accepts the existence of formal 
and informal hierarchies. 

At the operational level of delivering joint programmes and exchanging students the 
essential issue is one of standardisation or equivalence. Arguably the only practical 
option is equivalence and, even more arguably, acceptance of the practices of each other's 
institutions. 

When we look to the future we should also observe that continental European higher 
education provides for a mass student market. Can England's expensive quality assurance 
procedures survive the arrival of mass higher education? And should it? In the last 
analysis quality does depend upon the professional competence and integrity of individual 
tutors and professors. Arguably no system assures real quality. And perhaps market 
perceptions could, and should, be the final judge? 

References 

Council for National Academic Awards (1991 ) Models of Quality Assurance in Europe: 
discussion paper No 6 CNAA London 

Latre G and Del Martino F (1991) Traditional Universities in New Europe: the case of 
K U Leuven TEXT Conference paper Toulon September 16-19 



3 "What's in it for me? 11 Organisational 
culture, rewards and quality 



Christopher Hart and Michael Sholbred, University of Central England at 
Birmingham, Birmingham, B42 2SU, UK 

INTRODUCTION 

Quaiity systems generate a quality culture and this is what is at the heart of successful 
organisations. Here we have examined some of the literature on successful quality 
organisations. We have followed this path because both manufacturing and service 
industries spend a great deal of time addressing the topic of cultural change. We believe 
that if further and higher education are going to make serious moves towards effective 
quality assurance, whether through a QMS (Quality Management Systems) or a TQM 
(Total Quality Management) approach, they need to be aware how much the culture may 
have to change. This may be highly uncomfortable for senior management. For instance, 
a 1989 survey of 3.000 company quality schemes in the UK identified one major 
problem: 

"Almost all companies have fouM it difficult to achieve cultural change and to get 
management, especially lower and middle management, to change their ways of working" 
(Develin 1989). 

As we move towards quality in HE, everyone will have to make changes. 

WHAT ARE ORGANISATIONAL CULTURES AND 
CLIMATES? 

After identifying no less than 164 different definitions of culture, Kroeber and Kluckhohn 
(1952) conclude that: 

"Culture consists u patterns ... of ... behaviour acquired and transmitted via symbols, 
constituting the distinctive achievement of human groups, including their embodiments in 
artifacts; the essential core culture consists of traditional (ie historically derived and 
selected) ideas and especially attached values; culture systems may, on the one hand, be 
considered as products of action, on the other as conditioning elements of future action/ 

In the 1980s and 90s we have rather snappier definitions. "Culture is the set of values, 
behaviours and norms which make an organisation tick" (Atkinson 1990). Snappier still, 
Marvin Bowers* immortal quote that organisational culture is: "the way we do things 
around here" (Deal and Kennedy 1982). 

An organisation's culture takes years to mould, develop and grow, and years to change. 
An organisational climate, on the other hand, is described by Botterill (1990) as: "the 
collective current impressions, expectations and feelings of the local work units". 

The climate of a unit or department is the "now". It can change instantly. A sharply 
worded memo or a derogatory remark in a meeting or overheard criticism can, on 
occasion, radically change the climate of interaction. Climatic attitude is the day-to-day 
relationship the individual (and sometimes the group) has with others in the organisation. 
Unlike climatic attitude, culture is more enduring and stable. Some key characteristics 




ORGANISATIONAL CULTURE, REWARDS AND QUALITY 17 



of that culture are that it is: commonly held rather than shared, real only in its use, ie 
consequences, historically based, learnt, heterogeneous. 

What indications are there of corporate culture? These are many and varied and in HE 
they might include: specific management styles, "telephone style within the organisation, 
willingness to give personal tutorials, how staff spend their lunch breaks, feedback to 
students, teaching styles, tidiness in offices, graffiti in the loos, quantity of visual 
stimulation on the walls. 

All these are possible indications of organisational culture within an educational 
environment. They can be positive indications of an organisation with a quality culture. 

WHAT IS DIFFERENT ABOUT QUALITY ORGANISATIONAL 
CULTURES? 

From our own observations and from our researches we are quite certain that there are 
some major differences between the culture of organisations which are held to be 
successful and which present a quality image and those of other organisations. What we 
are now going to describe is surely as true for further and higher education as for any 
other service organisation. 

Firstly, quality organisations tend to have quite specific quality values, whatever sector 
of the market they are working in. 

In many organisations values may not be explicit. Perhaps further and higher 
education are different in this respect. Staff will frequently be aware of their own 
personal value systems and they will sometimes contrast those with the mission 
statements of their organisation, or with management actions which seem to belie those 
mission statements. 

So what kind of quality values underpin quality behaviours? Linkow (1989) suggests 
seven core values: 1. customer focus, 2. employee focus, 3. teamwork, 4. safety: for 
employees, communities and users-of products and services, 5. candour v 6. total 
involvement, 7. process focus: emphasising the continuous improvement of all processes. 

This is by no means a complete list. We might want to add to it such values as: 
intimacy, integrity, unity, consensus, excellence. 

We believe that it is absolutely essential that values in higher education are made 
explicit and that there is both the unity of purpose and a unity of belief between 
managers, teachers, administrators and students. 

RITES AND RITUALS, MYTHS AND LEGENDS 

One increasingly common way of examining the culture of an organisation is to look at 
its manifestations through such areas as heroes, rites and rituals (Linkow 1989). Heroes 
are an important influence on behaviour in an organisation. Heroes are individuals who, 
in deed and presence, personify and transmit the culture's values. They provide tangible 
cues and act as role models for others to follow. From our own varied experience we 
have come across a number of heroes. Comments from colleagues on their heroes 
include: 

"She's so laid back with the HMJ" 
"The guy was a walking libraiy" 

"Everything he says carries so much force and conviction" 

"She stepped in at the last moment and gave a key note speech better than anyone else 
could have dene!" 

"He showed a slide of a field and had students enthralled with tales of Eric Blood Axe" 



18 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



Heroes are influential within the organisation. Most are situational heroes. They don't 
charge out of the trenches but make the right movement at the right time. They are the 
stuff that organisational legends and exaggerations are made of. A commonly 
acknowledged fact of higher education is the preponderance of anti-heroes over heroes. 
Anti-heroes are often the focus of gossip. They are the deviants of the organisation. 
Their deeds signify boundaries of what is and what is not currently acceptable behaviour 
(Atkinson 1990). When office doors are shut, stories of anti-heroes are often told, such 
as: 

"She skives off all the time" 

"He is never around when you want him!" 

M He has got a private business on the side" 

"Power has gone to his head" 

"She abuses the rules" 

"He sleeps with students" 

Rites and rituals (Trice and Beyer 1984) are the programmed and routine interactions of 
organisations. In higher education we have numerous rituals including the annual 
"induction ritual" for new students. This cyclical ritual minimises disruption and shows 
newcomers the kind of behaviour expected of them. 

Common rituals for members of the Department are, "the staff meeting" or "the Board 
of Study". A particular ritual of higher education in recent years has been the "touching 
base" meetings ritual. This consists of staff, often from diverse Departments, being called 
to meet and "look at" some project or other. Vague indication is given that some 
document or other needs to be produced and this cohort will produce it. For instance, 
the "Faculty Planning Group" and the "Research Strategy Group" are common groups 
throughout higher education. They meet and, without much experience or direction, 
manage to produce a document. Once produced, members are seen to have "served their 
time on this one". 

Rituals in higher education are widely experienced as lacking direction or clear 
integration with the values of the organisation. 



WHAT'S YOUR OWN ORGANISATIONAL CULTURE 
REALLY LIKE? 



Values are the bedrock of any culture. They provide a sense of common direction for 
all members of the organisation and guide day-to-day behaviour. The following exercise 
will allow you to look at the pattern of values in your organisation. It will assist you in 
understanding where your organisation has got to on the road to quality values. Go 
about the exercise by using the following instructions with figure 3.1 



INSTRUCTIONS 
Look at the list of values 
in the box "zones of relevance' 
in figure 3.1 

Tick up to 7 values that you 
believe represent the values 
held by each role holder 



COMMENTS 



1 



Some represent values 
of quality practice 



2 



Play yourself and take 
the role of the other 
Understanding other 
people's role is crucial 
to quality 




ORGANISATIONAL CULTURE, REWARDS AND QUALITY 19 



3. Add up the score for each role High scores indicate 
placing the total at the end a base from which quality 

of each row values can be developed 

Low scores indicate need 

for change in values 

From our experience of working with several institutions of higher education we have 
noticed an inverted relationship between the values and expectations of management and 
those of staff. Where staff widely practise quality values and expect the kinds of rewards 
associated with such values (see Vroom 1964 on "expectancy theory") management has 
yet to embrace these values and recognise the expectations they create. We conclude that 
the kinds of values held by some management are a major barrier to a quality 
organisation. If management does not address these issues we predict that polarisation 
will increase, leading to entrenched positions between management and staff. The result 
will be internal conflict - the antithesis of quality. 



Zones of Relevance 
(Degree of attachment to values) 



Blackend Institute /& . 'J f . '/. /. >°.^. '/. '/. '/. >?. ' £f 
of Higher Education * . ff . /. f. /. g. ^ 

Roles 12 11 10 9 8 7 I 6 5 4 3 2 1 Score 



Roles 


12 


11 


10 


9 


8 


7 


6 


5 


4 


3 


2 


1 


Score 


YOU 




























THE DEAN 




























THE DIRECTOR 





























Figure 3.1 Matrix of values 



WHAT'S IN IT FOR ME? 

As further and higher education institutions move towards a quality culture, regardless 
of the system that they use to introduce that culture, they will need to reward and support 
their staff. If staff are expected to work towards targets, levels of performance and 
diversify their activities, they will expect rewards. Management will need to recognise 



20 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



this expectation and address it with equity and openness. Contrary to popular belief, 
management will not need to attend to motivating staff, but to harnessing the values staff 
hold as the foundation for organisational quality. 

In commerce, the drive for customer satisfaction is often linked with remuneration. As 
yet, further and higher education in this country is only just beginning to grapple with the 
introduction of performance related pay for middle management and other staff. If staff 
are to work hard towards promoting quality they will surely need both rewards and 
recognition. How are these to be achieved? Rewards and recognition might come 
through: training and staff development, an improved working environment, improved 
mutual respect, some form of financial remuneration. 

CONCLUSION 

We've suggested that if further and higher education are to deliver quality systems then 
they will need to examine their cultures. Quality in education is concerned with having 
an appropriate range of services, and getting the service right on each and every occasion. 
Clearly, quality costs. But in making decisions about quality, consider the costs of 
ignoring quality: poor communication, misinformation, fear and insecurity, lack of 
direction, crisis management, lack of information flow, too many conflicting priorities, 
no identifiable mission, "make it cheap" attitude, time wasting in all forms, time table 
changes at short notice, crisis cycles. 

Avoiding the waste and demoralisation of traditionalism necessitates management 
change. Senior management needs to get its internal customer relationships right if it is 
to provide excellent quality relationships with its external customers. 

An appropriate and supportive culture is at the very heart of a quality organisation. The 
entire workforce must have some kind of common identity. 

"All employees must believe in the company itself, its culture and their own importance 
to the overall success of the product or service. This belief must emanate from within the 
leadership where the key to corporate culture is PUTTING YOUR PEOPLE FIRST. Your 
employees are your greatest and only appreciable asset, and your treatment of them will 
be reflected in the quality of both product and service" (Botterill 1990). 

Ultimately, it is that sense of corporate belief and shared values which will distinguish 
quality organisations in education. That may be what is in it for you. 

References 

Atkinson P F (1990) Creating Cultural Change in Management Services 34(11) pp6-l0 

Atkinson P F (1991) Creating Culture Change: The key is successful total quality 
management IFS Ltd Kemptson. Bedford 

Botterill M (1990) Changing Corporate Culture in Management Services 34(6) pp 14-18 

Deal T and Kennedy A (1982) Corporate Cultures; the Rites and Rituals of Corporate 
Life Penguin London 

Develin N (1989) What's Wrong with British Quality? in Quality Today November 
pp 44-50 



3 j 



ORGANISATIONAL CULTURE, REWARDS AND QUALITY 21 

Kroeber A L and Kluckhohn C (1952) Culture: a Critical Review of Concepts and 
Definitions Vintage Books New York 

Linkow P (1989) Is Your Culture Ready for Total Quality? in Quality Progress 22(11) 
pp 69-71 

McClelland D C and Burnham D H (1974) Power is the Great Motivator in Harvard 
Business Review 54 pp 100-110 

Trice H M and Beyer J M (1984) Studying Organisational Culture Through Rites and 
Ceremonials in Academy of Management Review 9 pp 653-659 

Vroom V (1964) Work and Motivation J Wiley New York 

Walsh S P (1990) Successful TQM Training in TQM Magazine 2(3) pp 165-169 



4 Quality and the academic 
administrator 



Catherine D Payne, Leeds Metropolitan University, Leeds, LSI 3HE t UK 
INTRODUCTION 

It is self-evident that the quality of a product or service is underpinned by the efficiency 
and effectiveness of all the operations that contribute towards the product or provide the 
service. Thus quality initiatives which seek to improve efficiency and effectiveness, 
particularly TQM, have an impact on all groups of staff but each group may view some 
of the issues differently. Whilst it is recognised that all staff are involved to a greater or 
lesser extent in administrative functions, the academic administrator can be seen as the 
facilitator between the teaching staff as the deliverers of the primary service (courses) and 
management as the providers of policy and strategy. Here I shall draw on research into 
the efficiency and effectiveness of Course Monitoring and Review process in the School 
of the Environment, Leeds Metropolitan University. 

There are many questions raised by consideration of TQM in this context. I shall focus 
on three only: 

• how to establish the internal customer/supplier relationships; 

• the use of performance indicators; 

• the application of TQM principles in a non-TQ environment 
INTERNAL CUSTOMER AND SUPPLIER RELATIONSHIPS 

The principle raison d'etre of a TQM organisation is to ensure that its product/service is 
fit for its purpose and satisfies customer requirements in order that they will repeat their 
custom and encourage others to be customers thus ensuring the continuing enterprise of 
the organisation, hopefully at a substantial profit. Within the education context we know 
that our principal customers - students - require a good quality education that will fit 
them for their chosen career. It is the Institution's responsibility therefore, to ensure that 
courses are fit for that purpose both through the curriculum and delivery. However, 
Atkinson (1990) argues that the focus on the external customer must not be at the 
expense of ignoring the internal customers and finding out their requirements. 

As part of the research into Course Monitoring and Review, interviews were conducted 
with staff in various functional roles including managers, teachers and 
administrative/clerical staff. A question about the external customer and internal 
customer formed part of the semi-structured interview. Whilst everyone understood very 
clearly who was the external customer(s) - primarily the students (but also employers, the 
professions. Government) - the concept of an internal customer was not so clearly 
perceived. However, as Leeds Metropolitan University is not as yet a practising TQM 
culture this is perhaps not so surprising. 

Who is our Internal Customer/Supplier? 

John Oakland (1989) states that: "The next person who checks [uses] your work will be 
your customer" p 243. 



QUALITY AND THE ACADEMIC ADMINISTRATOR 



23 



Like many of the principles of TQM this is very simple to state. It is suggested that 
the reality is much more complex. The many inputs and outputs to the process of Course 
Monitoring and Review typically include: employers, validating bodies, professional 
groups, HMI, external examiners, students, staff, committees, boards, student services, 
academic standards. These may be said to represent the myriad internal customer and 
supplier relationships that exist for this process alone. If we superimposed on this the 
numerous other activities that we are involved with we would see that, whilst some 
customers and suppliers would be removed, others would be added. Indeed even this list 
has failed to include those support staff who provide the domestic arrangements for a 
review. 

Professor Oakland (1989) uses his Quality Chains diagram to illustrate the links through 
an organisation from external supplier to external customer (see figure 4.1). 



Q 




Outside organization 



Supplier/ 
Customer 



, External customer 



Supplier/ 
^ Customer 

Supplier/ 
^ Customer 

Supplier/ 
Customer 

Supplier/ 
Customer 

Supplier/ 
Customer 

Supplier/ 
Customer 

Supplier/ 
i Customer 



Supplier/ 
Customer 



External supplier 

Outside organization 
Figure 4.1 The quality chains 

It could be argued, however, that the customer chains obtaining within HE institutions 
do not and cannot take such a simple linear form. For instance, a senior administrator 
in a Faculty works with managers, other administrators and teaching staff. Sometimes 
our relationship to one or other of these groups is as a supplier and sometimes as a 
customer. Often they can both be in the same process! An example from my own 
experience would be: to produce the School Examination Timetable the teaching staff 
have to supply me with their requirements. I piece the jigsaw together which necessitates 
liaison with a central Polytechnic administrative office - I supply information to them, 
they make accommodation bookings for me (customer). I produce a timetable for my 
customers, the teaching staff (previously suppliers), and ultimately for the external 
customer the students. 



C kJ 



24 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



What is suggested by this example therefore is that the notion of internal customer and 
supplier fails to take into account that these relationships flow in both directions, at least 
certainly for the administrator who is caught in the middle! This suggestion is not 
intended to be an argument against the value of establishing internal customers and 
suppliers and finding out their requirements but to suggest that as we do not work in a 
production line outfit, a different interpretation is perhaps required. For instance, quality 
chains may be more appropriately applied on an activity basis as opposed to a role basis. 
Figure 4.2 shows how this interpretation has been applied to the activity of examination 
timetabling. 







EXAM TIMETABLING 








Me as Customer 




Me as Supplier 




s 


Teachers 


Quality of 


Awards & Exams 


C 


u 




Service 




u 


p 


Awards & Exams 


dependent 


Draft/Secretary 


s 


p 




upon 




T 


L 


TT wordprocessed 


accuracy 




O 


I 




and 


Teaching Staff 


M 


E 




timeliness 




E 


R 




of data. 


Students 


R 


S 








S 



Figure 4.2 Internal customers /suppliers of exam 
timetabling process 



This approach helps unravel the complex links of a non-linear chain. Once unravelled 
we can more easily answer the key questions: 

• Who are my customers for this activity? 

• What are their requirements? 

• Who are my suppliers for this activity? 

• What are my requirements? 

PERFORMANCE INDICATORS 

The extent to which we do or do not meet requirements is the performance indicator (PI) 
of the quality of our service in our particular role or function. 

Once the internal customer/supplier link is established the next stage is to establish the 
precise requirements in that relationship. We are probably all too familiar with how the 
requirements of a particular role or task are so often fogged by expectations on the one 
hand and assumptions on the other. We have probably all heard and used the expression 
"But I thought you were supposed io be doing that." When roles and requirements 
remain undefined it is inevitable that this will happen. 

TQM gurus use the expression "right first time" (RFT). If this is the ultimate 
performance indicator why is it so difficult to achieve, bearing in mind that Performance 



31 



QUALITY AND THE ACADEMIC ADMINISTRATOR 



25 



Indicators (Pis) should be achievable? Is it an appropriate PI given the nature of our 
tasks and the environments in which we work? The days of tradition and stability are 
over, replaced by a high-speed culture that is constantly changing, leading to the 
phenomenon of "grey stress" (Bedell 1992) as we come under pressure to adapt and 
perform. Does this culture permit the time and space to perform RFT? 

Not everyone is convinced that TQM is the best thing since sliced bread. RFT would 
make all our lives easier - no more redrafting of notices of meetings because you got the 
date wrong, because you were interrupted by the telephone and someone came to your 
desk wanting just a quick word; no more redoing the student numbers spread sheet for 
the nth time because some enrolment forms have been discovered on a staff member's 
desk three months after enrolment; no more redrafting the Course Document because it 
has been decided to change the sequence of sections. Yet these events are the realities 
of our daily lives. Indeed it is as much a part of our service to internal customers, as to 
external customers, to respond to requests and change. Is RFT not then a rather 
sanctimonious view of an ivory-tower theorist? Add to this the patronising comments 
from senior management such as "we never have the time to get it right first time but all 
have the time to get it right second, third time" and RFT as a PI can really stick in your 
throat. 

What is the nature of this PI and others within our real environment? Are they not in 
fact a stick with which to threaten rather than motivating objectives? Certainly at the 
macro level they are used as such by the funding bodies. If objectives and targets are not 
met then fu.ding is reduced. In this regard Pis take on the quota function of the 
production line. The answer to these criticisms of course is that it all depends on the 
culture in which Pis and all the other principles of TQM are applied - the subject of my 
final section. 

Ignoring the arguments against Pis it is contested that they are still a very problematical 
area for the academic administrator. For Pis to be non -subjective they have to be 
measurable, ie. subject to SPC (statistical process control). What sort of Pis could be 
applied to the administrator? 

A few suggestions might be: turnaround time - how long for a specific task?; 24-hour 
response time for standard information; accuracy; sensitivity to different perspectives; 
value as an information source; use of secretaries. 

For Pis to be realistic and achievable they must be looked at in terms of the internal 
customer/supplier relationships and the number of those that exist. If you have a 
reputation for being good then you attract more customers but how many customers can 
be realistically and effectively dealt with before the service is reduced and your reputation 
tails off - as with a business over-extending itself? It may actually be necessary to 
reduce customer voiume in order to improve quality. (Heresy!!) 

There is another difficulty consequent upon applying Pis in a non-TQ environment. 
Whilst they may be legitimate to measure the performance of an organisation, in our 
present culture they become personalised - they become focused on an individual rather 
than on the structures/teams in place to support that individual in their activity. Taking 
account of the above limitations, the reality of our present (non-TQ) lives could counter 
each of the suggested Pis as follows: 

• turnaround time 

... four weeks for an Exam Timetable - what if the Dean wants a breakdown of 
student numbers by domicile in the meantime? 

* 24-hour response time for standard information 

... what is standard? If its not in "the book 1 ', is it not standard? What 
happens when the person who can give you the information is away till next 
Tuesday? 



26 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



♦ accuracy 

... no argument on this provided we work in an environment that is conducive to 
concentration and checking 

♦ sensitivity to different perspectives 
... how do you measure sensitive? 

♦ value as an information source 
... how do you measure value? 

♦ use of secretaries - this is a real one! 

... how many times do we send a piece of work back because we have 
redrafted it as opposed to there being typographical mistakes to be corrected? 

Ideally - in a TQM environment - none of these provisos should apply. In the meantime 
the reality is that they do. 

TQM PRINCIPLES IN A NON-TQ ENVIRONMENT 

Readers will probably know that TQM is a philosophy which sets out to achieve optimum 
efficiency and effectiveness by reducing waste and error through the application of certain 
principles and operating methods to achieve quality in every aspect of an operation to 
satisfy external customer requirements. 

So far I have focused on just two of the principles/operating methods required for a 
TQM culture - establishing internal customer/supplier relationships and the use of 
performance indicators. However desirable, it is suggested that these principles cannot 
be applied in isolation of an all-encompassing TQ culture. I shall now explore some of 
the key features that are required for this culture. 

Leadership and Commitment 

Quality is a strategic issue and an attitudinal position. The decision to go down that road 
must be led by the Senior Managers of an organisation for it is from this level that 
attitudes in an organisation are inculcated. It is through the example and commitment of 
Senior Managers to quality that the whole organisation is enabled to adopt a quality 
ethos. 

Resources 

The provision of resources for education and training is crucial - to educate people in 
ways of working in a quality environment and training to practise error prevention rather 
than error detection; in the use of problem-solving techniques and tools. 

Another valuable resource is time - time to do the job RFT. Time management courses 
may help but consideration also needs to be given to the actual management of staff 
resources to support the various activities. 

Communication 

Open communication is a crucial feature of a TQM culture. In order for employees to 
feel committed to an organisation's objectives they need to know and understand what 
these are. Decisions are made with regard to objectives - how to implement them, why 
they nrght change, how they bear upon an employee's place and operation of work. 



QUALITY AND THE ACADEMIC ADMINISTRATOR 



27 



Employees are the implementation channel through which an objective is met. In a TQM 
culture, however, communication goes further to become an interactive process leading 
to participation. 

Participation 

An organisation may have very good channels of communication - newsletters, up-to-date 
notice boards, regular policy statements - but if people are not involved in the decision 
making processes a sense of belonging and commitment is not cultivated. A non- 
participative style of management perpetuates an "us and them" syndrome. My research 
showed evidence of this attitude at Leeds Metropolitan University - between teaching 
staff and management, teaching staff and administrators, administrators and managers. 
Each perceived the other group as taking decisions which affected their lives but without 
reference to them. "Old style" management might consider that it is the prerogative of 
management to make "managerial decisions". TQM does not sustain this prerogative. 
Instead it recognises that participation has a lot to do with an individual's sense of power 
and control. Powerlessness undermines confidence and motivation and does not inspire 
a sense of responsibility. Participation brings about a sense of involvement, identity and 
responsibility - for the task and the objective. 

Team Building 

Establishing a culture of open communication and participation can be a foundation for 
team building. The value of team building has often been stressed in management 
handbooks. In TQM, teams may come under the guise of "quality circles", "problem 
solving teams". Whatever their guise it is suggested that they can achieve more than their 
purpose at hand if they are cross- functional and/or cross-departmental. They help build 
on participation and involvement. They help build communication. A suggested quasi-PI 
was sensitivity to another's perspective. I can iiave great empathy with fellow 
administrators. I know "where they're coming from", what their problems arc. This 
helps establish a strong cooperative spirit. Teams can help establish empathy across 
different roles and functions, it enables us to get to know each other better and our 
different perspectives. This increased awareness reinforces the principle of internal 
customer and supplier. 

These selected basic principles of TQM are in themselves not that original in 
management theory. But it is the combination of them in a specific context that makes 
TQM different and so very difficult to achieve. It is difficult to implement because it is 
a very long term and continuing process, requiring wholehearted commitment and. 
probably, an initial outlay of resources. More significantly however, it requires a 
fundamental change to most organisations' corporate culture: from a closed culture that 
is secretive and dominated by coercion and intimidation, resulting in staff being defensive 
and non -critical of performance, to an open culture where fear is driven out, barriers 
broken, targets eliminated. Where people are self-critical and take pride in their 
workmanship - these are the values which provide the impetus for improvement. 

CONCLUSIONS 

My conclusions are simple. Quality can be a personal standard. Right first time can be 
a personal standard within the constraints of an individual's environment. TQM is posed 
as the panacea to lift those constraints so that quality can be applied across the board. 



28 PERSPECTIVES ON QUALITY IN ORGANISATIONS 

But does TQM pose more questions than it answers? A few have been suggested from 
the administrator's point of view, there are many more view points. TQM is different 
from our usual experience, it is another change: 

References 

Atkinson P (1990) Creating Culture Change: the key to successful total quality 
management IFS Ltd UK 

Bedell G (1992) Tension mounts as the slow lane speeds up The Independent on Sunday 
8th March 

Oakland J S (1989) Total Quality Management Butterworth-Heinemann Oxford 



Jo 



5 Total quality management in an 
education and training context 



Barbara Ellis, Education and Training Consultant, Arlesford, Hampshire, S024 
0HH f UK 

DEFINING QUALITY 

Philip Crosby, an American quality "guru", defines quality as, "Everyone in the Company 
from the doorman to the managing director, routinely doing things right". 

This is a helpful, but in one respect a puzzling, definition of quality for Further 
Education and Training. It is in taking what is helpful, and in grappling with what is 
puzzling, that further education establishments can find a direction to move forward on 
a coherent and planned approach to assuring and improving quality. 

I imagine that no-one in education would quarrel with the assertion that everyone within 
the organisation should be involved in the drive for quality. Students attend or interact 
with colleges in a variety of ways and have a complex range of expectations and needs 
which they require the college to fulfil; these may range from the achievement of defined 
learning outcomes, to personal development, social satisfaction, counselling and guidance 
and will undoubtedly include the expectation of courteous and efficient service in all of 
their dealings with the college. Clearly, everyone employed by the college will affect the 
quality of the student's experience. 

Neither is there initially any difficulty with the notion of "routinely doing things right". 
At the first level of understanding this expresses a commitment to delivering each part 
of our service on a right first and every time basis. Anything less would imply an 
acceptance of sloppy work and inferior standards. 

However the problem arises when we examine the notion of the meaning of right in 
education and training. Our core technology, the enabling of learning, is a complex and 
interactive process and a simple exhortation to get things right begs the fundamental 
question - What is right in education and training? 

There are numerous and frequently changing interpretations of what constitutes a quality 
learning experience; these may be subject to political dogma, changing fashions in 
education technology and pressures from stake holders other than the learner. A tempting 
solution to this dilemma would be to limit our quality assurance to those elements of our 
service which would be easier to describe and quantify; it is right and laudable that such 
aspects of our service are included in any quality improvement process but to ignore the 
fundamental issue of the quality of learning would be to relegate quality to the periphery 
of our activities and would fail to capture the imagination of our teaching staff. 

Given that learning is an interactive process and that, in the literature of Total Quality 
Management, quality is seen as meeting the requirements and expectations of the 
customer, a way forward becomes clear - we consult those most intimately involved, the 
learners. By a process of negotiation between the suppliers (the staff of the college) and 
the customers (the students) we can arrive at an explicit and realistic set of quality 
requirements that we can guarantee to achieve. 

But unlike other service industries, the quality of our "product", the learning outcome, 
is not wholly within our control. We can only control our processes. The learner's 
contribution is vital and therefore should form part of the stated quality requirements. 
Thus, by negotiation we will arrive at a quality contract or a quality entitlement owned 
by the learner and the college. This contract will guarantee certain services, processes 
and behaviours so that the student and the college staff will fully understand what to 



30 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



expect from each other. The aspiration would be negotiated quality requirements with 
each learner but at first it may only be possible to arrive at negotiated quality 
requirements for groups of students. 

Thus the quality we assure is realistic within resources, meets our customer's reasonable 
expectations, is owned by the customer and the supplier and is achievable. By stating 
specific requirements we can measure achievement and plan to improve on any 
shortcomings. 

Quality is performing to requirements each and every time. 
PLANNING TO ACHIEVE QUALITY 

Total Quality Management is concerned with people, systems and culture. 

It is a truism that people are any organisation's most valuable resource but, in the 
context of quality improvement and assurance in education and training, it is worth 
repeating. Management's task within TQM is to create a situation in which the 
commitment, flair and experience of the staff can be released and guIUed to meet the 
agreed requirements of the customer. It is essential that senior managers themselves are 
trained in the philosophy and practices of TQM prior to committing to a programme of 
quality improvement. The genuine involvement of all in negotiating quality requirements, 
planning to achieve those requirements, identifying problems and offering and 
implementing solutions will be exciting but challenging and it's all too easy to retreat into 
hierarchical and authoritarian behaviour when the rhetoric becomes the reality of genuine 
involvement and problem identification. Management commitment must be visible, 
genuine and ongoing. 

Management commitment is demonstrated by; 

• generating the quality policy through consultation 

• publicising the quality policy 

• allocating money and time to the quality process 

• attending quality team meetings 

• talking with staff both informally and formally about quality 

• modelling the quality culture 

• valuing and respecting people (all of them, all of the time) 

• insisting on clear quality requirements and documented procedures 

At the beginning of the quality improvement process, managers will need to win their 
staff's commitment to, involvement with and awareness of quality by providing an 
opportunity for them to agree and make explicit the shared values of the organisation. 
Secondly, they will need to set up systems for team working and open communication. 
The notion that quality is achieved by the requirements of the customer is equally 
applicable to customer/supplier relationships within an institution as it is to the external 
customer and supplier. Therefore teams will be most effective if based on these internal 
customer/supplier relationships. The teams will be able to work together to agree quality 
requirements for specific services or functions, plan the work processes to ensure that 
these requirements are met, identify where errors or problems are occurring, devise and 
suggest solutions to these problems and act to eliminate the error. They will need to feel 
secure that this work will be regarded as positive and commendable and to know that 
there are formal and safe channels of communication by which problems and solutions 
can be communicated to line and whole college managers and decision making bodies, 
Having established these quality improvement teams and channels of communication, 



TQM IN EDUCATION AND TRAINING 



31 



managers will need to ensure that the team members have been trained in the skills and 
techniques of work process analysis, problem identification and solving and collaborative 
group working. It will also be advantageous to establish a team of staff who have an 
overarching leadership, advice and support function in the quality improvement process. 
The members of this team will need to be representative of all of the working groups and 
management levels within the college and will need in depth training in quality 
philosophy, systems and techniques. In support of these activities there will be a need 
to provide systems to document defined quality requirements and working procedures. 

Expert advice and training on assurance and monitoring systems should be sought. 
Wherever possible these functions should be in the control of the quality improvement 
teams but with some impartial auditing. It is self evident, but sometimes overlooked, that 
any audit process must involve customer consultation on their level of satisfaction on the 
achievement of quality requirements. The quality loop will then ensure that corrective 
action is taken where a failure to meet a requirement has been identified. 

An essential prerequisite for success in the quality improvement programme is the 
creation of an organisational culture that is supportive of the aims and processes of total 
quality. The first step for this will have been taken by involving all staff in the 
identification and expression of shared organisational values. There are some cultural 
characteristics which are essential to enable the process to succeed. Perhaps most vital 
of these is a real commitment to a no blame culture. Open communication is also a vital 
ingredient with everyone believing that a genuine attempt will be made to understand and 
to trust the intentions of the communicators. The culture will also be one in which it is 
expected that people will act; education is all too prone to analysing problems and 
spending an rnordinate amount of time and energy on finding the intellectually and 
ideologically pure solution only to move on to the next problem before corrective action 
has been taken. Tnus at all levels there will need to be a commitment to act, or at the 
very least to respond even if only to explain why a suggested solution cannot be 
implemented, within a specified and short time scale. 

Thus a total quality culture will have: shared values, commitment to always getting it 
right, open and explicit communication, time for teamwork, training in quality, total 
involvement, sensitivity to others' needs. 

ONE COLLEGE'S EXPERIENCE OF IMPLEMENTING TOTAL 
QUALITY 

Background Information 

Mid Cornwall and Saltash Colleges had, supported by a county wide project, developed 
systems and expertise in some aspects of quality assurance through a process of customer 
consultation based on student questionnaires adapted by course teams from work 
undertaken in Avon. The students were consulted, in detail, on their satisfaction with 
access arrangements (everything up to settling into the course), on the quality of their 
learning and assessment experiences and the validity of their course for the next stage of 
their lives (further or higher education or employment). Employers and/or further and 
higher education would also be consulted at the validity stage. The information gained 
from the questionnaires was used by the course team in their course review processes and 
as an aid to action planning for improvement. 

A number of full time courses had implemented this approach and the desire was to 
spread the process to all full and part time courses. 



32 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



The college management team had consciously decided not to seek course evaluation 
feedback from the customer consultation processes as they wished, at least in the pilot 
phase* to ensure that course teams owned the process and the information and that staff 
did not feel threatened or defensive. 

However, after the first year of the project, the course teams themselves felt that they 
would like to have clear ways of communicating information and requests to 
management at least about those issues which could be resolved by management or whole 
institution decision making groups. 

There was also a feeling by key members of the management team that the time was 
right to implement a whole college quality system both in response to an internal desire 
to do so and in response to external pressures. The College wished to develop and 
implement a quality approach which matched its culture rather than adopt a prescribed 
quality management system such as ISO 9000. They were attracted by what they had 
learned of TQM and by the notion of continuous, or incremental quality improvement. 

Some funding was available from a Regional Department of Employment project. 

Beginning the Process 

The College's management team decided to use a consultant to assist them with the initial 
stages of investigating TQM, to provide some management training in TQM and to help 
facilitate the introduction of their chosen approach to the whole staff of the College. 

The first activity was a development day for the College management team in which 
they explored the essential concepts of TQM and used TQM techniques to generate a 
draft College Quality Policy and draft • ;ollege Quality Aims. 

The second phase was to consult the College Governors who had a crucial role in 
management commitment and an evening session for managers and governors was held 
in which a presentation on quality systems in general and in education was made. The 
focus was on TQM as providing a framework which fitted College culture, which could 
if required embrace ISO 9000 and which could build on the work undertaken by the 
course teams in integrating regular and rigorous customer consultation into course review 
processes. During structured discussions, following the presentation, governors supported 
the approach that was recommended. Additionally a high degree of experience in 
industrial applications of TQM was identified and their assistance was enlisted with the 
next phase of the programme. 

The third phase was to launch the TQM programme with the whole College staff. An 
afternoon and evening session was organised to which all full time staff and part time 
employees who worked more than ten hours per week were invited. Management 
commitment to the programme was effectively demonstrated by the large investment of 
resources that such an event entailed, by the closing of the College for the afternoon and 
by the high profile that they took during the proceedings. 

Presentations by two College Governors with industrial experience of TQM, by the 
consultant and by the College Principal and Vice Principal (Development) were made and 
questions sought. Workshop sessions followed in which staff were asked to comment on 
the draft Quality Policy and the draft Quality Aims and to identify key areas within the 
College that should be the subject of the first phase of the quality improvement 
programme. 

Following this session with all staff the suggestions for areas for quality improvement 
were circulated and staff were invited to volunteer to be involved in the work. The 
College Quality Policy and Aims were approved by the Board of Governors and 
circulated to all staff. 



A 



TQM IN EDUCATION AND TRAINING 



33 



A structure for teams was agreed and established with the overarching role being taken 
by a Quality Steering Group and with four Quality Improvement Groups to address the 
four areas for whole College improvement that had been identified. It was decided that 
a regular Quality Newsletter would be produced and distributed to staff. 

The College also devised and administered a Quality Audit in the form of an attitude 
survey with ali staff. It is intended to use the results of the survey to inform management 
but also as a benchmark against which the results of the whole College Quality 
Improvements Groups' work can be measured. 

Training in some of the tools and techniques of TQM and in group working was 
undertaken by the College management team. They have begun to cascade this training 
to members of the Quality Improvement Groups. 

Some Outcomes from the Programme 

The first action, the senior management development day, took place in October 1991. 
Identifiable outcomes to date include: 

• A College Quality Policy 

• College Quality Aims 

• Four teams to address quality improvement issues that relate to the whole 
College and which were identified as priorities at the whole staff event: 
Communication, Management, Accommodation, Course Evaluation and 
Review 

• A Quality Steering Group 

• A staff attitude survey, the results of which have been communicated to all 
staff 

• Extension of the customer consultation and course review processes with the 
intention of spreading this to cover all College courses 

• The Quality Improvement Group for Communication has recommended that 
details from management meetings should be more widely publicised and that 
senior staff should make themselves more accessible. As a result the 
Principal and Vice Principals will attend meetings by invitation and will be 
available for informal contact. Agendas and summary notes of Heads of 
Department Meetings are posted on notice boards and widely circulated to aid 
communication and accurate and timely information flow 

• Four editions of the Quality Newsletter have been published 

• The Quality Improvement Group for Accommodation is reviewing computer 
programmes for time tabling and room use. A system for reporting and 
rectifying "minor" repairs/replacements h?s been devised and implemented 

• The Quality Improvement Group for Management has reviewed the College 
committee structure and has prepared recommendations for a more effective, 
efficient and accountable use of committees 

• The Quality Improvement Group for Course Evaluation and Review has 
received the results of an audit devised by the Curriculum Committee and is 
using these to evolve a systematic approach for the whole College 

• Senior staff have been trained in aspects of quality improvement 

• A large number of College staff have been consulted and involved. Some are 
taking an active part in whole College quality improvement initiatives 



6^ 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 

The Governors have approved a Mission Statement for the College and 
strategic aims for the College for the next five years. These have been much 
influenced by the information gained and the culture established from the 
quality initiatives 

The College has discussed the Investors in People programme with DCTEC 
and has registered an interest in an initial assessment visit 



6 Performance management in a 
competence framework 



Elaine Sauve, TDA Consulting Group Ltd, Brentford, Middlesex, TW8 OHF, UK 
INTRODUCTION 

In the current economic climate and with 1992 and the open European market upon us, 
companies are becoming increasingly aware of the importance of investing in training. 
It is, however, recognised that training per se is not always a good thing. Training needs 
to be targeted. Corporate goals are identified and the training required for individuals to 
help the company meet those business goals can be designed and introduced. 

Tliis approach has the advantage of being cost effective in helping the company meet 
its objectives but is also motivating for individual staff who feel that their training 
requirements are being met, helping them to cany our their jobs more effectively, whilst 
also recognising their current skills. 

Nationally there is a thrust towards the development of standards of competence in all 
occupational sectors and the introduction of training programmes based on these national 
standards. This may or may not be linked to the acquisition of National Vocational 
Qualifications (NVQs). 

Most companies appear to accept this approach for craft and technician grades and for 
clerical and administrative support staff. There is a tendency to feel that it is not possible 
to identify standards of competence for managerial and professional level staff. This is 
at best a questionable premise, probably based on a misconception of the definition of 
"competence". A competence-based approach should not merely be a list of tasks but 
should reflect broader issues such as task management, health and safety, and the ability 
to deal with organisational environments, relationships with other people and unexpected 
events (NCVQ 1991). It is therefore not only possible to identify occupational 
competence for managers, it is positively desirable in order to ensure targeted, cost 
effective training. 

It is probable that some of the resistance to the "competence" model is due to the 
intimidating amount of jargon which is associated with it. Some of the jargon associated 
with NVQs includes: competence, standards, unit, element, performance criteria and range 
statement. Introducing these specialised terms into a company training and development 
programme can be intimidating for all concerned, be they line manager, trainer or 
employee undergoing training. I will demonstrate here the relevance of competence in 
a performance management system and the relationship between established management 
terminology and "competence" terminology. A performance management model will be 
described, which is generic but which can be tailored by individual organisations and has 
been introduced in a number of companies as a key contribution to their total quality 
approach. 

WHAT IS COMPETENCE? 

Some of the concerns about the introduction of NVQs, particularly from managers, is a 
perception that "competence" refers to the possession of practical or manual skills. This 
was perhaps reinforced by the early accreditation of NVQs, many of which were largely 
associated with practical skills. In fact the NCVQ definition of competence, whilst it 
certainly does recognise the need for the performance of tasks, is much broader. An 
individual must also possess the relevant knowledge and understanding to perform 



36 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



effectively in the work situation. 

Competence then, in NVQ terminology, is much broader than is often perceived and 
applies from trainee to senior management. 

At this stage it will be useful to give some broad definitions of NVQ terms, before 
discussing the performance management model and why performance management is an 
important factor in the establishment of total quality. 

NVQ Definitions 

• Unit of Competence - discrete work area 

• Element of Competence - describes what a trainee should be able to do 

• Performance Criteria - specify the standards of performance 

• Range Statement - details of the range of applications or contexts across which 
the performance criteria apply 

• Essential Knowledge - underpins performance 

WHAT IS PERFORMANCE MANAGEMENT? 

There are probably as many different performance management models as there are 
definitions of "competence". Probably the most familiar one is the annual appraisal 
system. Often the appraisal is based on the subjective judgement of the individual line 
manager, who considers personal attributes such as: contribution to the team, co- 
operation, attitude, effort and promotability. 
The Performance Management Model described here differs in that it is: 

• an on-going cyclical model rather than an annual interview 

• based on clear objectives, agreed between job holder and line management 

• based on continuous review of progress 

• not dependent on an annual performance review 

The model is essentially a three stage cyclical model - see figure 6.1 . 
STAGE ONE: JOB CLARIFICATION 

This stage involves line manager and job holder working together to agree the objectives 
of the job and identify areas of training and development required to enable the job 
holder to meet the objectives. From this a personal development plan can be produced, 
detailing: 

• clear objectives 

• training and development requirements 

• agreed timescales 

• agreed monitoring and review process 

STAGE TWO: MONITORING AND REVIEW 

This is an essential part of the performance review cycle and involves the lino manager 
in taking responsibility for reviewing progress against the agreed objectives and providing 
constructive feedback. There may be a requirement to re-negotiate the personal 
development plan at this stage. 



MANAGEMENT IN A COMPETENCE FRAMEWORK 



37 




S l Alih ON K 

JOB CLARIFICATION 

job Roll 
Elements of Competence 
(Objectives) 



ST AC J K THRKK 

ANNUAL 
PERFORMANCE 
REVIEW 




st\c;f. two 

MONITORING 

(Assosiment) 



Figure 6.1 The performance review cycle 

STAGE THREE: ANNUAL PERFORMANCE REVIEW 

This may be seen as part of the on-going review process or it may be that the 
organisation's practice requires an appraisal interview. Either way, the "annual review" 
should present no unpleasant surprises to the candidate. He/she will be aware of progress 
and of steps which need to be taken to meet objectives. 

In describing the performance management model, terms which will be familiar to 
human resource and training managers have been used. It is our belief that there is no 
conflict between this model and the competence framework and below we suggest some 
broad "equivalences" of terminology. We recognise that this is probably an over 
simplication. Nonetheless we believe it provides a platform for employers to begin to 
relate to the national jargon attached to competence terminology. 

TERMINOLOGY 

COMPETENCE PERFORMANCE MANAGEMENT 



Functional Analysis 
Key Job Role 
Units of Competence 
Elements of Competence 
Performance Criteria 
Outcome Based 
Assessment 



Job Clarification 

Job Purpose Statement 

Key Results Areas 

Objectives 

Success Criteria 

Outcome Based 

Monitoring 



38 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



WHAT ARE THE ADVANTAGES OF A PERFORMANCE 
REVIEW CYCLE? 

It is suggested that advantages for the organisation are: 

♦ Corporate objectives are clarified and publicised 

♦ People are working towards the achievement of corporate goals 

♦ Overall performance is raised 

♦ Identification of resources required to achieve objectives takes place 

♦ Means for monitoring progress are provided 

♦ Training and development needs are identified 

Advantages for the individual are: 

♦ Clarification of the job, purpose and goals 

♦ Increased motivation 

♦ A basis for a two-way discussion is formed 

♦ A basis for monitoring progress and formative feedback is provided 

♦ Resource requirements and constraints are identified 

♦ Development needs are identified 

♦ Links may be made with National Standards of Competence and Nationally 
recognised qualifications 

THE LINK WITH TOTAL QUALITY 

More and more companies are beginning to investigate "Total Quality Management" as 
a vehicle to increase organisational efficiency and effectiveness. There are a number of 
TQM models and initiatives, but most stress the need for: clear corporate goals and 
objectives: clear individual goals and objectives linked to the corporate goals: investment 
in relevant training and development; on -going staff monitoring (assessment) against 
agreed objectives. 

A recent initiative which makes explicit these features is * Investors in People'. 

An Investor in People: 

♦ Makes a public commitment from the top to develop all employees to achieve its 
business objectives 

♦ Regularly reviews the training and development needs of all employees 

♦ Takes action to train and develop individuals on recruitment and throughout their 
employment 

♦ Evaluates the investment in training and development to assess achievement and 
improve future effectiveness 

The Confederation of British Industry (1991) published Weld Class Targets, which 
outlined a series of progressive achievement targets for individuals and organisations, 
which must be attained if Britain is to maintain a competitive edge in world markets. 
The introduction of a structured performance management system will help with that task. 



6 



MANAGEMENT IN A COMPETENCE FRAMEWORK 



39 



References 

NCVQ (1991) Guide to National Vocational Qualifications National Council for 
Vocational Qualifications March 

CBI (1991) World Class Targets Confederation of British Industry 



J 



7 Quality in further education: an 
unchanging agenda 

David Shepherdson, West Sussex Advisory and Inspection Service, Chichester, West 
Sussex, UK 

INTRODUCTION 

"Politicians' time-scales are always much shorter than anyone else's; there are always new 
agendas for LEAs and we are in a state of perpetual change in education. However, one 
agenda will never change, and that is to provide the best educational opportunities for all 
... and that is what we must do" (Sir Roy Harding 1990). 

The focus here is on the unchanging agenda of quality development from a Local 
Education Authority (LEA) perspective. The West Sussex Initiative is based on the 
author's experience of working with the LEA quality development team in Birmingham, 
and it is primarily concerned with developing Total Quality Management (TQM) systems 
in all four Further Education Colleges in West Sussex. It is concerned with a belief in 
the self -evaluating professional working within the self-evaluating institution. The 
Advisory and Inspection Service is supporting the development of systematic training in 
monitoring and evaluation techniques for all college staff, through a programme of 
workshops. The process is designed to equip whole institutions to achieve "self -critical 
academic maturity 11 within a specified timescale. 

THE INITIATIVE 

The overall aim of the initiative is to link the demands for accountability with the desire 
for improvement; often they are seen to be mutually incompatible. Improving can be a 
statistical exercise, fulfilling requirements in terms of performance indicators but, when 
linked to the qualitative process of developing a narrative related to classroom practice, 
the two processes can be both developmental and powerful, A keynote of the Initiative 
is to gain the commitment of identified teams of staff in FE colleges to the idea of 
monitoring and evaluation as a positive and educative process in itself. Too often, quality 
is associated only with audit, review and inspection. If quality is genuinely in-built and 
integral to the whole institution, the integrity of the process may eliminate the 
fault-finding, problem-solving focus of many quality systems. 

The identified teams are asked to work through 20 tasks in the handbook over one year. 
This is supported by systematic staff development and accredited training in evaluation 
techniques for team leaders. The teams identify 'quality gaps' in the area of provision 
for which they are responsible. These 'gaps 4 are then converted to quality targets that 
arc tackled by the team itself. Notionally each team will identify 3 targets to be met each 
term. In order to address these targets, the team formulates an action plan identifying 
what is to be tackled, by whom, how and where. This provides the basis for a bid to a 
central college group (the Curriculum Review and Development Group) which will then 
approve or modify the bid for funding. The funding may take the form of release time, 
equipment or materials. This forms the basis of reciprocal contracting - where the team 
contracts to deliver on its own targets and the Review Group contracts to provide support 
for the team to meet its targets. The Senior management team would contract to deliver 
on its own targets in the same way as all other teams. 



QUALITY IN FURTHER EDUCATION 



41 



All targets and timescales are 'public knowledge* and conform to the College strategic 
framework for quality. This means that the College Development Plan can be enacted 
through the contracting process and the regular review of progress each term. 

IMPLEMENTING THE INITIATIVE 

The first stage is to agree some key principles that provide the focus of improvement. 
Many teams include classroom practice, the quality of teaching and learning or the 
interaction between lecture and students. These concepts need to be dis -aggregated into 
a series of developmental tasks. At this stage, it is useful to begin the process of 
identifying critical success factors or criteria for making judgements about the success of 
the improvement process. Although a necessarily lengthy process, it is essential to ensure 
a commitment through ownership - the team decides what the issues are, and what the 
criteria for success will be. The accredited training programme provides the necessary 
support for key individuals to lead the programme as 'professional evaluation*. 

It is important that the central group that receives bids from teams undertakes a 
monitoring and evaluation role for the whole college. This Curriculum Review and 
Development Group is normally representative of all levels of staffing and operates 
rotation of membership to ensure full representation over several years. The CRDG remit 
is based on the College Development Plan and Quality Development strategy for the 
institution. Team progress is monitored by brief half-termly reporting and end of term 
reviews. 

The reciprocal contracting of the TQM system extends to learner contracts as an integral 
part of "entitlement". This replicates a work contract for students and would include 
enhanced opportunities for learner support and study skills programmes. 

METHODOLOGY OF THE INITIATIVE 

The methodology of the initiative is based on the work of David Kolb and D A Schon. 
This model of experiential learning and the reflective practitioner, exemplified through 
the research and development function, is not usually part of the FE College system. If 
time for 'reflection' is built into the teaching and learning programme, it becomes 
possible to identify needs and meet planning requirements within "normal practice". The 
teams therefore use a variety of monitoring and evaluation techniques, each suited to its 
own purpose. The widespread use of questionnaires is generally found to be of limited 
use. Both students and staff can become "anaesthetised" to their use and the quality of 
some information is doubtful. By using a variety of methods, it is possible to triangulate 
on issues, and gain a much closer insight into the origins and nature of the issues. Many 
of the so-called problems identified by some quality systems are products of the system 
itself rather than being symptomatic of real underlying course culture. 

Evaluation techniques can be built into learning programmes, using tutorials, seminars 
and workshops. The common complaint is that "we don't have time ... (for quality)." 
Teaching hours are being reduced and many staff are under considerable stress as a direct 
result of increased administration. A possible solution lies in revisiting teaching 
methodologies, timetabling and room allocations. It becomes necessary for the team to 
have increasing ownership of the means to deliver the curriculum that meet their 
identified needs and the needs of the students more effectively. 



42 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



Traditionally colleges have been driven by SSRs and the requirements of resource 
allocations based on student numbers. Increasingly, the notion of performance related 
funding may produce a more efficient and effective way of delivering learning 
programmes that will incorporate "self -critical academic maturity 1 '. These are the issues 
that can be picked up very quickly by the 'empowered* course team. Their initial 
solutions to perceived problems may be crude, but they offer a genuine R and D function 
that many commercial operations value highly. The emphasis on discrete elements of 
college operations eg. marketing, recruitment, quality, tends to fragment and dispel 
responses such as - "its someone else's responsibility". You cannot delegate or 
subcontract quality and the fundamental approach of TQM lies in a set of values and 
attitudes that embody commitment to continuous improvement. In the same way, quality 
is not a project, nor is it a standard to be achieved. This serves only to limit achievement 
and leads to a "lip-service 1 ' model of the way that things should be done. 

QUALITY IMPROVEMENT 

The "steps" to quality improvement begin with a view about "what do we mean by 
quality". Programme Area teams may have a different view to that of the management 
teams; providing interesting perspectives on the college's view of quality. Agreeing 
common principles is part of the first reflective stage. It is based primarily on experience 
{as in the Kolb model), and may be a function of a series of critical incidents, concerns, 
problems, good ideas or hunches. 

These form the basis of the keynote principles to be outlined in the college Quality 
Development Action Plan, that outlines the "mission" and incorporates commitment to 
specific quality attributes. 

Secondly, decisions about the composition of teams will provide a basis for identifying 
areas of "common effort". In itself an interesting exercise, this can identify the "black 
holes" of provision and the "team that never meets" or the "team that never was". Team 
forming provides a basis for m:>re collaborative working and for improved 
communications and shared resources. Indeed, the benefits of this single exercise often 
promote a more integrative approach to all aspects of the college provision and lead to 
consideration of structured responses (for example, in relation to modularisation). 

The third stage is for each team to identify their own response to "quality gaps", as 
perceived by the team. These gaps should become targets for improvement, recognising 
that in some cases there are resource implications for management attention. Initially, 
it is useful to identify smaller (incremental) issues for improvement, rather than medium 
or long term targets. If all teams engage in the process, by contracting to deliver, the 
whole college can move rapidly to a stage of critical mass. At this point the management 
strategy should incorporate a moderating influence, ensuring a degree of realism and 
consistency in both targeting and methodology. This is important to ensure that diversity 
of approach does not preclude a corporateness of purpose. 

It would be possible for a system to be "bought in" - but this does not generally 
engender ownership or indeed, commitment by all. Most alternatives are superstructures 
rather than infrastructures - built on, rather than built in. It is essential to grow the 
system within the organisation rather than impose a structure for accountability; this is 
because the reaction to the exogenous system is often to subvert or at least to ignore it. 
Systems for accountability can often be counter productive, generating whole sub-systems 
and consuming large amounts of time and energy. 



5~ 



QUALITY IN FURTHER EDUCATION 



43 



CONCLUSIONS 

The advantages of TQM and the "home grown" model of quality improvement mean that 
ownership, empowerment and commitment go hand-in-hand, leading to real progress 
towards commonly agreed goals. There is no reason why this should not meet external 
demands for accountability and, additionally, provide the basis for systematic 
improvement. 

The real benefits are usually in terms of creating genuine enthusiasm and motivation; 
creating some space for reflecting on what business we really are in, and finally creating 
the basis for a continuous quest to provide the very best learning opportunities for the 
benefit of the students. 

There are several key points that are embodied within the West Sussex Initiative. The 
unchanging agenda is concerned with continuous improvement with the aim of improving 
the quality of the learning experience, increasing participation in training and education 
and improving attainment. Above ail, quality is judged by the customer and requires us 
to deliver consistently above expectations. In this way it is quite feasible that educational 
ideals and economic realities are not necessarily mutually exclusive. 



8 Defining quality in higher education - 
the stakeholder approach 



Alison Burrows and Lee Harvey, University of Central England at Birmingham, 
Birmingham, B42 2SU, UK 

QUALITY IN HIGHER EDUCATION - THE LAST DECADE 

During the last decade qualify has become an important issue in higher education for two 
separate, but related, reasons. In the first place there has been ~ drive for greater 
accountability by the government for the way in which higher education spends the funds 
allocated to it. Society is no longer prepared to take on trust that higher education is 
providing value for money. Higher Education is being asked to demonstrate that it is 
spending public money efficiently and effectively. This parallels approaches to other 
public services, such as Health (Cave et al 1990). 

In part, this demand for greater public accountability came from a sense that higher 
education had not been responding effectively to the needs of a modern industrial society. 
So, in the 1985 Green Paper "The Development of Higher Education into the 1990s" we 
find: 

"there is continuing concern that higher education does not respond sufficiently to 
changing needs. This may be due in part to disincentives to change within higher 
education, including over-dependence on public funding, and to failures in communication 
between employers and institutions" 
and: 

"The Government believes that it is vital for our higher education to contribute more 
effectively to the improvement of the performance of the economy" (DES 1985). 

The government's commitment to ensuring greater public accountability for public 
expenditure on higher education has continued throughout the 1980s. Its stance is 
demonstrated by the following extract from a letter from the Secretary of State to the 
Head of the PCFC: 

"I shall however expect to see two key features. Hie first is a means of specifying clearly 
what polytechnics and colleges are expected to provide in return for public funds. The 
second is a systematic method of monitoring institutional performance. I attach particular 
importance to the latter, since without measures of performance, the Council will have the 
means neither of satisfying itself that institutions are providing what has been promised 
at acceptable quality, nor of making comparative assessments of institutions as a basis for 
future allocations of grant" (Baker 1988). 

This drive for greater accountability has not been restricted to the United Kingdom but 
should be viewed as part of an international trend. Similar developments have occurred 
in many other countries (Neave 1986; Goedegebuure et al 1990; Teather 1990). 

The second important development that has influenced the quality debate in higher 
education in the United Kingdom has been the move away from an elitist system of 
higher education towards one of mass participation without a corresponding increase in 
finances. This reduction in the unit of resource in higher education and the increase in 
numbers entering the higher education system, many without the traditional entry 
requirements, has led some to argue that quality must inevitably suffer. 



QUALITY AND THE STAKEHOLDER APPROACH 



45 



The debate has gained added impetus through the publication in 1991 of the White 
Paper 'Higher Education - A New Framework 1 and the subsequent Bill on Further and 
Higher Education which was enacted in March 1992. The new law heralded significant 
changes in the structure of higher education in the United Kingdom. All of the 
polytechnics were granted degree awarding powers and the Council for National 
Academic Awards, a major quality assurance body for the polytechnics and colleges 
sector, was wound up. A single higher education funding council was established for 
England with similar arrangements for Wales, Scotland and Northern Ireland. The new 
legislation required the funding council to establish a Quality Assessment Committee and 
to "secure that provision is made for assessing the quality of education provided in 
institutions ..." (Further and Higher Education Bill 1992). The institutions have been 
encouraged by the government to set up their own Quality and Access Organisation to 
focus on the audit of quality assurance systems. 

DEFINING QUALITY 

What is quality in higher education and how do you go about assessing it? Quality is a 
concept, like freedom or justice, of which people have intuitive understanding but whose 
meaning they find difficult to articulate. Does it mean the same to everybody or do 
people use the same word to mean very different things? The Quality in Higher 
Education (QHE) project is looking at precisely these issues. The Project was launched 
in September 1991. It is sponsored by a partnership of education, government and 
business. There are currently 27 sponsors in the United Kingdom including the 
Department of Education and Science, the funding councils, accrediting bodies, major 
employers, employers' organisations, higher education institutions, research and training 
organisations. It is independent of any single organisation or institution but its physical 
base is the University of Central England. Each of the sponsors is represented on the 
project steering group which meets about once every three months. Sponsors are 
expected to take an active part in determining the scope and development of the project. 

The ultimate aim of the project is to develop a methodology for assessing quality in 
higher education. However, before you can develop techniques for assessing something 
you ha\e to be clear what it is that you are trying to assess. The first year of the project 
is being spent establishing a clear set of criteria forjudging quality in higher education. 

An underlying hypothesis for the first phase of the project is that quality in the context 
of higher education is a relative concept. Its definition varies according to: 

• which aspect or dimension of the higher education process is the focus of 
attention; 

• who is making the assessment; 

• the purpose for which the assessment is being made. 

THE STAKEHOLDER APPROACH 

In order to test this hypothesis the research team has identified a range of interest groups 
or "stakeholders" in higher education, eight in all: students; employers; 'government*, 
subdivided into ministerial departments (DES, DTI, DE); funding councils; teaching staff 
in higher education institutions; managerial staff in higher education institutions 
(including representative bodies such as CDP, CVCP, and administrative and support 
staff); accrediting bodies (including the professions); assessment bodies (HMI, the Audit 
Unit and the new Assessment Unit). 



5° 



46 PERSPECTIVES ON QUALITY IN ORGANISATIONS 

Our aim is to find out the views on quality of each of these groups by looking at: 

• the focus of interest of each group - is the focus on the input, process or output 
of higher education; is it on teaching or research? 

• the underlying definitions of quality used by the groups and their political and 
philosophical underpinnings. 

The stakeholder approach is an analytical tool which other researchers have used to look 
at quality in higher education (Vroeijenstijn 1991; Yorke 1991). Primarily, it has been 
used in the Netherlands to try and establish a set of performance indicators for assessing 
quality which would have wide credibility (Dochy et al 1990). The crucial difference 
between this research and our own is that, at this stage, we are not looking at indicators 
of quality but at the criteria for judging quality itself. We are trying to find out what 
constitutes teaching quality rather than how you might measure that teaching quality is 
present (which might involve, for the sake of argument, looking at the degree 
classifications of graduates). 

To use the stakeholder approach effectively means that there should be no assumptions 
about how a particular group might define quality nor about its main focus of interest. 
The project does not concentrate specifically on the quality of teaching and learning or 
research or institutional quality - anything may be considered relevant. 

At the end of the first stage of the project, we aim to have a set of quality criteria 
ranked in order of preference, which reflect the views of all the stakeholders. At one end 
of the spectrum each stakeholder group might think about quality in a different way and 
its focus of attention might also be different. M ^e likely, however, there will be some 
convergence and overlap of views on particular issues. 

RESEARCH METHODS 

A variety of methods is being used to discover the views of the different stakeholders. 
These have been combined into three broad groups for practical purposes. 

Staff and Students in Higher Education 

A questionnaire based survey is being conducted covering sixteen institutions selected 
according to territorial, cultural and structural criteria. Within each institution, a 
representative stratified sample of students (approximately 400) and random sample of 
400 staff (teaching and non -teaching) is being surveyed. 

To design the questionnaire, we undertook a literature search to identify the quality 
criteria and measures currently in use by bodies such as HMI, PCFC, UFC, CDP, CVCP, 
CNAA, BTEC, NATFHE and the AUT. Using the research conducted by the Student 
Satisfaction Project at Birmingham Polytechnic (Mazelan et al 1991) and parallel work 
by the project team on employers' views, we identified around 300 criteria. These were 
distilled down, by merging overlapping criteria, to around 100 different elements. 

We then attempted to specify more clearly the "definitions" of quality. "Definition" was 
considered too grandiose and so we reverted to the rather less precise "ways of thinking 
about" quality. From this we identified nine in total: 

• Meeting the requirements of the student. 

• Enhancing the knowledge, abilities and skills of students. 

• Empowering students to effect changes in their education. 



QUALITY AND THE STAKEHOLDER APPROACH 



47 



• Fulfilling the stated intention (or mission) of the institution. 

• Providing mechanisms to ensure that students get what has been offered. 

• Striving for excellence in all aspects of the institution and programmes of study. 

• Making efficient and cost effective use of educational resources. 

• Checking that standards have been met. 

• Providing a distinctive, special and "high class" education. 

Staff and students are asked to indicate how important they think the criteria are for 
judging quality in higher education and which "way of thinking about quality" seems 
most appropriate to them. 

Government, Funding Agencies, Accrediting Agencies 

This work is ongoing - a review of published material has been conducted which will be 
followed up by interview-based verification. 

Employers 

The following methods were selected and have been used: interviews with organisations 
representing employers; a short employer questionnaire; two employer seminars. 

Interviews with employer representatives on the steering group took place to help 
determine what methods should be used to find out the views of employers about quality 
in higher education and the character of the employer questionnaire and seminars. This 
source was backed up with two in-depth interviews with organisations representing 
employers: the Confederation for British Industry (CBI) and the Council for Industry and 
Higher Education. 

As a result of the interviews, the research team reached the conclusion that a focused 
approach, concentrating specifically on the outputs of higher education relevant to 
employers, was most likely to yield results. 

Higher education can be viewed &s a process which has two main outputs: an educated 
workforce and research (Cave et al 1991). The research team has sought the views of 
employers on these outputs in two ways: a short questionnaire on graduate cruitment; 
two seminars, one focusing on quality in relation to an educated workforce and the other 
on quality in relation to research, consultancy and sponsorship. 

The discussions during the two seminars have been drawn together to produce a position 
statement on employers' views on quality in higher education and a set of criteria for 
judging quality. 

The project steering group itself provides another resource as the majority of 
stakeholder groups are represented. The project has been able to make use of the group, 
via discussions within meetings, depth-interviews with members and contacts with other 
key informants. 

FUTURE DEVELOPMENT OF THE PROJECT 

During the second phase of the project, quality assessment techniques will be tested 
within higher education institutions. The criteria identified in the first stage will be used 
as a basis for developing quality indicators. A methodology will be developed 
incorporating these indicators. Although it is difficult to be clear about the methodology 
at this stage, it is likely that the research team will seek the cooperation of institutions 
to monitor and evaluate methods which are already being developed and used. In the 



5" 



48 



PERSPECTIVES ON QUALITY IN ORGANISATIONS 



case of methods that are not currently being utilised in UK higher education, a selected 
number of institutions will be invited to take part in the testing and evaluation exercise. 

It is likely that certain approaches to quality management, such as BS5750 and Total 
Quality Management, will also be evaluated. The objective here will be to see whether 
these approaches can be successfully adopted for use in the context of higher education. 

How successful these techniques are in measuring or managing the quality criteria 
identified in phase one of the project will be the focus of the evaluation. 

References 

Baker K (1988) Letter to the Chairman of the PCFC, 1st November 

Cave M, Kogan M and Smith R eds (1990) Output and Performance Measurement in 
Government: state of the art Jessica Kingsley London 

Cave M, Hanney S. Kogan M (1991) The Use of Performance Indicators in Higher 
Education - a critical analysis of developing practice Jessica Kingsley London 

DES (1985) The Development of Higher Education into the 1990s Green Paper Cmnd 
9524 HMSO 

DES (1991) Higher Education: A New Framework White Paper Cm 1541 HMSO 
DES (1992) Further and Higher Education Bill 

Dochy FJRC, Segers M S R, Wijnen W H F W (1990) Preliminaries to the 
Implementation of a Quality Assurance System Based on Management Information and 
Performance Indicators in Dochy FJRC. Segers M S R, Wijnen W H F W (eds) 
Management Information and Performance Indicators in Higher Education: an 
international issue Van Corcum Assesn/Maastricht pp 69-94 

Goedegebuure L C J. Maassen P A M and Westcrheijdcn D F (1990) Quality 
Assessment in Higher Education in Goedegebuure L C J, Maassen P A M and 
Westcrheijdcn D F Peer Review and Performance Indicators - quality assessment in 
British and Dutch higher education Utrecht Uitgeverij Lemma B V pp 15-45 

Mazelan P. Brannigan C. Green D. Tormay P and O'Shea J (1991) Using Measures of 
Student Satisfaction: the implications of a user- led strategy of quality assurance in higher 
education Broadcast 18 Winter pp 4-5 

Neave G (1986) The All-Seeing Eye of the Prince in Western Europe in Moodie G C 
(ed) Standards and Criteria in Higher Education SRHE & NFER/Nelson Guildford 
pp 157-170 

Teather D (1990) Performance Indicators in Australian Higher Education: the context and 
an appraisal of the 1988 report* in Dochy FJRC. Segers M S R. Wijnen W H F W 
(eds) Management Information and Performance Indicators in Higher Education: an 
international issue Van Corcum Assesn/Maastricht pp 103-118 



5B 



QUALITY AND THE STAKEHOLDER APPROACH 



49 



Vroeijenstijn A I (1991) External Quality Assessment: servant of two masters? Paper 
presented at Hong Kong Council for Academic Accreditation conference on Quality- 
Assurance in Higher Education 15-17 July Hong Kong 

Yorke M (1991) Performance Indicators: obsen'ations on their use in the assurance of 
course quality Council for National Academic Awards Project Report 30 



Section 2: Implementing Quality in 
Organisations 



The contributions here are largely in the form of case studies that describe in some detail 
how quality approaches have been implemented in specific organisations. Taken together 
they represent a significant range and type of organisation. 

At one end of the scale, in "TQ implementation - cultural issues and training", we find 
implementation in a very large multi-national company described by Brian Hurley, with 
its problems of worldwide permeation and scale. John Oakland and Leslie Porter 
describe the application of their European model of quality in the context of a single 
university in "Developing the European quality model", 

Robert Newton, in "Strategic quality management", argues for and takes a more strategic 
view of quality management in his description of the implementation of quality 
approaches in his Training and Enterprise Council, 

The strategies that were employed in quality initiatives in a former polytechnic are 
explained by John Heap and Hazel Solomon, who also present a balance sheet of the 
"Profits and pitfalls in establishing a qualytechnic". In a significant change of emphasis, 
Patti Mazelan and her colleagues focus specifically on the student and describe their 
work to elicit views of students that might inform the monitoring of quality in "Student 
satisfaction and perceptions of quality". 

Douglas Edgar, in "Experiences in the design and conduct of enterprise audits", explores 
the methods and strategic approaches that can be taken in the complex, and often 
controversial, process of educational auditing. In "Monitoring the quality of quality 
control systems" , Henry Ellington and Gavin Ross move into the realms of meta quality 
control as they describe and examine systems in their own higher education institution. 




9 TQ implementation - cultural issues and 
training 



Brian Hurley, formerly Quality Implementation Manager, BP Chemicals Ltd 
INTRODUCTION 

BP Chemicals, one of four main businesses in the BP group, took 20,000 employees, at 
90 locations worldwide into a Philip Crosby based total quality process in 1987. 

The early successes and lessons highlighted the need to reinforce and recommit 
everyone to take full ownership. A team was set up in 1990 to catalyse and focus this 
recommitment to quality implementation. 

Tne team, led by the author, and assisted by outside training consultants, developed a 
new in-house training package and process, built on the Crosby phase, but customised to 
BP Chemical's needs. A BP group initiative for overall culture change reinforced the 
emphasis on combining hard and soft factors in the integrated training process then used 
to launch the BP Chemicals recommitment, which was called "Towards 2000 with Total 
Quality". 

The training package and processes demanded follow-up into "business as usual" by 
requiring the line and senior management role models to lead the change process by 
personal involvement in training activities, as the platform for new ways of thinking and 
working. This raised issues and barriers, created opportunities and benefits which are 
outlined here. 

Total quality implementation as the foundation for underpinning a culture focused on 
customer satisfaction, and the need to continue our value improvement of all work 
processes, is hard work but not optional. 

CONTEXT 

The BP Group in 1991 comprised the main businesses indicated in Figure 9.1. 

Employees 



Exploration and Production 10,200 

Refining and Marketing 56,000 

Chemicals 20.400 

Nutrition 20.600 

Miscellaneous and Corporate 9.600 

Total Worldwide 116.800 



Figure 9.1 BP business and employee levels (1991) 

BP Chemicals had been a loss making part of BP in the early 1980's but by focusing on 
core businesses, and reducing costs and people numbers during 1980 to 1984. we became 
profitable. We alone in the BP group embarked on Total Quality (TQ) ui late 1987 to 
establish a culture of continuous improvement focused on "Internal" and "Exicrnar 
customer satisfaction. 



34 



IMPLEMENTING QUALITY IN ORGANISATIONS 



We are organised into business areas, cross matrixed with a set of essential functions, 
which characterise our company's approach to the way we operate. A snapshot of our 
recent business performance and scale for 1989, 1990 and 1991 is indicated in Figure 9.2. 





1989 


1990 


1991 


Turnover(£m) 


3,503 


3,164 


3,064 


Profit <£m) 


540 


129 


NIL 


ROCE (%) 


28 


6 


NIL 


Capital Employed (£m) 


2,143 


2,081 


2,436 


Employees 


25,000 


20,400 


20,000 


Locations 


90 


70 


70 



Figure 9.2 BP chemicals recent business indicators 
STARTING THE TQM JOURNEY 



The principal reason for starting TQ was to secure competitiveness by a strategy based 
on a vision that we needed to: 

• Compete continuously for customers, ie. be preferred suppliers. 

• Create a culture of continuous improvement of all work, as a process. 

• Improve the participation, job satisfaction and value of all employees. 

• Focus on process management 

In short, we needed to improve our effectiveness at adding value in a series of chains 
both internally and especially across the external supplier and customer interfaces. 

Our initial priorities were the internal processes, believing we had to learn ourselves 
before offering TQ to our customers, and demanding it from our suppliers. 

Our model of TQ embraces three key elements: sysiems assurance guidance, team work, 
techniques and tools. 

Various parts of our organisation had been pressed to quality assurance (1SO9000) and 
some techniques by customer demand. Our thrust in 1987 was based on a training 
programme for all employees, using Philip Crosby's Total Quality philosophy and 
training package which is founded on the four absolutes of: 

• conformance to requirements 

• system of prevention 

• standard - defect free 

• measurement - price of non -conformance. 

1 will not dwell on quality assurance (BS5750/ISO9000) or the techniques and tools of 
quality except to say they are essential to provide a hard disciplined and systematic way 
of applying the philosophy of TQ, to analyse, improve and measure process variability 
and capability. 

Our experience with the Crosby approach in 1987/1988 was long on philosophy, and 
a fine start giving us a common language, in many national tongues; but we had to 
develop our own answers to the key question "How to do it?" 



TOTAL QUALITY IMPLEMENTATION 



55 



RECOMMITMENT 



In late 1988 we asked ourselves, "how well are we meeting the requirements?" 

Not only did we not have clear answers, but many parts of the organisation had mixed 
messages, or they said they had. We conducted a series of workshops asking three basic 
questions: "Where are we now?"; "Where do we need to be?"; "What do we have to do 
to get there?" 

We had learnt some lessons: 

♦ Clear demonstration of commitment at the top is essential 

♦ TQ implementation can result in stress at all levels 

♦ Fear and insecurity can be outcomes 

♦ People don't like to admit problems and failures 

♦ Problem prevention is not as much fun as problem fixing. 

Also we realised the need to focus our quality improvement on a project by project basis. 

A major requirement was to take ownership and, to this end, we had started and rapidly 
completed a totally new in house customised training package and process. It was 
founded on enhancement of the basic Crosby philosophy but added our own examples 
and two special new features. 

We emphasised the use of good quality assurance and quality tools and techniques to 
focus on team projects - the hard issues. We also realised the need to address the main 
cultural barriers - the soft issues. 

So, the two special features were the need to address culture and training in a TQ 
context. 



What is culture? We discovered tLat everyone had their own view but no one could 
define it in terms suited to a multi-national industrial organisation which covered several 
countries each with its own ethnic and social culture. We did produce a model, which 
viewed culture as a product of process, structure and behaviour. 

We recognised that our recommitment process needed to address the behavioural aspect 
in particular. 



We used an in-house team, assisted by external training consultants, in a joint project 
which developed a comprehensive modular training package. 
Our training was based on 3 main premises: 

♦ that all work is a process and the horizontal value chain is the focus for teamwork 
and leadership. 

♦ good prevention systems hold gains by preventing regression. 

♦ to continuously improve, people must be engaged to drive an innovative dynamic, 
which we called the Four Stage Cycle of Continuous Improvement - the core of 
our TQ approach (see Figure 9.3). 



CULTURE 



TRAINING 




56 



IMPLEMENTING QUALITY IN ORGANISATIONS 



But the follow-through depends entirely on the line team leaders. To assist in this we 
used our own line managers as trainers, which required a huge investment in selection, 
training and now support and maintenance. Conflicts of time and line management 
participation become important - but we believe these pain barriers are unavoidable if 
ownership is to be achieved. 

Quality has to become part of "business as usual" to facilitate the changes from 
segmental, functional and often vertically focussed people, overburdened by systems, to 
a business and work approach based on innovation and external customer requirements. 

As we were about to deliver our new package, the BP Group initiative of "Project 90" 
gave new emphasis to the key factors which should be addressed to stimulate and get best 
effect from TQ: process, structure and behaviour. 

We had recognised behaviour in our new training modules but Project 1990 re- 
emphasised this in terms of a BP group vision: "The most successful oil company" and 
some explicit values related to a variety of stakeholders: employees, suppliers, 
community, customers and shareholders. 

We had begun restructuring and delayering. On the behaviour side, we have resolved 
that our employees and managers must become capable of a number of essential 
behaviours: open thinking, personal impact, empowering and networking. These are now 
incorporated into our staff appraisal, development, reward and recognition processes. 



1. Define requirements 



4. Foiiow-up 




2. Assess 
& plan 



3. Implement Improvement 



Standards 



ADD 



Methods 



INPUTS 




Product 


— \ 


Service 




Information 






Facilities 



Training 



OUTPUTS 



Product 
Service 
Information 



C 
U 

s 

T 
0 
M 
E 
R 



t 


Process management 


t 




Prevention not appraisal 





Figure 9.3 Cycle of continuous improvement 



TOTAL QUALITY IMPLEMENTATION 



57 



TRAINING AND CULTURAL ISSUES 

Reference has been made to the new training elements addressing behaviour and 
leadership. The training process in all of this was deliberately interactive and the Board 
were the first to use the new material by setting aside two whole days. 

We chose to build the training on well established material, some of which was not new 
to many of our managers. They still need reminding of the 110 role models in our 
20,000 strong organisation. 

The training process was founded on line management creating the need for training, 
taking a lead in it and the support function satisfying that need - with good material and 
a process for training line managers to be the trainers. The fundamental lesson we got 
over about training, was that it is the beginning of a process and team leaders must own 
it and follow up if the training effort is to be valuable. In short "behaviour breeds 
behaviour". 

The many elements that go to provide a TQ culture and the essential behaviours were 
identified in our material which was based on a range of theoretical perspectives: 

♦ Team roles and group dynamics: leader, worker, achiever, reflector, maintainer. 

♦ Teams and action -centred leadership: forming, storming, norming, performing. 

♦ What makes good leaders: drive, dedication, competitiveness, honesty, realism and 
maturity. 

♦ Effective leadership: vision, communication, trust, self-knowledge and 
positiveness. 

♦ Continuum of style: tells, sells, consults, shares and delegates. 

We have made a good start! Four years along our journey we had been reinforced by a 
BP group initiative that reinforces the process focus with the need to change behaviour. 

So how is it in the real world of a large organisation; coping with recession? Here are 
some of the painful issues we continue to address and which require commitment and 
participation from the top. 

Do the role models really value wealth creation and team work or has short term profit 
and self interest aggravated the class barriers to team work? In business terms, have we 
replaced "captains" of industry with "pirates"? 

1 am grateful to Rosabeth Moss Kanter (1985) for listing some relevant dilemmas and 
paradoxes: process needs management, participation by command, top orders middle to 
do, empowerment with resources, delegation not abdication, need for a 25 hour day. 
participation threatens status, what's in it for me, teams equalise individuals, belief in 
team synergy. 

The changes we seek will only occur if we all perceive a need for a better world and 
a way to get there. We recognise there are behaviour issues which have to be tackled. 
Jind top management must lead the way. We can summarise the key forward issues: 
behaviour, line management ownership, staff selection, staff appraisal, training, promotion 
policy, recognition, continuous education, internal marketing and leadership participation. 

CONCLUSION 

Is it worth it? That is not the real question; we have no choice. Some of the issues are 
national, social and economic ones, but we have to bite the bullet. Total Quality seems 
to be the right vehicle. But it is worth it too! Despite most of what I've said about pain 
and barriers, there are gains and successes which prove the worth of embarking on TQ 



58 



IMPLEMENTING QUALITY IN ORGANISATIONS 



as the process for underpinning all enterprises . 

Let me finish with some good news, which encourages us to conclude that our journey 
so far has been successful. Two hundred and fifty quality line trainers, 90% certified to 
ISO 9000, a common language, changing behaviour, own training material/processes, 
improved supplier and customer alliances, quality cost savings approximately £120m over 
3 years, all bear witness to the progress made. 

Finally I mentioned the recession and business performance. You will note that in 1992 
we just broke even. We believe that proves the value of embarking on TQ. In the last 
downturn of the early 1980's we made huge losses during a less severe recession. 

References 

Kanter R M (1985) The Change Masters: corporate entrepreneurs at work Unwin 
Paperbacks London 



10 Developing the European quality model 



John S Oakland and Leslie J Porter, University of Bradford Management Centre, 
Bradford, UK 

INTRODUCTION 

The University of Bradford's Management Centre is one of Europe's oldest and largest 
business schools. The Management Centre pursues an extensive programme of research 
that has earned an international reputation in many fields of study. It is a fully integrated 
business school, providing comprehensive and innovative programmes of management 
education for undergraduates, postgraduates and post-experience students. The interaction 
of students with experienced managers and staff from diverse educational and industrial 
backgrounds provides a stimulating and creative environment. 

The Management Centre has a matrix structure with seven Professors, responsible for 
the subject groups of economics, financial management, international business, marketing, 
organisation behaviour, production and operations management, and total quality 
management. Non -professorial Programme Chairmen manage the undergraduate* 
postgraduate (MBA), doctoral, and executive development (EDP) programmes. There is 
also a Director and Assistant Director of the Centre who are both professors. The 
Director, Assistant Director, Programme Chairmen, and some Professors also sit on an 
"Executive Committee", which is the decision-making body of the Management Centre. 

There are approximately eighty full-time academic and related staff, fifty non-academic 
staff, five hundred undergraduates, three hundred full-time and part-time MBA students 
and thirty doctoral students on site. Almost two thousand executives and managers from 
industry, commerce and the public sector pass through the EDP each year on various 
types of short course programmes, including the Executive MBA. 

fhe establishment of the industry funded European Centre for Total Quality 
Management (ECTQM) in 1987 was a major innovation. The ECTQM is actively 
involved in research, teaching and advisory work in all areas of quality management. A 
wide range of students, including those on executive development programmes, 
experience TQM training and education at Bradford. The Centre is a member of EFQM 
and the TQM group are involved in many EFQM activities. 

In the summer of 1990 it was decided to introduce TQM into the Management Centre 
itself using the expertise of the European Centre for TQM. The 1990*s will be a very 
challenging phase in the life of the Management Centre. Business schools generally 
operate in an increasingly competitive environment and the Management Centre's future 
success will depend on the commitment of all its staff to the improvement of its 
processes, 

AWARENESS AND ORGANISATION FOR TQM 

The seven Professors held two TQM strategic planning workshops to examine the 
feasibility of introducing TQM into the Management Centre. A new mission statement 
was developed and the factors critical to the achievement of the mission were identified. 
There were seven critical success factors (CSFs) and associated critical processes were 
listed. 

A model of TQM, figure 1 0.1 . covering the main areas of the criteria of the European 
Quality Award was adopted for implementation at the Manager icnt Centre. 



60 



IMPLEMENTING QUALITY IN ORGANISATIONS 




Fig'jrc 10.1 The Bradford TQM model 



The consensus view was that TQM at the Centre should involve: 

• Everyone striving to meet customer requirements - internally and externally. 

• Managing business processes. 

• Continuous improvement in everything we do. 

• Open good communications. 

• Participative management style and involvement of everyone. 

• Documented, auditable systems for the way we do things. 

• Teamwork, built around our processes. 

• Training and education in the identification of processes and the tools and 
techniques of improvement. 

• Empowering people to act wherever improvements can be made. 

It was recognised that the necessary culture change will happen slowly and that the 
Management Centre's total quality culture would not be built overnight. There was a 
strong belief among the Professors that the TQM approach could be used at the Centre 
in a disciplined approach to never-ending improvement. 

It was also recognised that to devise and implement a TQM process takes considerable 
time and dedication and it must be given the status of an executive project. It was also 
essentia! that any TQM initiatives be fully integrated into the Management Centre's 
operating philosophy and management systems. The Professors suggested to the 
Executive Committee that the Centre should embark seriously on a TQM implementation 



DEVELOPING THE EUROPEAN QUALITY MODEL 



61 



programme, but pointed out that the commitment would be continually questioned and 
be weakened, perhaps destroyed, by any failure of the 'senior management* to support 
the initiatives. 

THE QUALITY COUNCIL 

It was decided that a Quality Council (QC) should be formed to guide the TQM process. 

Purpose 

The main purpose of the QC was to develop the Centre's culture into one of Total 
Quality, thereby ensuring that we identify, understand, and achieve our "mission". 

Charter and Responsibilities 

The members of the Quality Council were drawn from various levels in the organisation. 
The Council works within the strategic framework laid down by the Professors and the 
Executive Committee. The membership is the Director, two Professors (including the 
Professor of TQM as facilitator, and chairperson) two Programme Chairmen, a Lecturer, 
a Secretary and a Computer Officer (technician). 
Their responsibilities include: 

• Updating the Mission Statement 

• Identifying the Critical Success Factors 

• Providing overall strategic direction on TQM for the Management Centre 

• Establishing plans for TQM implementation 

• Setting up Process Quality Teams and Quality Action Teams to make 
improvements 

• Reviewing progress and plans for quality improvement 

• Revising plans for the development of TQM and process improvement 

The Council acts as a senior problem-solving group. It holds meetings monthly, 
following the meeting of the Executive Committee to review quality strategy, 
implementation, progress, and improvement. 

First Tasks 

The QC reviewed the previous mission statement and found it to contain suitable 
components for the development of a new mission statement and critical success factors 
(CSFs). These were set down as follows, including a list of "Stakeholders": 

Mission Statement 

To be a growing centre of excellence in teaching and research in the disciplires or 
management. 

To improve the practice of management, world-wide. 



62 



IMPLEMENTING QUALITY IN ORGANISATIONS 



Critical Success Factors (CSFs) 

1 We must have a UFC research rating of 5. 

2 We need a demonstrably excellent reputation for an innovative practical 
approach. 

3 We must have products and services which meet current and future market needs. 

4 We need financial independence. 

5 We need an excellent infrastructure. 

6 We must have a stimulating and rewarding work environment. 

7 We must achieve a critical mass of quality staff. 

8 We need quality inputs. 

These were deemed to be all necessary and together sufficient, for the Mission to be 
achieved. Some are directly measurable (eg UFC research rating) , some are aspirations, 
hopes or fears. 

Stakeholders 

These are: students; staff (academic and others); state; industry/business/public 
organisations; professions; university. 
Current Aims were derived from the Mission and the CSFs and are as follows: 

To achieve a University Funding Council (UFC) research rating of 5 in 1993. To 
improve the management processes in the external world by research, teaching and 
advisory work. 

To be the 'number one* UK business school, in terms of the services and products 
offered to targeted markets. 

Be a total quality business school which fully involves and develops all employees 
and revolves around never-ending improvement of processes to meet customers" 
requirements completely. 

To empower employees and teams to act in making continuous improvements. 

The Quality Council also identified 28 critical processes. These are the activities which 
must be carried out especially well in order for the CSFs to be achieved. The Council 
then produced a 'process quality matrix* in order to prioritise critical processes for 
improvement. This identifies which processes have a high impact on each CSF. Again 
the necessary and sufficiency rule applies. The matrix also showed the subjective quality 
ranking given by the Council to each process on the scale: 

A Excellent Performance 

B Good Performance 

C Fair Performance 

D Bad Performance 

E Embryonic Processes 

A second matrix showed the results of this work - a plot of the quality of each process 
against the number of CSFs. showing the critical processes in most urgent need of 
attention. 

7* '« 



DEVELOPING THE EUROPEAN QUALITY MODEL 63 

All 28 processes were grouped under seven headings and these main groupings given 
generic titles. Seven Process Quality Teams were set up to manage these processes. 

PROCESS QUALITY TEAMS (PQTs) 

The PQTs own and manage the process improvements. 

Purpose 

To define certain business processes and to set them up to run perfectly. 

Each PQT Chair will form a team of approximately 8 staff. Their responsibilities will 
include: 

• Breaking down and describing the assigned critical processes 

• Prioritising and selecting processes for improvement 

• Setting up quality action teams (QATs) 

• Reviewing and supporting QAT activity, 

QUALITY ACTION TEAMS (QATs) 

The actual detailed improvements will be carried out by QATs. 

Purpose 

To define and improve any particular process assigned by a PQT. 




Figure 10.2 Organisation for TQM 



7£" 



64 



IMPLEMENTING QUALITY IN ORGANISATIONS 



Composition 

6-10 members of staff, selected by a Team Leader, representing all those involved in the 
assigned process. The team leader will be selected and asked by the PQT to form the 
team, preferably using a flow chart of the process. It is hoped that every employee at the 
Management Centre will be involved in quality improvement through a QAT (or PQT) 
at some point. Their responsibilities will include: 

• Drawing a flow chart of the process to identify its customers and suppliers 

• Identifying measurement points 

• Measuring and comparing results with requirements 

• Improving the process and documenting it 

This teamwork structure, which is shown in Figure 10.2, should ensure the widest 
possible involvement in the TQM process and its effective implementation. 

INTRODUCTION AND TRAINING 



Everyone took part in a one day workshop to explain what had been done by the Council 
and the plans described above. There were several workshops to accommodate all staff, 
led by the Professor of TQM. 

Further training, on process improvement methods, will be provided to the individual 
teams on an "as needed" basis. It was expected, however, that academic staff at a 
management centre should be familiar with techniques such as flow charting, force-field 
analysis, Pareto analysis, brainstorming, cause and effect analysis, and the use of simple 
data presentation methods. 

The Quality Council prepared a very brief force-field diagram of the pressures for and 
against TQM at the Management Centre (Figure 10.3). 



TQM 



Pushing For 

If we don't do it there will be disaster - 

It will affect promotion 

Quality Council 



Professors committed to TQM - 
PQT's have chairs 



Share ideas with AUT - 



Improves efficiencies and effectiveness ■ 



Quality Action Teams to be formed - 
- wjih equal opportunities 



Pushing Against 



« : We are doing bits of it anyway 

What about promotion? 

Existing Management Structure 



* AUT will be against 

* Current Appraisal Schemes 

* Why do extra work? 

4 No equal opportunities 



Figure 10.3 Force field analysis 



72 



DEVELOPING THE EUROPEAN QUALITY MODEL 



65 



It was agreed at the senior level that only by measurement would people know when the 
Centre had succeeded in its Mission and in meeting its aims. To measure the start point, 
a survey of the staff's perceptions of quality at the Management Centre was carried out. 
The survey addressed all the key areas of the Bradford TQM model. Many positive 
things showed in this survey, eg people taking pride in the Management Centre, but the 
survey also highlighted many areas for improvement. Deficiencies in teamwork, 
communication, systems and measurement were identified. The most alarming statistic 
was that 88% of the respondents did not feel that the senior staff had established a clear 
policy and strategy to improve quality. Clearly this illustrated that the ground was fertile 
for a TQM initiative. 

The TQM training programme commenced with a teamwork workshop for the executive 
committee and the Professoriate - see Figure 10.4. 

The senior team profile had previously been exposed to the Myers-Briggs Type 
Indicator (MBTI) which is a psychological measure that attempts to explain the central 
aspects of people's personality, both to the individuals themselves and to their colleagues. 
An understanding of the MBTI measures helps to promote effective teamwork. This 
profile was reviewed at the workshop together with the TQM concepts and models. The 
action plan for implementation of TQM at the Management Centre was also reviewed. 

Four TQM workshops have been run to date and 85 members of staff have received the 
first stage of their TQM training. The objectives of the workshops were: 

♦ to enable staff to share information on their work and associated quality 
problems, 

♦ to develop staff understanding of the concept of TQM, 

♦ to have fun, 

♦ to start building more effective teamwork for quality improvement. 



Strategic planning 
Workshop 




TQM strategy 




TQM teamwork 
Workshop 




Staff TQM 
Workshops 



-^Staff awareness7"X 
4 understanding and ) 
involvement 



Figure 10,4 The training programme 



? 73 



66 



IMPLEMENTING QUALITY IN ORGANISATIONS 



The outcome of these workshops was that staff were encouraged to participate in the 
TQM process at the individual and team level. A team approach to continuous 
improvement around the Management Centre's key processes was launched by the 
Quality Council, once a core of staff had been trained. All staff have been asked to do 
some work on improvement immediately on an individual basis, using the tools and 
techniques of TQM learnt on their workshops. 

THE WAY AHEAD 

The first phase of the TQM programme was concerned with obtaining the commitment 
of the Professoriate and executive committee to the principles of TQM. This commitment 
to TQM has been reinforced by the following statement from the Director of Bradford 
Management Centre, Professor David Weir: 

"We are entering a very challenging competitive phase in the life of the Centre, and we 
are committed to creating a quality environment. Our success over the next few years 
depends on our ability to liberate the enormous positive energy within the Centre to meet 
our customers' requirements. Universities, old and new, all over the UK, Europe and the 
World, will have to adopt never-ending quality improvement processes to succeed. We 
want to be one of the first Departments to use TQM to gain a clear competitive advantage 
over our competitors." 

A strategy for implementing TQM was developed at a strategic planning workshop. A 
teambuilding workshop was used to weld the Professoriate and executive committee into 
an effective senior management team. A Quality Council has been formed to guide the 
programme. The commitment to TQM was communicated to all the Management 
Centre's staff through a series of TQM workshops. The development of a TQM culture 
has started to evolve through the training programme - see Figure 10.4. 

How will the necessary improvements be effected in these areas? Some of the harder 
tools and techniques of TQM will undoubtedly be called into play. The role of quality 
management systems such as ISO 9000 can make a significant contribution to 
improvement. The review of existing foimal and informal procedures and the adoption 
of best practices is a fundamental step in the TQM process. It is important to produce 
a documented system of what is actually done. The danger in all improvement activities 
is that people prefer to improve processes first before writing down what is done. This 
results in an indisciplined approach and progress is usually not sustained. Measurement 
and recording systems required by a good quality system will also result in more effective 
operations. Many organisations in the educational sector are considering the contribution 
of ISO 9000 to quality improvement. The role of teamwork and the tools o<" TQM will 
also make a major contribution to the programme. 

The combined effects of good systems and people working in teams using the tools and 
techniques of TQM sustain the process of continuous improvement. The process quality 
teams and quality action teams will be empowered to drive the improvement process 
forward. 

Initiating and sustaining a TQM programme in a large and well established academic 
institution raises many interesting questions. Hopefully, the TQM initiative at Bradford 
is fully integrated into the Management Centre's operating philosophy arid management 
systems. The TQM awareness level of all staff has been developed by a systematic 
training programme. As the programme evolves, further development of the quality 
model may be required. The contribution to quality improvement of the vari ms elements 
of the European Quality Model will be evaluated. The ultimate success of the programme 



74 



DEVELOPING THE EUROPEAN QUALITY MODEL 67 

will depend upon people working effectively together, with the shared values of TQM, 
to achieve the shared aims critical to the future of the Bradford Management Centre. 

The TQM work will now be extended to the main University and work has started on 
workshops for the Vice Chancellor, Registrar, Pro-Vice Chancellors, Deans, and Heads 
of Departments, 



11 Strategic quality management: if I pull 
it up the flagpole will anyone salute? 



Robert James Newton, Greater Peterborough Training and Enterprise Council 
(GPtec), Peterborough, UK 

INTRODUCTION 

In recent years there has been a proliferation of articles on organisational development, 
very few of which have drawn the essential links between quality, strategy and policy 
which this paper addresses. As any reader of Business Week would know some of the 
highly proclaimed, quality conscious companies identified by Tom Peters and colleagues 
in 1983 have since experienced major problems. As Tregoe and Zimmerman (1989) have 
so accurately proclaimed, the problems "did not stem from operational inadequacies but 
from strategic deficiencies. These companies failed because their vision of quality was 
flawed". 

To avoid this strategic trap, into which I expect many more British companies to fall, 
this paper advocates the understanding and use of the technique commonly referred to as 
Strategic Quality Management (SQM). Nowhere is the use of SQM more appropriate 
than in the Training and Enterprise Councils (TECs) who are not only in the constant eye 
of their local communities but are under the rigorous scrutiny of various government 
agencies. Furthermore in preaching the significance and importance of quality assurance 
to local businesses each TEC must be seen to be practising it within its own organisation. 

Like Tregoe and Zimmerman, 1 do not wish to criticise the prevailing wisdom regarding 
operational excellence and the use of various quality operational systems to bring this 
about, but to emphasise that the last thing a company headed in the wrong direction needs 
is to get there more efficiently. This paper therefore addresses an apparent imbalance in 
the literature in respect of quality debate about how organisations, like those of the 
Training and Enterprise Councils, can improve company performance through the project 
management triad of strategy, policy and quality. 

STRATEGY 

As Stephen Connock (1991) has pointed out, Strategic planning processes have increased 
in importance in organisations in recent years, so too has the meaningfulness of strategic 
notions such as vision and mission. Very rarely, however, does an organisation take 
pains to explain its vision of quality and how such a vision determines the 
interrelationships of strategy and policy. The particularly interesting thread of values 
which weaves its way through this project management triad is ignored in this paper, not 
for its lack of priority in quality debate, but because its significance in organisational 
development is paramount and is deserving of a paper in its own right. 

For Training and Enterprise Councils, the very effective trap which they must avoid at 
all costs is to allow the obvious intensity of operational pressure to overwhelm their 
strategic vision. In order to preserve the long term, strategic well-being of an 
organisation, Training and Enterprise Councils must go beyond the quarter by quarter 
operational outlook. 

For Greater Peterborough Training and Enterprise Council (GPtec), strategy is the 
description of what the organisation is responsible for delivering and how we intend to 
determine priorities in the light of available resources and conflicting commitments. The 
vision of quality shared (in value terms) by the senior management group of GPtec is that 



76 



STRATEGIC QUALITY MANAGEMENT 



69 



it is a strategic business management function which will help us, and other companies 
with whom we work closely, to change our cultures. In this respect, our SQM 
programme began with the senior management team taking responsibility for laying 
down objectives in the form of policy and strategy. 

The central thrust of GPtec's approach to implementing our total quality policy is, 
therefore unequivocally, a strategic one. Continued success of our operation depends as 
much on the quality of our strategic thinking and on how well this thinking becomes 
imprinted on every decision made within the TEC, as it does on the effectiveness of our 
internal day to day operations. At the risk of using definition as explanation, for GPtec 
SQM is the process by which we can translate our very clear vision into action. Or put 
differently, by the use of SQM techniques we are able to take the strategic direction 
formulated by the renior management group and translate this direction into a quality 
operation. 

In committing ourselves to quality the usually hidden relationship between strategy and 
structure becomes manifest - structure follows strategy. Thus, the quality of strategic 
decisions must be management's first priority in bringing about effective organisational 
structural change. In putting quality of strategy above all other management processes, 
GPtec has shifted away from a dominant Civil Service culture of cell-based operation to 
a team-driven culture commensurate with the much more business-like framework of the 
TEC marketplace. In the development of our SQM organisational culture we have shifted 
from a hierarchy based on status towards one based on service, explicitly recognising the 
notion of both internal and external customer satisfaction. 

IMPLEMENTING SQM 

The major lesson within the domain of quality, and in particular the SQM context, is that 
organisational restructuring is an integral part of the strategic management process and 
cannot and should not be divorced from it if quality - defined simply as meeting customer 
requirements, - is to be an overarching strategic aim. 

As Miller and Innis (1990) have so lucidly put it. "There is little point in having a well 
worked out strategy which commits the service ... to the improvement of quality if this 
message is not thoroughly understood throughout the whole organisation" (p 34). As the 
senior management group of GPtec therefore, our first essential task was to raise the 
awareness of all our staff about the concept of SQM and how we firmly believed that 
quality improvements continually derive through staff's commitment to it. To this extent 
our guidirg principle in achieving this outcome was to recognise that. "In involving 
people in the debate on quality and on strategy, it is therefore important to specify 
precisely the nature of the issues, the assumptions behind the approach and type of 
response expected from the people being communicated to?" (ibid p 36). For this reason 
we chose the communication process of open forum debate. 

To contextualise the debate and to give the feeling to staff that the meeting was but the 
start of an explicit use of SQM principles we devised a company SQM project called 
'MORALE'. To foster understanding that the senior management group were fully 
committed to this SQM project and to convey our desire for far more than lip service to 
be paid to it. the implications were explained in detail to all staff. 

Tne starting point for the presentation was that ownership and genuine empowerment 
of staff is. in essence, the major focus of a strategic quality management approach. 
However, for this to be not only said but implemented it was pointed out how a SQM 
stance to organisational development suggested the use of what we now call a quality 
project management system. It is a firm belief of the senior management group of GPtec 



77 



70 



IMPLEMENTING QUALITY IN ORGANISATIONS 



that such a quality system necessitates the use of modern management techniques and 
information technology as major ways of preventing poor performance and assuring 
quality of service to all our customers, whether they be internal or external. Furthermore, 
this project management structure will avoid the decision taken by so many other 
companies, often by default, to maintain an imbalance of executive time and effort spent 
between finance and strategy. 

The essential purposes of the company Morale project are: to involve ail staff in the 
strategic challenge facing all TECs; to commit staff to the view that short term gains are 
no substitute, in quality terms, for long term security and getting the TEC firmly 
established; to inform staff of strategic decisions made by the senior management group 
only after which such a company project can begin; that the largest failure cost of all 
occurs when bad quality strategic decisions are made. 

Strategic quality management conforms to the precept that the real objective of any 
quality assurance project is to get decisions made correctly by all the project team 
members involved. The Morale project is GPtec's attempt to implement a quality 
assurance programme which runs on the twin engines of strategy and policy at action and 
decision making levels, involving all staff. Hopefully, as is the intent behind the project, 
we will avoid the sorts of quality problems which beset most organisations namely; lack 
of top management support, lack of training, lack of discipline, lack of resources and lack 
of time. 

The introduction of a project management system as a team managed operational 
mechanism, permits the full involvement of all staff with clear understanding that quality 
of performance is measured against team determined project objectives which must be 
strategically aligned to GPtec's corporate and business plans. Quality of operation cannot 
be gauged without reference to strategy and policy - the driving forces of service 
delivery. 

The significant involvement of each and every member of staff in a project area team 
with joint responsibility for writing performance objectives and establishing quality 
standards gives the greatest possible sense of ownership that staff can feel. Taking this 
one step further and allowing teams to set targets based on an analysis of GPiec's annual 
business plan, should also help in improving strategic decision making at operation team 
level. 

The essential staff training and deveiopment profile, that so depicts any quality 
organisation, will emerge from the strategic base of this SQM project system and 
hopefully result in the growth of individuals as well as in the growth of the TEC itself. 
Future decision making by senior TEC management may thus be influenced in respect 
of strategic plan development. Decision making is a process that is dependent upon the 
quality of the input as much as the quality of the process itself. 

By the use of management auditing it should be possible to measure the effectiveness 
of this SQM project - and evaluating vision and mission is a rare phenomenon in British 
enterprise today. Pursuing a SQM approach to translating vision into action is about 
articulating the long-term human resource goals, core values, key behaviour and 
underlying philosophy which derive from, support and complement the business mission 
and strategies of GPtec. 



This paper suggests that greater weight of emphasis should be given to quality discussion 
at a strategic level. When quality, policy and strategy arc brought together in a company 
management model, operational quality flows from it. The quality philosophy 



CONCLUSION 




STRATEGIC QUALITY MANAGEMENT 71 

underpining GPtec's SQM model is to practice what we preach. Our quality tenet is: if 
it doesn't add value its a waste of time. Our quality slogan is: if it is not in the strategy 
we don't do it. 

References 

Connock S (1991) Human Resource Vision: managing a quality workforce Institute of 
Personnel Management 

Miller A J and Innis R B (1990) Strategic Quality Management: the strategic 
management of a quality further education service Crown Copywright 

Smith R A and Tobia P M (1989) Putting a Winning Strategy to Work Simon and 
Slater Publications 

Tregoe B B and Zimmerman J W (1989) Vision in Action: putting a winning strategy 
to work Simon and Schuster New York 



* 79 



12 Profits and pitfalls in establishing a 
qualytechnic 



John Heap and Hazel Solomon, Leeds Metropolitan University, Leeds, LSI 3HE t 



The University, like other academic institutions, has a sophisticated quality assurance 
system for its academic provision. Many of the supporting functions - personnel, finance, 
etc - had been provided, at least in terms of policy and structure, by the local authority. 
Incorporation of the Polytechnic as a higher education corporation required the institution 
to take over and develop many of these services. Initially, attention was focused on 
simple operational provision - we needed administrative, personnel and financial systems 
in place and the timescale to provide them was very short. The move to encompass such 
systems within a quality framework simply resulted from : 

• a recognition that the total experience of students was significantly -Influenced by 
such systems - in some cases (such as catering) almost as much as it is by the 
academic provision 

• dissatisfaction with the nature and level of service provision - arising, in part, from 
the haste with which they were, of necessity, introduced 

• a critical mass of staff who were aware of the wider quality revolution taking 
place within industry and commerce. 

The Assistant Director of the University who has responsibility for academic quality 
called together a number of people who he knew had an interest in, or experience of, 
quality matters. He established a quality steering group to evaluate various approaches 
to quality assurance and to attempt to identify a direction for the university to improve 
the quality of its non-academic services. This group included the deputy chair of the 
board of governors, who had recently undertaken an exercise within his own 
(manufacturing) company to apply for and receive BS5750 accreditation. The main 
debate centred on the difference between procedurally-based approaches (such as BS5750) 
and culturally-based approaches such as Total Quality Management (TQM). 

At the time, there was no clear drive for quality improvement coming from the senior 
management of the institution and it was felt that TQM. in particular, could only be 
successful if such a drive existed. Additionally, the steering group was concerned about 
the nature and pace of change that the staff of the university had already been subjected 
to over the previous two or three years. 

Accordingly, it was felt that a procedurally-based approach would : 

• allow quality issues to be raised and discussed in a relatively 'hard -edged' manner, 
without the need for cultural change or significant resource input 

• form the basis for education of the university management and for later 
consideration of cultural change to initiate a more broadly- based quality movement 

• assist in the process of service definition for many of the still relatively new 
systems and procedures. 



UK 



INTRODUCTION 




ESTABLISHING A QUALYTECHNIC 



73 



The concept of a university-wide initiative was sold to the senior management team 
through a series of informal meetings of the more senior members of the steering group 
with the senior university managers and culminated in a seminar for the entire senior 
management team of the university. Even after this process, it is fair to say that there 
were reservations amongst some members of the management team about the relevance 
of industry-based quality approaches to a higher education institution. 

It became obvious that certain areas of operation were suitable for immediate 
application of a procedural ly-based approach. But, allowing for the fact that most of 
those within the institution that had knowledge or experience of quality matters had jobs 
that would allow little time for direct involvement, resources were needed to prioritise 
and co-ordinate work in different areas. 

Short course provision was identified as an area in which some form of quality 
accreditation would have immediate marketing benefits and a bid was made to the 
PICKUP agency for funding to support the establishment of BS5750 accreditation for the 
short course provision. The success of the bid resulted in the appointment of a quality 
development manager whose brief was to examine the short course area and to make 
recommendations which may result in eventual BS5750 accreditation. Since many of the 
supporting activities of short courses are also supporting activities of other parts of 
university provision, this actually resulted in a wider consideration of quality generally. 
The Quality Development Manager is thus involved in a number of specific projects 
within the university initiative. 

This may sound straightforward. However, the nature of the debate on quality within 
the university meant that there was a some confusion amongst many people (including 
senior managers) about the nature o f quality systems and about BS5750 in particular. 
There was certainly resistance in some quarters to the idea that existing systems and 
procedures could be described as quality systems or procedures and a commonly held 
view that any accreditation system must be bureaucratic and cumbersome. 

The real brief of the quality development manager was therefore to begin the process 
education and of attitude change that would result in positive work towards quality 
improvemeni, and then to act as driver for the short course accreditation project and as 
facilitator for other projects. This is bast considered in three main phases. 

PHASE ONE 

The process began with data gathering around the institution, interviewing key managers 
and a cross-section of staff to gain a greater understanding of their attitude towards 
quality. This sensing and networking was also useful in gaining access to areas for future 
work. Taking into account the size of the organisation, a realistic plan was put together 
of what could be achieved with estimated timescales - see figure 12.1. 

Pitfalls 

• Pace and diversity needed to be controlled 

• Wider quality debate concerned with the applicability of BS5750 to education 

• No quick results likely 

• Funding and budgets 



81 



74 IMPLEMENTING QUALITY IN ORGANISATIONS 

Profits 



A genuine concern for quality was evident 

Lecturing staff are very conscious of the need for quality 

Enthusiasm to see improvements 



SHORT TERM 



MEDIUM TERM 



LONG TERM 



To demonstrate results 



♦ To raise the profile of the 
Quality Initiative 

♦ To demonstrate what 
effect procedures 
can have 

♦ Pilot BS5750 in 

the Language Centre 

♦ Preparation of Quality 
manual 



BS5750 for short 
courses in the 
Schools/Faculties 

Quality Improvement 
Projects 



Widespread implementation 
Quality Systems 

Quality Improvement 
Projects 

Quality Polytechnic 



C-1 2 months 1 -3 years 7-1 0 years 

Figure 12.1 Aim and approach 



PHASE TWO 



Most of the short course co-ordinators were interviewed to gain an understanding of the 
current operations and to determine which area would be appropriate for a pilot study. 
After careful consideration, the Language Centre was chosen as the pilot area. This 
satisfied the criteria of being containable and had a good chance of success due to the 
existing systems and enthusiastic management. 

At this time it was also considered whether it would be beneficial to use consultants. 
It was felt that this would increase the likelihood of success due to their experience of 
applying the standard within an educational context. 

Pitfalls 



• Cost of consultants 

• Diversity of approaches for short course provision 

• Pressure to increase short courses to generate income 

• Limited incentives for staff to run short courses 

• Concerns that BS5750 was too bureaucratic or would impose a system. 

Profits 



• Increased chance of success by using consultants 

• Increased chance of success by selecting the pilot area wisely 

• Enthusiastic management in the pilot area 



82 



ESTABLISHING A QUALYTECHNIC 



75 



PHASE THREE 

This phase included application of the standard to tailored language courses and 
involvement in ongoing quality projects within central services, eg finance, reception and 
personnel. 

The approach taken for the pilot was to assess current practices in the light of the 
requirements of the standard and produce an audit report. All Language Centre staff were 
involved in the results of the report and the proposed approach. 

Drafting of procedures was then begun with substantial involvement from the Language 
Consultant w r ho manages the Language Centre. Once implemented, practices were to be 
audited against the procedures and modified. 

Pitfalls 

• Applying the standard to education 

• Labour intensive 

• Controlling the pace of all the quality projects whilst providing the support 
required. 

Profits 

• Accepted practices are questioned and improved 

• Staff became more involved in making systems more effective 

• Transfer of learning for future application of the standard 

• Budget was allocated for the quality initiative 

PRESENTATION TO MANAGEMENT 

To serve as a progress report and to encourage ownership which would be essential for 
assessment purposes, a presentation was given to management on the requirements of 
BS5750 and progress made. This presentation served to not only achieve the above aims, 
but also to encourage a debate around the direction of the quality initiative and the need 
to address cultural issues. 

CONCLUSION 

The strategy adopted to establish a Qualytechnic at Leeds has been successful to date. 
The aim was to begin with a systems approach which would lead to a greater 
commitment to quality. This has been demonstrated following the presentation to senior 
management. The systems approach has also benefited the Language Centre by 
questioning and improving current practices. These lessons can now be transferred to 
other short course providers. 



13 Student satisfaction and perceptions of 
quality 



P M Mazelan, D M Green, C R Brannigan and P F Tormey, University of 
Central England, Birmingham, UK 

INTRODUCTION 

The purpose of this brief paper is twofold. First, to provide a context in which to view 
the emerging importance of student-based information. Second, to briefly describe some 
recent research which illustrates how student-based information may be gathered and 
interpreted to reveal patterns of student satisfaction. These trends are valuable inputs to 
managerial decision-making and may be used to inform innovations in quality policy. 

PERCEPTIONS AND EXPECTATIONS OF QUALITY 

The quality debate in higher education traditionally focuses on whether profit-orientated 
private sector concepts can be readily transferred to the public sector. This is a complex 
issue but even the limited view of the student, as a customer or client, goes against the 
grain of many in HE. They are very reluctant to empower a group who are traditionally 
treated as passive recipients of education. 

Both the broad consumer perspective, that service quality can be equated to the extent 
to which the perceived level of service delivered matches consumer expectations (eg. 
Lewis and Booms 1983) and the more specific view that service satisfaction equals 
consumer perception minus consumer expectation, is not without problems in the HE 
context. However, the implication that satisfaction will result if perceptions are greater 
than expectations has a great deal of intuitive appeal. 

This balance between consumers* perceptions and expectations as a measure of service 
quality has been fairly widely-discussed (eg. Churchill and Suprenant 1982; McMillan 
1986: Parasuraman et al 1986) and appears to be useful. Nevertheless antagonists to the 
quality initiative often argue that in the HE context it is futile to attempt to define quality 
in this way because student expectations are poorly developed, variable and unpredictable. 
Validly measuring student perceptions and expectations is not a simple matter but can be 
approached in a systematic way to reveal useful information. 

To complicate matters further, antagonists to the quality initiative often point out that 
different stakeholders in HE (ie. students, teachers, managers and employers) will all have 
different sorts of expectations from a particular institution or service. Consequently they 
claim that different stakeholders will have disparate ideas about "quality" jeopardising any 
attempt at valid assessment. Clearly there is more than one client group to be considered 
in examining the perceived value of higher education, but this does not mean that these 
various perceptions cannot be systematically studied. 

Despite these conceptual caveats, two things are clear. First, the most important client 
group are the "front- line" consumers, the students themselves. Second, and equally 
important, it is evident that "quality" (however defined) will become the principal means 
by which higher education institutions will attempt to differentiate themselves from their 
educational competitors. 



84 



STUDENT SATISFACTION AND QUALITY 

THE WORK OF THE SSRU 



77 



Until recently, students' views of their academic experience had been of interest only to 
a handful of educational evaluators but increasingly the value of this information has been 
recognised. The realisation that students' attitudes and perceptions are not only 
interesting but may play a central role in shaping the decisions of those who manage our 
higher education institutions has widespread implications. 

Since its inception in 1988, the Student Satisfaction Research Unit (SSRU) at 
Birmingham Polytechnic has produced an abundance of information related to student 
perceptions of educational quality and their satisfaction with their educational experience. 
Our interest in quality and our attempts to measure it through student satisfaction have 
three broad aims. First, we are attempting to identify and examine the components of 
satisfaction and the components of quality and to consider how these two domains are 
interrelated. Second, we are investigating whether there is a common core to students' 
perceptions of satisfaction and quality. Third, we are attempting to model student 
satisfaction and quality of education and to represent this model in ways which are 
designed to inform management decision and policy-making. 

Nowadays, collecting information about student satisfaction in educational 
establishments is a growth industry. Information of this kind has the potential benefit of 
guiding managem :.n decision -making with respect to the planning and provision 0i 
educational services. Those HE managers who can clearly identify and subsequently 
measure client-centred quality will be able to capitalise on student-based information to 
support their claims for resources. 

Eliciting Areas of Student Concern 

One approach to directly eliciting students' views of the Polytechnic which has proved 
to be particularly productive over the last few years is a form of Nominal Group 
Technique which we have dubbed the "Group Feedback Strategy" (GFS). Essentially, 
this takes the form of a structured participative discussion with groups of students in 
order to provoke a span of student-nominated indicators of satisfaction and dissatisfaction. 

Experience has shown that engendering an atmosphere of frank and open 
communication encourages students to talk freely and honestly about what their education 
means to them. These sessions, facilitated by experienced group leaders, generally result 
in eliciting a wide and diverse range of satisfiers and dissatisfiers which are used on an 
annual basis to update the design of the instruments used to survey student satisfaction 
in the Polytechnic. The GFS exercise is used primarily as a device to validate the content 
of the questionnaire survey prior to it's annual administration, although the procedure 
itself produces some interesting findings. Some results of the GFS session are described 
in the next section. 

Three factors were used to define the sample frame: 
a Faculty - 8 categories in all (Health and Social Sciences, Built environment etc.) 
b PFC Classification * 3 categories (Postgraduate, Degree and Other) 
c Mode of Attendance - 3 categories (FulUime, Part-time and Sandwich). 

16 classes were representatively sampled from this frame and participated in GFS 
exercises, enabling the views of 265 students to be obtained. 

It is worth noting that the overwhelming majority of ■ tudents welcomed the opportunity 
to express their opinions in a confidential climate and were delighted to have some input 
into any future decision-making process. 



85 



78 



IMPLEMENTING QUALITY IN ORGANISATIONS 



Areas of Student Satisfaction and Dissatisfaction 

The outcomes of the GFS procedure were consensus, but wide-ranging, lists of satisfiers 
and dissatisfiers elicited from students in each of the 16 participating classes. In order 
to provide a framework for analysing these unstructured statements, each "good" or "bad" 
statement was allocated to one or more of 44 core activities which had been identified 

through synthesising categories used in our previous work. These core Ivities are 

reasonably exhaustive of students' concerns and are clearly associated with student 
satisfaction and dissatisfaction. The 44 core activities themselves represent four broad 
areas of student concern. The four broad areas are shown in Table 13.1 together with a 
mean percentage satisfaction score for each area (derived from the relative proportion of 
good and bad statements allocated to each area). 



| 1 Teachmg/Learning Activity Area 


36.9% | 


i 2 Personal/Interpersonal Activity Area 


76.5% 


[ 3 Using Facilities Activity Area 


17.6% 


| 4 Spending/Budgeting Activity_Area 


4.6% 



These results are interesting because they suggest that students have a natural tendency 
to regard some aspects of their educational experience as innately and potentially more 
satisfying than others. The Personal and Interpersonal area which includes such activities 
as "broadening my experience and outlook", "socialising with other students" and "talking 
to staff freely and openly" is seen in a very positive light (77%) whereas Spending and 
Budgeting has little potential for satisfaction (5%). Teaching and Learning Activities can 
provide some satisfaction (37%) whereas Using Facilities generally tends to be a focus 
for discontent (18%). In other words, students tend to stress negative aspects or 
grievances when considering their finances or using polytechnic facilities. In contrast, 
students are inclined to be more positive when considering their study experiences and 
even mow enthusiastic when reflecting on their personal and interpersonal experiences. 

Ratings of Importance 

In addition to expressing their opinions about their likes and dislikes, students were also 
required to rate each of the 44 core activities with respect to two aspects of importance. 
The first rating was of the importance of each activity to enhancing the effectiveness of 
study; the second rating was of the importance of each activity to enhancing students' 
personal sense of well-being. Part of our purpose was to investigate the relationship 
between these twin aspects of importanc e in order to provide information about students' 
priorities. 



Table 13.1 Areas of student concern 




STUDENT SATISFACTION AND QUALITY 



79 



The results revealed that students see some activities as consistently more important 
than others. Some activities were seen as critical to both study effectiveness and personal 
well-being. Some other activities were perceived as relatively unimportant in both 
domains; yet others were seen as differentially important to effectiveness of study or 
well-being. The two ratings of importance appeared to be tapping different underlying 
student perceptions. Table 13.2 below shows the five most and the five least important 
activities together with their associated correlations between ratings of importance to 
effective study and importance to personal well-being. 



Most Important Activities for Effective Study Rank r 

Understanding the content of lectures 1 0.30 

Receiving well organised and structured tuition 2 0.24 

Receiving feedback on assessed work 3 0.40 

Receiving guidance and supervision by staff 4 0.41 

Using time productively 5 0.45 



Most Important Activities for Enhancing Wcll-Bcing 

Paying my way/spending/budgeting 1 0.69 

Increasing my self confidence 2 0.67 

Having a base to leave personal property 3 0.59 

Receiving well organised and structured tuition 4 0.24 

Receiving feedback on assessed work 5= 0.60 

Broadening my experience and outlook 5= 0.40 



Least Important Activities for Effective Study 

Joining societies and groups 1 0.66 

Using the accommodation service 2 0.77 

Using student services (e.g. medical etc) 3 0.80 

Using canteen facilities 4 0.59 

Buying acceptable (cost, range & quality) food... 5 0.60 



Least Important Activities for Enhancing Wcll-Bcing 




Using the accommodation service 


1 0.77 


Joining societies and groups 


2 0.66 


Accessing relevant information from Faculty 




learning Centres 


3= 0.64 


Using student services (e.g. medical etc) 


3= 0.80 


Using Faculty Learning Centres 


5= C.6j 


Accessing relevant information from IT facilities 


5= 0.66 


Table 13.2 Most im/x>rtant and least important activities 



Mapping Satisfaction and Importance Ratings 

Both satisfaction and importance are essential ways of considering students' perceptions 
of their educational activities. Polcyn (1986) has suggested that importance and 
satisfaction can be represented as quadrants on a graph and each quadrant has different 
organisational implications. Figure 13.1 shows the four key areas of student concern 
mapped onto an importance/satisfaction graph. 



'87 



80 



IMPLEMENTING QUALITY IN ORGANISATIONS 



Higher Importance 









♦ Spending/ 
Budgeting 






+ Teaching/ 
Learning 

Quadrat A: 
Hjjh Importiooe 
LovSariR&ctkn 




♦ Personal/ 
Interpersonal 

Quadra* B: 
High Importance 
Ui^h Samfaoicc 


Lower 




Higher 


i 1 1 1 1 

Satisfaction . 

♦ Using 

Facilities 




H 1 1 1 1 

Satisfaction 


QadranrC: 
Low Importance 
Low Satijfacrioc 




Quadra D: 
Low Imparuocc 
High Sukitaioa 









Lower Importance 

Figure 13.1 Importance 'satisfaction map 

The graph shows thai both the Spending/Budgeting Area and the Teaching/Learning Area 
fall within Quadrant A (ic. High importance and low satisfaction). Consequently, both 
areas may be regarded as those where management may consider intervention a priority. 
Spending and budgeting is an area of increasing concern to students at Polytechnics and 
this is reflected in its position on the graph. Personal and Interpersonal Concerns fall 
within Quadrant B (high importance and high satisfaction) on the graph and this reflects 
students repeatedly- voiced opinion that being able to approach teaching staff and relate 
to them in a free and open manner is of great importance to their general levels of 
satisfaction. From the management perspective, the graph shows that it is necessary to 
maintain course organisation so that opportunities for good quality interaction between 
staff and students are optimised. Using Facilities can be seen to be the least important of 
the four broad areas and it can be seen to fall in Quadrant C (low importance and low 
satisfaction). Often it seems that good student facilities are weighted as more critical by 
management than they are by students themselves. The graph shows that no area falls 
within Quadrant D (low importance and high satisfaction) and this is probably an 
outcome of the GFS procedure itself which is designed to target issues of high importance 
rather than focus on issues of little or no importance. 

CONCLUSIONS 

These days, cost and quality arc two fundamental concerns in the quest for greater 
accountability in higher education and this has heralded public sector consumerism as an 



8S 



STUDENT SATISFACTION AND QUALITY 



81 



increasingly compelling force for change. As Pollitt (1988) has pointed out, the consumer 
approach can appear in many different guises. At its weakest it can range from being a 
visible but harmless cosmetic. Alternatively, it may manifest itself as a means of 
providing information to consumers or managers. It may even consist of real attempts 
to empower service users or inform HE managers. In the context of higher education, 
this new public orientation is particularly significant in stressing the importance of 
obtaining the opinions of stakeholders in the process. User feedback on quality of 
educational services is increasingly acknowledged as critical in understanding what 
quality means, although much work remains in developing appropriate methodologies and 
techniques to measure and use this feedback to maximum effect. 

One of the dangers in attempting to improve quality through students' views is that 
expectations about change may become elevated to unattainable levels. If resource 
constraints mean that managers are unable to respond to students suggestions for 
improvements, then skepticism and distrust are likely to become additional 
stumbling-blocks to change. It is our view that these dangers can be minimised if clear 
and practical methodologies and models are derived which link student satisfaction 
information, performance indicators, and management decision -making. 

References 

Churchill G and Suprenant C (1982) An Investigation into the Determinants of 
Customer Satisfaction Journal of Marketing Research 19(6) pp 491-504 

Lewis R and Booms B (1983) The Marketing Aspect of Service Quality in Berry L L, 
Shostack G L and Upah G D (Eds.) Emerging Perspectives on Services Marketing 
American Marketing Assoc Chicago 

McMillan J R (1986) Including Satisfaction Data in Health Care Marketing Information 
Systems in Cooper P D (Ed) Responding to the Challenge: health care marketing comes 
of age Academy for Health Services Marketing American Marketing Assoc Chicago 

MaisterDH (1985) The Psychology of Waking Lines in Czepicl J A, Solomon M R 
and Suprenant C F The Sen-ice Encounter: managing employee -customer interaction in 
sen ice businesses Lexington Books Lexington MA 

Morris Committee Report (1990) Performance Indicators: report of a committee of 
enquiry Chaired by Dr A Morris Polytechnics & Colleges Funding Council 

Parasuraman A, Zeithaml V and Berry L L (1986) SERVQUAL: a multiple item scale 
for measuring customers ' perceptions of senice quality Marketing Science Institute 
Cambridge MA 

Polcyn L J (1986) A Two Instrument Approach to Student Satisfaction Measurement 
College & University 62 pp 18-24 

Pollitt C (1988) Bringing Consumers into Performance Measurement: concepts, 
consequences and constraints Policy and Politics 16(2) pp 77-87 



83 




14 Experiences in the design and conduct of 
enterprise audits 



Douglas Edgar, Glasgow Caledonian University, Glasgow, G4 OBA, Scotland 
INTRODUCTION 

In determining an audit methodology to be employed there are a number of questions 
which have to be considered, questions such as: 

Is it to be "top down" directed by management or "bottom up" directed by the staff 
involved (ie who "owns" the audit)? 

What is its purpose? Is it to attract funding support or identify a present position or 
raise EHE awareness or identify those courses or groups of staff in need of enterprise 
development? 

To what extent is it to be quantitative (allowing comparisons) or qualitative (allowing 
insights) in nature? 

Is it to be course driven (via course committees) or subject driven (via subject specialist 
groups)? 

The audit undertaken at Glasgow Polytechnic, which was a pre-cursor to our main five 
year EHE project, tended towards a top down model in that its approach and 
methodology was directed by a management steering group. It was largely subject driven 
through subject specialist groups. It was essentially qualitative in nature, although a 
decision was made during the study to use a quantitative approach in questionnaires to 
course organisers. It was concerned with awareness raising and position identification. 
It commenced in mid session, late January 91, and concluded in November 91. 

No attempt was made by the audit team to define enterprise in the form of some verbal 
statement. However, the steering group's view of enterprise was about giving students 
greater responsibility for their own learning; about developing their transferable personal 
skills: and, through these, enhancing the students' economic and social worth. 

The audit involved three major activities: 

• A series of structured interviews, concerning current teaching and assessment 
methods, with subject area staff to examine ways in which opportunities were 
given for students to exercise responsibility for their own learning and develop 
core transferable personal skills. 

• A series of structured interviews on the placement experiences of a sample of 
students (and their supervisors) to attempt to identify the characteristics of an 
"enterprising" placement. 

• The completion of a questionnaire by course organisers of degree courses, with 
follow-up interviews, on the teaching and assessment of core transferable personal 
skills. (These core transferable personal skills were identified as basic IT 
keyboarding skills; study/learning skills; and communication skills.) 

From the conduct of this audit six main issues were identified in its evaluation: 

• The consequence of the use of the term "audit" and ownership of the audit. 

• Time and timing of the audit. 

• The role of the audit in raising EHE awareness. 

• The choice and role of the audit team. 



GO 



DESIGN AND CONDUCT OF ENTERPRISE AUDITS 



83 



• Qualitative v quantitative approach. 

• Course driven v subject driven approach. 

USE OF THE TERM "AUDIT" AND OWNERSHIP OF THE 
AUDIT 

The approach adopted was very much a top down approach directed by management as 
opposed 10 a bottom up approach "owned" and directed by the staff involved. Essentially 
management wished to identify where we were, in terms of the penetration of enterprise 
development in the curriculum, and to identify examples of good practice. The audit 
team planned and implemented an audit strategy with no prior consultation with course 
teams or subject areas. 

At the outset we found considerable concern in the minds of subject area colleagues 
about the audit and a certain element of suspicion. This resulted in an over cautious 
approach being adopted by some subject areas. For example the view was expressed on 
a number of occasions, that subject areas would have to be careful how much they 
disclosed lest they be considered as "enterprising enough" and lose out on any future 
funding. This caution regarding disclosure took a little time to overcome and, to assist, 
we wished to be as flexible as possible in terms of how we interfaced with subject area 
representatives in order to gain maximum co-operation. 

Consequently, whilst the agenda for the structured interview remained the same, the 
actual procedures adopted, from one department to another, differed quite widely. This 
resulted in some departments resolving to appoint coordinators with whom we interfaced, 
and others writing their own departmental reports in order to assist the audit team. 
Whilst this was welcomed and did have many advantages - it passed ownership of the 
audit to departments - it did create some problems for the audit team in terms of 
controlling the audit. In particular, it became impossible to keep this part of the audit 
running to its pre planned completion time and there was an over- run on this aspect of 
the study. 

Wc believe such a top down approach can be criticised. It does, for example, perceive 
the audit as an event rather than as an ongoing reflective review process; that is one 
which could allow, in the right organisational culture, a commitment to action research 
through team building and the development of a shared vision by staff about where the 
Polytechnic wishes to move. Pennington (1991) encapsulates the problem well: "Audits 
tend to be done by experts to others: by contrast review is something everyone is capable 
of undertaking". Essentially it has to do with working with people, not working on 
people. 

Those holding this viewpoint would argue that if an institution seeks meaningful change 
in its practices (in this case enterprise development in the curriculum), then such a bottom 
up approach is desirable allowing course teams a feeling of confidence and competence 
to undertake an audit themselves and letting them determine, on the basis of a set of 
shared understandings, how best to manage this audit. 

Given our experiences of undertaking the audit, particularly our experiences of working 
with those departments who by volunteering to write reports for us took "ownership" of 
the audit, we are sympathetic to this view. Letting subject teams determine, on the basis 
of their shared set of understandings, how best to manage the audit seems the way to 
create a climate for meaningful change. There are, of course, very clear costs associated 
with such a process. It is developmental, can be lengthy and even indeterminate in terms 
of its outcomes and therefore can be difficult to control in the project management sense. 
But the arguments for this approach lie in the very fact that it is developmental and, as 



91 



84 



IMPLEMENTING QUALITY IN ORGANISATIONS 



Pennington states, "its momentum is sustained by the improvements staff themselves 
identify as necessary". 

We have been able to determine where we are in terms of enterprise development in the 
curricula. Whether the audit, conducted the way it was, achieved that momentum towards 
team building, meaningful change and a shared vision about where the Polytechnic 
wished to move is an open question. 

TIME AND TIMING OF THE AUDIT 

The audit was plaimed to commence in late January. In three respects the audit seriously 
over- ran its initial planned time. The first was agreeing the methods to be employed. 
It took longer than planned for the management steering group to agree its preferred 
strategy. The second was the study via subject area groups due to the methods adopted. 
Hie third was in the time required for the data analysis, given the loosely structured 
interviews approach adopted during our study with subject groups. 

Bell (1991) reports that in the Hatfield Polytechnic audit one full session was allowed. 
Our original timescales were far too optimistic for the qualitative route we took and the 
commencement of the audit, in late January and straddling the summer leave period, was 
ill-timed. 

We believe audits should commence at the beginning, or close to the beginning, of an 
academic session and thai a realistic period of time should be allowed for consultation 
in the planning of the audit by any steering group established for that purpose. The 
comments by Pennington, on the need for the rate of progress to be "acceptable to those 
who are responsible for delivery", seem particularly apt. 

ROLE OF AUDIT IN RAISING AWARENESS 

There was a general lack of awareness within the Polytechnic of the Government's EHE 
Initiative. This gave the audit team problems. Bell indicated similar problems and 
suggested that, from his experiences, earlier briefings - both of papers and presentations 
- would have assisted. This Glasgow audit fell into the same trap. Whilst an attempt at 
awareness raising was made, through letters from the Vice-Principal and presentations to 
heads of department, from our experiences we would suggest that this is inadequate. 
Ewins (1991) indicates that a successful procedure adopted at the Polytechnic of Wales 
was the holding of one day workshops involving all academic staff in the institution, and 
a scries of workshops to key communicators. We believe this should also have been our 
approach at the outset. Nevertheless, the audit has played a major part in raising 
awareness and creating the right kind of climate for the main project. Indeed that may 
be the audit's single most important outcome. 

CHOICE AND ROLE OF AUDIT TEAM 

The decision to appoint a full time audit assistant from outwith the body of the academic 
staff, and without any previous experience of teaching in higher education or in the 
process of the management of degree level courses, had its advantages and its 
disadvantages. Its advantages lay in the audit assistant being a new face • every member 
of staff she met resulted in a new professional relationship and was not influenced by the 
perceptions of previous encounters between the parties. She was not a permanent 
member of staff from any one constituency within the Polytechnic and could, therefore, 
be seen as independent. Bell reports that Hatfield appointed a Senior Fellow who had, 



DESIGN AND CONDUCT OF ENTERPRISE AUDITS 



85 



from his previous post, extensive knowledge of the Polytechnic and considerable 
credibility with staff. On occasions the audit assistant did comment upon her status - 
both in terms of her lack of "well knownness" within the Polytechnic and the grading of 
her post - as influencing the priority academic staff perhaps assigned to dealing with her 
requests for interviews and information returns. 

For the audit to be worthwhile it requires to be perceived as an ongoing review process 
and not as an "event". Whether such a review process needs an external audit team - ie. 
external to the review groups - depends very much on the perceived role of that team. 
We would suggest two possible roles exist: the first a facilitating role, the second a 
reporting role. As facilitators their main task would be to raise awareness via the 
mechanisms of workshops and the like, and in performing a co-ordinating and 
communicating role between review groups. In their reporting role their main task would 
be to draw together information on the activities and outcomes of review groups and to 
consolidate these into meaningful statements which both the community of review groups, 
and possibly a wider community beyond that, could consider. 

As for personnel, there are clear advantages in having senior personnel well versed in 
the processes and procedures of the institution to undertake such roles. What is critical, 
however, is the independence and credibility which these personnel are able to bring with 
them to these roles. 

QUANTITATIVE v QUALITATIVE APPROACH 

Bell, has commented on the problems of having a vague definition of enterprise and the 
consequences this poses in questionnaire design. Earl (1991) also drew attention to the 
limitations of questionnaires in terms of their ability to elicit qualitative information. In 
this audit the management steering group was concerned about the well known 
psychological reaction to questionnaires and that staff should not be discouraged from 
participating by any complex questionnaire. However Ewins points up some of the real 
problems in undertaking a less structured survey approach. He found in his audit thai: 
replies did not always address the questions asked, the lack of response did not always 
indicate that a feature did not exist in the course, responses were not always consistent. 
This experience has been shared by the Glasgow audit team. We see a problem here. 
On the one hand we have supported the bottom up, action learning approach espoused 
by Pennington, yet on the other we have experienced the same problems as Ewins in 
using structured (or loosely structured) survey methods. Pennington acknowledges that 
the process he advocates "may appear to be lengthy and overly complex" and certainly 
the experience of this audit team and Earl would indicate that structured interview 
methods do require very considerably more time than would at first be imagined. By 
contrast, we found the use of a questionnaire with course organisers later in the audit very 
efficient and had little difficulty in working with them towards its completion. 

We think we were right not to use questionnaires in the initial stage. However, we did 
underestimate the problems of control and, subsequently, data analysis when we 
encouraged departments, or groups of staff within them, to write their own reports. On 
balance we accept the value of this (ic. transfer of "ownership") and, whilst we conclude 
that it does impact very significantly on the time required to complete such an activity, 
it can be justified. On the other hand we think it a mistake to assume that the use of 
questionnaires will somehow, automatically, "switch off" staff. This was not our 
experience with course organisers. Perhaps preparation for, and timing of any use of, 
questionnaires is thti key to this issue. We believe, therefore, that there is a place for 
both quantitative and qualitative approaches in such audits. 



93 



86 



IMPLEMENTING QUALITY IN ORGANISATIONS 



COURSE DRIVEN v SUBJECT DRIVEN APPROACH 

This audit differed from the others mentioned in this report in that it approached 
information gathering via subject area boards. We took this approach for two reasons. 
First, efficiency - data on the teaching of a subject could be collected once from the 
teaching group, rather than repetitiously through sundry course committees. Second, it 
was felt that course organisers were unlikely to have the depth of detailed knowledge 
about the teaching methods on their course to be able to answer the audit team's 
questions without extensive referrals to subject specialists. Our experiences tend to 
confirm this latter concern. 

Conducting our study via subject areas created a level of "corporate consciousness" 
amongst subject areas (and departments) about their teaching and assessment methods and 
encouraged at least some subject groups into positive action. One group, for example, 
managed to have their departmental report published in a professional journal! On the 
other hand it left course boards largely untouched by the audit and so it may be that their 
"corporate consciousness" has suffered as a consequence. This could be to the 
disadvantage of the main EHE project at Gla; gow which is planned to work, essentially, 
through course boards. On balance we conclude that, given the audit was a pre-cursor 
to the main project and that we tend towards the philosophy espoused by Pennington (that 
audit is a process of review and not an event), we would have been better conducting the 
audit via course boards. It might have better set course boards "rolling" on the kind of 
process review adopted by Pennington at Teesside. However, whilst something would 
have been gained, something might also have been lost. 

CONCLUSIONS 

To .summarise from the experiences of this study and others, the following conclusions 
are drawn: 

a The term "audit" is inappropriate: "review" is a more appropriate term to describe 
such processes. 

b Reviews should be conducted "bottom up" (ie. ownership should reside with those 
directly involved) and not "top down". 

c They are useful mechanisms for raising enterprise initiative awareness, but this 
must be positively planned as part of the audit. 

d The role of the audit team should be to facilitate, co-ordinate and communicate 
between course review groups rather than an investigative role. 

e Qualitative (ie structured interview) methods are extremely expensive on human 
resources and make for difficult analysis but allow ownership of the audit to pass 
more easily to course review groups than quantitative methods. 

f Audits are better conducted via course groups (ie course driven) than subject 
groups if meaningful change is to be effected. 



ERLC 



34 



DESIGN AND CONDUCT OF ENTERPRISE AUDITS 87 

References 

Bell D (1991) Conducting an Enterprise Audit at Hatfield Polytechnic 

Earl S (1991) Conducting and Enterprise Audit at the Robert Gordon's Institute of 
Technology 

Ewins G (1991) Conducting an Enterprise Audit at the Polytechnic of Wales 

Pennington G (1991) Conducting an Enterprise Audit at Teesside Polytechnic 

Papers in the proceedings of the EHE Audit Methodologies Workshop April 91 (Copies 
of the proceedings available from Educational Development Service at Glasgow 
Caledonian University) 



mm 



95 



15 Monitoring the quality of quality 
control systems 



Henry I Ellington ard Gavin T N Ross, Robert Gordon University, Aberdeen, 
Scotland 

INTRODUCTION 

"Sed quis custodiet ipsos/Custodes?" - Juvenal 

Nearly 2000 years ago, the Roman satirist Juvenal made an extremely profound 
observation. He pointed out that it was all very well posting guards to look after 
something, but that the efficacy of such a measure would always be in doubt unless 
further measures were taken to ensure that the guards did their job properly. In other 
words, Juvenal asked. "Who will guard the guard? themselves?" 

Somewhat more recently, the Vice Principal of Aberdeen's Robert Gordon Institute of 
Technology (RGIT) asked himself a similar question about the quality system that had 
been set up within RGIT in order to meet the accreditation requirements of the Council 
for National Academic Awards (CNAA). In 1989, RGIT was accredited by CNAA to 
validate and review its taught degree courses and was scheduled to be considered for re- 
accreditation in 1996. The Vice Principal was only too well aware that any College 
seeking such re-accreditation would be required to demonstrate that its quality system had 
fully satisfied the standards set by the Institution itself, and that the responsibilities and 
obligations under CNAA accreditation had been properly met. He therefore proposed to 
RGIT s Academic council that a "meta-quality-control" system be set up within RGIT to 
help ensure that this did in fact happen, and to provide the necessary evidence. We 
describe here how this was done. 

RGITS QUALITY ASSURANCE AND QUALITY CONTROL 
SYSTEMS 

RGIT is one of Scotland's Central Institutions, having over 4,300 full-time and 1,300 
part-time students enrolled on over 80 different degree and other courses ranging from 
Fine Art to Mechanical Engineering. 

Academically, it is divided into four faculties (Design, Health & Food, Management, 
and Science and Technology), each headed by an assistant principal who acts as 
Executive Dean. Overall executive responsibility for the academic work of the institute 
and the quality thereof rests with the vice principal, to whom the four assistant principals 
report in respect of their faculty-related activities. 

In January, 1990 RGITs Academic Council produced a quality handbook defining the 
procedures that were to be adopted for the quality assurance and quality control of all 
Institute courses - a handbook that has since been considered with interest by a number 
of other HE Institutions. 

Quality-assurance procedures within RGIT are based on a rolling programme of formal 
validation and review events, with all taught courses being subject to rigorous validation 
when they are first set up. All of RGITs Schools (academic departments within 
faculties) and other academic units arc subject to periodic reviews and. within this school- 
based system, the individual courses arc also reviewed. RGIT is wholly committed to 
the principle of external peer review, as developed over the last quarter-century and more 



96 



MONITORING QUALITY CONTROL SYSTEMS 



89 



by the CNAA. For this reason, the panels responsible for conducting all such validation 
and review events include a majority of external members - usually two academics and 
tuo practitioners. RGIT is almost unique in that the panels are externally chaired, a 
policy that the Institute believes adds greatly to the authority and overall credibility of its 
quality- assurance system. The panel also includes two internal members, normally the 
dean of the faculty concerned and the chairman of the faculty advisory committee which 
handles the preliminary internal stages of the quality- assurance procedures. 

It is also worth noting at this juncture that RGIT deliberately did not set up an 
Academic Standards Committee or its equivalent, which again makes it unusual among 
CNAA-accredited institutions. The Institute's thinking in not so doing was that it wished 
the concern and the responsibility for quality to be built into every facet of the normal 
systems of academic management. In other words, quality procedures would be based 
and owned at the appropriate operational level. This did, of course, mean that there were 
no "quality police" in RGIT. Rather, each member of the academic community was 
charged with a self -responsibility to be actively involved in and personally aware of, the 
issue of quality at all levels of academic activity, 

RGITs annual quality-control procedures, on the other hand, are the responsibility of 
its internal hierarchy of committees. These range from course panels and school 
committees at operational level, through the faculty boards, which then report accordingly 
to Academic Council. The Council has delegated responsibility from the Governing Body 
for the overall planning, co-ordination, development and supervision of the academic 
work of the Institute. The Institute's quality-control procedures involve carrying out 
annual internal critical appraisals of the operation and success of all RGIT courses in 
meeting their stated aims and objectives. 



HOW RGITS INTERNAL QUALITY AUDIT TEAM WAS 
FORMED 



In the Spring of 1990, RGITs Vice Principal submitted a discussion paper to academic 
council setting out the rationale for the establishment of an internal quality audit system 
in RGIT. Such a systeir n'ould monitor the various quality- assurance and quality-control 
systems described in the last section, thus effectively "guarding the guards". Academic 
council subsequently agreed with the Vice Principal's proposal that responsibility for co- 
ordinating the new "meta-quality-control" system should rest with the Institute's 
educational development unit, since this had an institute-wide brief for promoting and 
supporting academic development work of all types. It was also neutral in course-related 
matters, as it did not itself offer courses leading to a CNAA award. 

In subsequent discussions between the Vice Principal and the Head of the educational 
development unit, it was agreed that the latter would set up and lead an institute- wide 
internal quality audit team, which would report to the Planning and Review committee 
through the Vice Principal. The team would include a representative from each of 
RGITs four faculties. It would also include a representative of the Registry, which is 
responsible for the management and co-ordination of RGITs quality programme. Its 
terms of reference would be "to establish appropriate mechanisms for the appraisal of the 
Institute's quality-assurance/quality-control procedures and their effectiveness 1 '. 

It was also originally envisaged that the team would plan and implement formal 
internal-appraisal events dealing with the Institute's quality procedures in 1991, 1993 and 
1995, in preparation for the full CNAA re-accreditation event that was (at the time) 
scheduled for 1996. Whilst subsequent external developments (notably the Government's 
decision to disband CNAA in the summer of 1992 and grant full degree-awarding powers, 





90 



IMPLEMENTING QUALITY IN ORGANISATIONS 



together with university status to institutions such as RGIT) have removed the original 
need for this programme, it is intended that RGIT will still organise internal appraisal 
events. The team has, however, concentrated its efforts to date on carrying out audits of 
specific aspects of the Institute's quality-assurance and quality-control procedures. 

THE FIRST QUALITY AUDIT CARRIED OUT BY THE TEAM 

RGITs internal quality audit team was formally established at the start of the 1990-91 
session, and immediately embarked upon the first task that the planning and review 
committee had asked it to address - carrying out a critical examination of the Institute's 
annual course appraisal system. Quality control at RGIT involves each course panel 
carrying out a detailed critical appraisal of the previous year's operations and submitting 
a report to the appropriate faculty board using a standard RGIT proforma. The faculty 
boards then report to Academic Council on the outcome of this exercise with respect to 
their own portfolios of courses. 

The quality audit that was carried out by the team took place between October, 1990 
and March, 1991, It involved critically examining all annual course appraisals that had 
been completed in RGIT since the Institute obtained CNAA-accredited status in 1988 ie. 
the annual appraisals for 1988-89 and 1989-90. The bulk of the work was carried out by 
the faculty representatives on the team, who were asked to seek answers to the following 
two key questions: 

Ql "Is self-criticism as an ethos operating widely and deeply throughout RGIT?" 
Q2 "Are issues identified by Course Panels subsequently being addressed?" 

The evaluation was carried out using two specially-designed proformas, the first of which 
was completed in every respect for every RGIT course that was subject to annual 
appraisal (55 at the time). The second was used to provide a summary of the findings 
in respect of courses operated by each faculty. Since the faculty of Science and 
Technology had operated almost half the Institute's courses during the period under 
review, the work on its courses was split between two people. 

Once the faculty members of the internal quality audit team had finished their work, 
they submitted their completed proformas to the team chair together with a short report 
on their overall findings. The resulting material was then circulated to the team and an 
overall report on the team's findings, conclusions and recommendations was produced. 
This was submitted to the planning and review committee in March, 1991. 

With regard to Question 1, the team came to the conclusion that an overall ethos of 
self-criticism could not be considered to be as fully developed throughout RGIT as was 
desirable. Substantial differences between individual faculties, schools and even courses 
were also identified. In broad terms, the probiem seemed to be that critical appraisal 
tended lo be reactive rather than proactive, since the annual course appraisal procedure 
then being operated had clearly not been successful in encouraging proactive self-criticism 
on the part of course panels. It was felt that there were two possible reasons for this 
unsatisfactory state of affairs. Firstly, the guidance given within the standard proforma 
had perhaps not been explicit enough. Secondly, a more intensive staff development 
exercise designed to promote a more sophisticated understanding of the underlying 
philosophy of "a self-critical academic community" would probably have been a good 
idea. As a direct result of the criticisms and recommendations made by the team, RGITs 
planning and review committee decided to carry out a radical review of the procedure. 



9§ 



MONITORING QUALITY CONTROL SYSTEMS 



91 



With regard to Question 2, the team concluded that the issues that we^ identified by 
course panels were in most cases being appropriately dealt with - faculty boards playing 
a key role in this regard. The main exceptions appeared to be issues connected with 
resourcing, which tended to be noted by the faculty boards but subsequently not always 
fulfilled - largely due to the general reduction in unit costs that is a characteristic of 
higher education today. 

The team's more detailed findings in respect of each faculty's portfolio of courses and 
the individual courses therein were passed back to each dean of faculty so that 
appropriate action could be taken where necessary. 



SECOND QUALITY AUDIT CARRIED OUT BY THE TEAM 



As soon as it had completed the report on its first audit , the internal quality audit team 
started work on the second task that the planning and review committee had asked it to 
address - carrying out a critical appraisal of all validation and progress review events that 
had been conducted in RGIT since the institute received CNAA accreditation. 

Work on this task was carried out between April 1991 and February 1992, using 
essentially the same approach that had been employed in the first audit. It involved the 
various faculty representatives conducting detailed critical examinations of the thirty five 
validations and progress reviews that had taken place in RGIT over the period in 
question. In order to standardise the procedure, two specially-designed proformas were 
again employed. The first proforma was completed in respect of each separate validation 
or progress review, and involved determining whether the detailed procedures set out in 
RGIT's quality handbook had been properly adhered to. The second proforma was used 
to provide a summary of findings in respect of the events held within each faculty. To 
help standardise the process, faculty scrutineers were provided with detailed guidelines 
on how to complete the two proformas. 

Once the faculty members of the team had finished their examinations, they again 
submitted their completed proformas to the team chair together with a short report on 
their overall findings. The resulting material was again circulated to the team for 
discussion and an overall report on the team's findings, conclusions and recommendations 
was again produced. This was submitted to the planning and review committee in 
February, 1992. 

On the whole, the team's findings were much more positive and satisfactory than had 
been the case in respect of the earlier audit. It concluded that virtually all the validations 
and progress reviews that had been carried out had been well organised and conducted. 
Where problems had arisen, mainly in respect of four events in the faculty of Health and 
Food, these were almost invariably due to the fact that the course panels concerned had 
no; properly followed the Institute £ Adelines. 

The team did, however, make a number of specific recommendations as to how RGIT\s 
quality-assurance procedures could be simplified and improved. Firstly, it recommended 
that the amount of documentation required in respect of both validation events and 
progress review events should be drastically reduced, since the faculty scrutineers had 
been unanimous in concluding that the present amount was both excessive and 
unnecessary. This has long been a familiar refrain in CNAA member institutions and it 
is not without a certain irony that , within the freedom of an accredited institution, this 
should still be a problem. From an ins 'tutional standpoint - as from CNAA in the past - 
it coHd be argued that the excessive dc umentatton is not in fact asked for, but rather 
that it indicates a lack of synthesis and selection of material on the part of the Course 
Panels. In other words, the blame lies mainly with the validated rather than with the 




99 



92 



IMPLEMENTING QUALITY IN ORGANISATIONS 



validators. Secondly, the team recommended that new course validation events should 
be kept separate from school review events. If incorporated within such reviews for 
logistical reasons, they should constitute clearly-id .itifia, *s f discrete events with separate 
reports. Thirdly, it recommended that the role O' tne faculty advisory committees (which 
scrutinise the documentation before each validation or review event) should be 
strengthened, thus enabling any major problems to be identified and addressed at an early 
stage. Fourthly, it recommended that a comprehensive staff development programme 
should be implemented in respect of the Institute's quality- assurance and quality-control 
procedures, so that everyone involved will be better prepared to carry out their respective 
roles effectively. These various recommendations are all being addressed at the time of 
writing and it is expected that they will all be implemented eventually. 

THE PROPOSED FUTURE PROGRAMME FOR THE TEAM 

Now that the internal quality audit team has successfully completed its first two audits, 
it is intended to thank the original faculty representatives for their efforts and allow them 
to demit office. Replacing them with new representatives will not only spread the work 
load, but will also enable new people to benefit from the not inconsiderable staff 
development opportunity that serving on the team provides. The EDU and Registry 
representatives on the team will be unchanged, thus providing executive continuity. 
RGITs planning and review committee has also identified the next two key issues that 
it wishes the team to address. The first is the effectiveness of the process by which 
matters relating to quality assurance and quality control are recorded in RGHT 
documentation minutes of meetings. The second is the extent to which RGIT is 
succeeding in meeting the CNAA criteria for accreditation, which are likely to be adopted 
by RGIT even after the demise of CNAA. 

CONCLUSION 

On the basis of the work that has been carried out so far, the two authors of this paper 
are in no doubt that the establishment of RGITs new internal quality audit system has 
been a great success. The system has proved to be both practicable and effective, with 
the internal quality audit team being able to complete both lis initial tasks more or less 
to schedule (both took slightly longer than planned). The first two quality audits have 
produced a large amount of extremely useful information about the effectiveness and 
efficiency of RGITs quality-control and quality- assurance procedures. But most 
important - the audits have identified specific ways in which both sets of procedures can 
be made even more effective and efficient. The au f, iors therefore commend the RGIT 
system to other colleges that are thinking about introducing "meta-quality-control" 
procedures, and will be happy to provide more detailed information on request. 



100 



Section 3: Quality :~i Courses 



The conference enabled participants to explore the applicability of different perspectives 
on quality to educational contexts and practices. Many of the contributions were in the 
fonn of case study reports, providing detailed and vivid data and making it possible to 
explore issues of quality arising in specific initiatives and settings. Participants valued 
the opportunity to hear what o f I,crs were doing, and in discussion to clarity for 
themselves some of the issues to be addressed. 

I Musallam, M Brophy and M Schilling in "Improving the quality of a national 
curriculum: the role of textbooks" report the preliminary results of the evaluation of a 
national curriculum reform programme which aims to raise the quality of basic and 
secondary education in Jordan, The curriculum reform strategy relies heavily on the 
introduction of new textbooks, but even with in-service training there appears to be a mis- 
match between the use of the books and the aims of the reform. The reformers, 
curriculum developers and teachers do not have a shared understanding of the meaning 
of some of the basic reform concepts, and use different criteria in judging the quality of 
a textbook. The pitfalls of "top-down" reform are illustrated. The medium does not 
convey the message! 

In "Developing quality in education: a way forward" R J D Rutherford demonstrates 
how the principles of total quality management can be applied directly and successfully 
in education. He identifies core elements and key questions in the systematic and 
continuous process of developing quality, and illustrates how these have shaped the 
design of a course for teachers which prepares them to engage in quality development in 
their own schools. "Practising what he preaches", the author describes some numerical 
performance indicators for the course. 

"Quality in course design" is explored by Eric Roper in the context of a professional 
development scheme for educators working in all sectors of the education system. The 
scheme is examined in relation to three perspectives on quality: fitness for purpose, how 
it sets out to meet participants' needs, and how it attempts to empower participants to 
take charge of their own professional development. Scheme processes and structures are 
described, emerging issues in relation to quality in course design are discussed, and ways 
forward in assuring quality are identified. 

Louise Gibbs in "Quality horses for quality courses" reminds us that an important 
element of quality is the matching of students to courses. She a^-csses the question of 
what a "quality" student is and suggests that "if there can be unsuitable students for 
courses, can there not also be unsuitable courses for students?" She identifies lessons to 
be learned by the new generation, market-led higher education from adult continuing 
education, explores some of the reasons why students have dropped out of music courses 
and reports on a training programme for course advisers. She also raises the notion of 
the place of intuition in advising prospective students. 

In a provocative paper, "Recognising quality in engineering education", D C Hughes and 
R G S Matthew consider both the purpose of education and course design features in the 
context of engineering education at degree level. They outline the components of the 
"engineering method" and ask how students would recognise that they were on a quality 
course - one which allowed them to adopt a deep approach to learning and to develop 
skills in using the engineering method. 




16 Improving the quality of a national 
curriculum: the role of textbooks 



I Musallam, M Brophy and M Schilling, The Ministry of Education, Hie British 
Council, Jordan, and Die University of Liverpool, 169 3BX, UK 

INTRODUCTION: THE CURRICULUM REFORM PROJECT 
IN JORDAN 

Human resource development has been a long-standing priority for the Government in 
Jordan and investment in the education and training system has served the country well 
in the past. In the seventies and eighties, for example, large numbers of educated 
Jordanians were employed in the Gulf region and their remittances were a major 
economic factor in the national economy. Today enrolments at the basic, secondary and 
higher education levels are amongst the highest in the world and it has been estimated 
that at present every third Jordanian citizen is involved in some form of education or 
training (General Directorate of Planning 1989). 

The nineteen fifties, sixties and even seventies were periods of rapid expansion in 
education. This expansion was often at the expense of quality, so that by the mid-eighties 
there was a growing awareness of the need for reform of the education system itself. 
Research into the quality of education, especially science education, revealed a picture 
of overcrowded, under-resourced classrooms and ineffective teaching methods ( Abu 
Sardaneh 1983; Ismail 1981; Abdalla 1990). 

In his speech at the opening of the National Assembly in 1985, His Majesty King 
Hussein called for the launching of a national programme for education reform and in 
September 1987 under his patronage a National Conference of Educational Reform 
(NCER) was inaugurated with the express purpose of developing a plan of action. 
Following from this conference a ten-year national reform programme was developed 
and is currently being implemented with assistance from a number of organizations 
including The World Bank, The British Overseas Development Administration and The 
British Council. 

The primary purpose of the reform is to raise the quality of basic and secondary 
education and so improve the nation's manpower base so that it will be better able to 
meet both the domestic needs and those of the region's scientific and technological labour 
markets. To achieve this the Government established National Subject Panels to revise 
the curricula. 

These teams were charged with developing curricula whose content and approach would 
emphasize: 

1 fostering creative thinking by the use of materials and approaches which 
promote critical thinking, problem-solving, analytical and information 
processing skills 

2 enhancing student achievement by building flexibility in curriculum especially to 
allow for the ability range of pupils 

3 modernizing curriculum content and making it relevant to current conditions and 
issues such as environmental and health issues 

4 re-orienting classroom teaching methods to emphasize activity-based 
experiences and applied learning 

5 providing teacher guidance to encourage greater creativity and flexibility in 
implementing the curriculum 



102 



96 



QUALITY IN COURSES 



6 enhancing the occupational and vocational orientation of students 
• 7 designing continuity and integration into the curriculum of different grades and 
subjects. 

THE ROLE OF TEXTBOOKS IN THE REFORM 

Two basic processes are being used to implement the curriculum reform, namely the 
development of new textbooks and A/V materials and the in-service training of teachers. 
Text books are seen as having a pivotal role in the reform process. In fact the project 
has been categorized as a textbook-led reform and a considerable part of the in-service 
training is devoted to training teachers on the use of the new books. 



Year Grades Levels to be Implemented 



1991 
1992 
1993 
1994 



9 
10 
11 
12 



Table 16.1 The schedule for implementing the reform 

Although the basic curricula for all five subjects were prepared some of the texts were 
not completed on schedule and so only twelve of the fifteen books proposed for 
introduction in September 1991 have been implemented. Nevertheless the project has 
started well. Almost 1.5 million new books have been distributed and are in use in 
schools. 

The plan of action for the reform called for the new texts to be evaluated during their 
first year of use. so that they could be revised during the second year and "final" editions 
introduced in the third year. 

To organize and carry out the evaluation the Ministry of Education established a 
Evaluation Division in September 1991 based in the General Directorate of Curriculum 
and Educational Technology (GDCET). 

DEVELOPING THE EVALUATION STRATEGY 

The first questions that the staff of the new division had to address were. "What was the 
purpose of the evaluation ? What was it that was to be evaluated and why?" It was 
difficult to get a consensus on this, partly because of the organizational structure within 
the Ministry of Education. Development of the curriculum and the text books is the 
responsibility of the GDCET while in-service training is the responsibility of the Training 
Centre. The Evaluation Division is based in and staffed by members of GDCET and one 
view was that it should only e* aluate the work of it's own general directorate, i.e. the 
curricula and the text books. A second view was that textbooks could not be evaluated 
in isolation from how they were used in the classroom. And how the books were used 
must inevitably depend on how teachers were trained to use them. 
A compromise was reached in that the strategy eventually accepted was designed to 



103 



ROLE OF TEXTBOOKS 



97 



evaluate three things. 

1 The books themselves 

What could be called their "technical" aspects (accuracy, printing quality, 
illustrations). 

2 The suitability of the books 

Eg. how well they matched the level of the pupils, the needs of the local 
community and the conditions in schools. 

3 The way the books were used 

Eg. were they used didactically with the teachers reading and explaining the 
concepts or were they used by the pupils in group work or for following 
activities. 

THE STRUCTURE AGREED 



Subject 
Grade 


Maths 


Arabic 


Science 


Islamic 
Studies 


Social 
Studies 


1 


+ 


+ 


+ 


+ 


+ 


5 


+ 


+ 


+ 


+ 


+ 


9 


+ 


+ 


0 


0 


0 



Table 16,2 New texts introduced into Jordanian schools in September J 99 J 

Twelve new texts were introduced into the schools in September 1991 and the Evaluation 
Division is responsible for evaluating them during 1991-92. Just as there had to be a 
compromise on the purposes of the evaluation so too was there a compromise on the 
structure used. Many different personalities and groups were involved in the discussions, 
and the specialists from the different subject areas and the "customers" who were to 
receive and act on the evaluation feedback had varying opinions on what information 
would be most useful to them. In the end after a three month cycle of proposal, 
discussion and revision a two stranded approach was agreed on and approved by the 
Minister and Committee of Education. The two strands are the continuing and the final 
phase. 

The Continuing Phase 

For each of the books being evaluated twenty-five teachers have been chosen from five 
representative regions of the country. The teachers are asked to keep daily teaching logs 
for one of the twelve books, using a prepared reporting format. 

A supervisor from each region visits one teacher each week to interview them about 
their experiences as recorded in the daily logs and to observe a lesson. Structured 
interview forms and detailed observational schedules are provided. The supervisors 
received a three-day training session on interview and observational techniques. Video 
films were used to establish standardized and agreed procedures. 

From the continuing phase it was intended that we should get information for each 



10 4 



98 



QUALITY IN COURSES 



book on the teaching experiences of twenty-five teachers who had taught it throughout 
the year. If each teacher was interviewed and observed once every five weeks then in a 
thirty-week teaching year we should have information from 150 (25 x 6) interviews and 
150 lesson observations. 

The Final Phase 

To supplement and provide some cross referencing for the on-going strand it was agreed 
that during the last month of the academic year questionnaires would be sent for each 
book to teachers in 5% of the schools in the 23 regional educational directorates. The 
questionnaires were designed to provided additional evidence about the suitability and use 
of the books rather than to elicit responses about technical problems. 

A parents* questionnaire has also been designed and will be distributed to the parents 
of 150 pupils over all regions following each book. Both the teachers* and the parents* 
questionnaires were piloted during February and are scheduled for distribution in May. 

THE RESULTS 

A number of the books are being published in two parts, one for each semester, so it was 
possible to carry out a preliminary analysis of some of the texts based on the results from 
the continuing phase collected over the first semester. Tne books have been generally 
well received by the teachers and are considered to be a major improvement on the old 
text. Although from the results obtained so far it is clear that the teachers and 
supervisors are generally more inclined to comment on the "technical" aspects of the 
books, the misprints and mistakes, rather than on their suitability and use. A variety of 
"errors" have been identified for each book including printing, cultural and contextual 
errors. 

The types of errors identified varied from book to book but followed a logical pattern 
in that in the lower level books there were relatively more difficulties with language and 
illustrations and fewer with exercise questions. Specific language difficulties were often 
to do with vocabulary. 





Language 


Illustrations 


Exercises 


Total 


Grade 1 Arabic 


101 (45) 


93 (41) 


33 (15) 


227 


Grade 5 Science 


58 (28) 


64 (31) 


83 (40) 


205 


Grade 9 Maths 


15 (24) 


16 (26) 


31 (50) 


62 



(percentages in brackets) 



Table 16.3 Comparison of types of problem reported for three books 

One of the main concerns was the level of dissatisfaction expressed with the illustrations 
in the books for the lower grades especially over the lack of clarity and the numbers of 
contextual and cultural errors. Exercise questions in the books were generally seen as 
being too difficult. Although in light of this there were surprisingly few reports about the 
books failing to cater for the range of pupils' abilities and their individual differences. 



105 



ROLE OF TEXTBOOKS 



99 



THE USE OF THE BOOKS 

One disappointment for those involved in the reform has come from the reports and 
observations about the way the books are used. A major aim of the reform is to 
encourage more pupil-centred and activity-based learning and teaching. It had been 
expected that with the introduction of the new texts and in-service training there would 
be a move away from didactic methods towards greater pupil involvement and that this 
would be reflected in the way the books were used. Unfortunately this does not appear 
to have happened so far. The reports on the science book, for example, show that the 
books were still predominantly being used by the teachers. The main pupil usages were 
for doing exercise questions and home work. There wasn't a single instance of that book 
being used for group work in science. 

IMPLICATIONS FOR THE EVALUATION 
Problems of Reliability 

One problem which the results have highlighted is the difficulty experienced in achieving 
a shared understanding of the meanings of some of the key concepts of the reform, 
concepts such as "problem-solving" and "catering for individual differences". Although 
the supervisors received specific training on using the observation schedule there was still 
a wide variation in the way that they reported seeing these concepts being implemented 
in the classroom. For example one supervisor reported that there was no evidence of 
problem -solving activities in any of the twelve lessons he observed while another reported 
that he saw it in nine of the eleven lessons observed. 



Region 


1 


2 


3 


4 


5 


Total % 


"Yes" 


9 


3 


6 


6 


0 


24 45 


"No" 


2 


5 


9 


1 


12 


29 55 


Tabic 16.4 Reports of problem-solving activities observed in lessons 


Region 


I 


2 


3 


4 


5 


Total % 


"Yes" 


12 


9 


8 


7 


12 


48 98 


"No" 


0 


0 


1 


0 


0 


1 2 



Table 16.5 Reports of observations of teachers catering for individual differences 



100 



QUALITY IN COURSES 



At present we are unable to say if this was a real difference (ie. in the teaching) or 
simply a difference in the supervisors' interpretations. There was more agreement 
amongst the supervisors about teachers catering for the individual differences of the 
pupils. K^re the supervisors noted that the teachers did this in all but one of the 49 
lessons observed. But the explanations of what was accepted as "catering for individual 
differences" varied greatly. Since these results a second round of training workshops has 
been held and modifications made to the instruments to try to achieve a greater degree 
of "mutual" understanding. 

The Difficulty of Defining "Quality" for Textbooks 

When starting the evaluation one of the questions asked was "What are the qualities of 
a good textbook? What characteristics should it have?" A recent study of a series of 
textbooks in Texas found that they had an average of over 700 mistakes per book yet 
the publishers claimed that 'everyone agrees these are the best textbooks they've ever 
seen" (The Guardian 1992;. 

So how important is accuracy as a criteria for a good textbook? 

As part of our study we surveyed teachers and curriculum workers to see what they felt 
were the most important qualities that a good textbook should have. Using the opinions 
of a group of regional supervisors a list of the ten most important criteria for good 
textbooks was derived. A sample of forty-five teachers were then asked to rank each of 
these criteria in order of importance. The results are shown in Table 16.6 below. 



Criteria Teachers (45) Curriculum 

Specialists (34) 
Rank Mean Rank Mean 



Relates to local community 


1 


(8.1) 


6 


(4.4) 


Suits ability level of pupils 


2 


(8.0) 


2 


(8.5) 


Meets curriculum objectives 


3 


(6.7) 


1 


(9.3) 


Ideas/concepts logically 










sequenced 


4 


(6.2) 


4 


(6.6) 


Clear presentation of 










ideas/concepts 


5 


(6.1) 


3 


(7.3) 


Clear printing 


6 


(5.6) 


9 


(3.7) 


Correct/no mistakes 


7 


(3.8) 


10 


(3.0) 


Has activities which can 










be done 


8 


(3.7) 


7 


(3.9) 


Uses a variety of approaches 


9 


(3.6) 


5 


(4.6) 


Good clear illustrations and 










diagrams 


10 


(3.5) 


8 


(3.7) 



Tabic 16.6 The qualities needed for a good text book: opinions of teachers and 

curriculum specialists 

For teachers "relating to local community" and "suiting the ability level of the pupils" 



107 



ROLE OF TEXTBOOKS 



101 



were the two most important criteria. Accuracy was seen as being relatively unimportant, 
being ranked only seventh in order of priority and "good clear illustrations and digrams" 
was rated as the least important. 

The views of the thirty-four curriculum specialists in the curriculum directorate were 
also obtained. Their opinions correlated only moderately (rank order p » 0.56) with those 
of the teachers. Not surprisingly the curriculum workers placed greater emphasis on the 
textbook meeting "curriculum objectives" Relating to the local community was ranked 
only sixth. Both groups gave high priority to suiting the pupils ability level and to 
presenting ideas and concepts in logical sequence. 

But what was noticeable was that like the teachers the curriculum specialists gave a low 
priority to "technical" aspects; accuracy was ranked as being the least important of the 
ten, clear printing was ninth and clear illustrations and diagrams eighth in order of 
importance. 

The high priority given to curriculum objectives, suiting ability level, sequencing and 
presentation (contrasted with the low priority given to the technical criteria) suggests 
that the textbook is still seen as being a content or sequencing guide and the chief source 
of information for teachers. 

Problems with the technical criteria are only important when the pupils use the books 
since they can be mediated if the materials are teacher-directed. Similarly the low ratings 
given to activities and different teaching approaches suggests either that these are seen 
as having a low priority in teaching or that they are the domain of the teacher not the 
textbook. 

From the results we suggest that for these teachers and curriculum specialists the 
textbooks have a very specific function. They are guides to the curriculum which the 
teachers can feel safe in using. The "good" textbook provides the teacher with the 
information he or she needs to teach, assured that it suits the level of the students, is 
relevant to the local community and is clearly presented and logically sequenced. Setting 
out the teaching methods and the activities to be used are not seen as being a major part 
of a text book's functions. Activities are to be determined and organized by the teacher. 

The question is, is this view of the role of the textbook consistent with the aims of the 
Jordanian reform? Will we ever achieve pupil -centred, activity-based learning which 
develops critical thinking and problem-solving skills if the curriculum is still being 
mediated through the textbooks which in turn are used almost exclusively by teachers? 

References 

Abdalla A M (1990) An Evaluation of First Year Practical Chemistry in Jordanian 
Universities Unpublished PhD Thesis University of East Anglia 

Abu Sardaneh H (1983) Surveying the State of Biology Laboratories and laboratory 
Work in Government Secondary Schools in Jordan Unpublished MA Thesis University 
of Jordan 

General Directorate of Planning, Research and Development (1989) Progress of 
Education in the Hashemite Kingdom of Jordan. Report Submitted to the 41st Session 
of the International Conference of Education Geneva 

The Guardian 6/2/92 Mistakes Turn American History Books into Bunk 

KTiai. H A (1981) Using Locally Available Materials for School Chemistry in Jordan 
Unpublished MSc Thesis University of East Anglia 



17 Developing quality in education: a way 
forward 



R J D Rutherford, School of Education, The University of Birmingham, B15 2TT t 
UK 

INTRODUCTION 



Quality has now become a major concern in all sectors of education. But what is meant 
by quality and, assuming we can agree on a definition, where and how can quality be 
developed? This paper will address these three key questions, explore the relationship 
with total quality management, and describe a course for teachers which not only 
attempted to exemplify the principles of total quality management but also to prepare the 
course members to begin the work of quality development in their own schools. 



There are a number of overlapping perspectives - drawn from industry and commerce - 
which seek to define the meaning of quality. Each perspective has a rather different 
emphasis. Quality may be about perfection, inspection, conformance to requirements, 
fitness for purpose, improving customer/client satisfaction, or empowering the 
customer/client. These perspectives raise interesting implications when applied to 
education. Does inspection - the ritual pouring on of quality at the end of a process - 
mean that 20% of students must fail in order to "preserve standards"? Does conformance 
to requirements - which emphasises "quality systems" and "getting it right first time", 
prevention rather than inspection - reduce a teacher's freedom to abandon a well-planned 
class and attend to a particular concern or difficulty? Does fitness for purpose beg the 
question of whose purpose should have priority: teacher's or student's? Does 
empowering the customer/client - giving power to the student - mean enhancing their 
ability to exact sanctions if they think that the education they have received is 
unsatisfactory? 

In the opinion of the author, quality is about improving customer/client satisfaction or, 
in educational terms, improving the quality of learning for students. In practice, this 
means engaging in a systematic and continuous process of development and incremental 
change that is heavily dependent for its success on all the "partners" having meaning, 
purpose and satisfaction in their work. 



A very simple model of a school can be constructed which identifies the major processes 
in the system (Hall 1 989): inputs; transformations; and outcomes. Inputs include policies, 
staff, students, resources, and so forth. Transformations include teaching, tutoring, 
learning, assessing, as well as staff training and development. Outcomes include the 
academic, personal and social development of students and staff, and examination results. 
The satisfaction of students, their parents, their future employers, and so forth, with the 
education that has been provided should also be considered. It is clear that to develop 
quality in all these aspects of a school's work would be a formidable and in some 
respects perhaps an unnecessary task: priorities for development must be established. 
This issue has been well recognised in schools with the introduction of school 



WHAT IS MEANT BY QUALITY? 



WHERE CAN QUALITY BE DEVELOPED? 




DEVELOPING QUALITY 



103 



development plans (Hall 1990; Hargreaves and Hopkins 1991). Such plans identify 
specific objectives to be achieved during the forthcoming year and include detailed action 
plans to show how these are to be achieved, and longer term objectives for the following 
two or three years. Staff appraisal schemes are now being introduced in schools and will 
help to ensure that the needs and aspirations of individual teachers are in harmony with 
the needs and aspirations of the school, as articulated in the school development plan. 



Whale and Ribbins (1990) have led a major initiative in quality development in the 
Birmingham LEA. They have brought together the experience schools have acquired in 
development planning with a systematic approach to monitoring and evaluating. Building 
on their work, the author has emphasised the importance of considering the present and 
future needs of the stakeholders or partners in the school, of a systematic strategy for 
implementing planned change, and of leadership and team work. Thus developing quality 
in education is, in the author* s opinion, a systematic and continuous process that involves 
four core elements: strategic and action planning; implementing, monitoring and 
evaluating planned change; responsive and flexible leadership; and co-operative and 
supportive team work. These four core elements are clearly related to Oakland's (1989) 
four major components of total quality management: a quality system; statistical process 
control; management commitment; and team work. The author has found Oakland's ideas 
and analysis invaluable. The following quotation, in particular, has proved most useful 
when talking to colleagues about total quality management. 

"The author and his colleagues have heard the excuse that 'our industry (or organization) 
is different to any other industry (or organization)* in almost every industry or 
organization with which they have been involved Clearly, there are technological 
differences between all industries and nearly all organizations, but in terms of managing 
total quality there are hardly any at all" (Oakland p 159). 



The course was a three-term, ninety -hour course for experienced teachers in primary and 
secondary schools leading to an Advanced Certificate in Education or. with further study, 
an MEd. The first two terms* work consisted of lectures, discussions and workshops, and 
tutorial support for the assignments. The third term's work was devoted to action learning 
projects which focused on quality development in the teachers' own schools. The course 
was held not at the University but, on alternate weeks, at two of the teachers' schools - 
outreach centres, fhe author's aim statement for the course was: 

"To design and deliver a course - based on the four core elements: planning; feedback; 
leadership; and team work - that will enable the participants to develop the knowledge, 
skill, experience, confidence and enthusiasm to begin and to continue the work of quality 
development in their own schools." 

The course was built around ten key questions. The first four of these focused on 
strategic planning: What are we here for? Where are we now? Where do we want to get 
to? What do we need to focus on? Thinking about the second and third of these 
questions naturally leads to the fourth: the "quality gap". The fifth question was about 
action planning: How do we get there? 



HOW CAN QUALITY BE DEVELOPED? 



A COURSE FOR QUALITY DEVELOPMENT 




104 



QUALITY IN COURSES 



A systematic strategy for implementing planned change was described in response to 
the sixth question: How do we effect change? This strategy involved identifying the 
various stakeholders who were effecting or were affected by the change, and then 
analysing each group in terms of six decisive factors. These are: linkage; openness; 
gain/loss; ownership; leadership; and power (Lindquist 1978; Berg and Ostergren 1979; 
Rutherford 1992). The different purposes for monitoring and evaluating and the different 
techniques - questionnaires, interviews, and observation schedules - (Hopkins 1989) were 
also described to answer the seventh and eighth questions: How are we doing? How have 
we done? 

PERFORMANCE INDICATORS FOR THE FIRST TERM 

1. Target number of students: 

2. Number of students at the first session: 

3. Number of students who actually enrolled: 

4. Percentage of number of students enrolled/target: 

5. Percentage attendances for the nine taught sessions 
(the tenth session was a "reading week" at half term): 
87 88 72 83 94 83 94 94 66 

6. Average percentage attendance: 

7. Number of visits to see students in their own school: 

8. Percentage of visits to see students in their own school: 

9. Number of standard end -of -term questionnaires returned: 

10. Percentage of questionnaires returned at the final session 
(13 students were present): 

11. Medians of the students' responses on a 1 to 5 scale (where 5 is 
the most favourable response) to the ten statements on the questionnaire: 
45^5544544 



PERFORMANCE INDICATORS FOR THE SECOND TERM 

1. Target number of students: 18 

2. Number of students who are continuing with the course: 17 

3. Percentage of students continuing: 94 

4. Number of first assignments handed in within three weeks of the deadline: 14 

5. Percentage of first assignments handed in within three weeks of the 

deadline (17 maximum): 82 

6. Grades for the assignments: 



A A- A- A- B+ B+ B B B- B- C+ C and two returned unmarked 



7. Percentage attendances for the nine taught sessions: 
82 88 88 88 88 82 88 88 88 

8. Average percentage attendance: 87 

9. Number of visits to see students in their own school: 9 

10. Percentage of visits to see students in their own school: 53 

1 1. Number of standard end-of-term questionnaires returned: 11 

12. Percentage of questionnaires returned at the final session 
(15 students were present): 73 



13. Medians of the students ' responses to the ten statements on the questionnaire: 
5 545544445 

Tabic 17.1 Course performance indicators 



in 



DEVELOPING QUALITY 



105 



However, following Oakland's suggestions, some numerical performance indicators were 
also collected for each term's module of the course (Table 17.1). For example, the 
average percentage attendance; the percentage of assignments handed in for marking; the 
medians of the students* responses to an end-of-term questionnaire; the percentage of 
students who continued on to the following term. This data was, arg?;ably, invaluable for 
monitoring the quality of the course and to demonstrate that the author "practised what 
he preached". It does not explain the reasons for problems but certainly shows if 
problems exist. Such data has not been commonly collated in the School of Education 
but as an increasing amount of our work takes place in outreach centres or is on contract, 
it will be needed to fulfil any requirements for accountability. When similar data 
becomes available it will be very interesting to compare the performance of the courses 
at all four outreach centres, and indeed with more traditional courses that are held in the 
School of Education. 

In practice, these eight questions or stages in planning for quality development fuse into 
one another. And the process of planning is at least as important as the final document. 
However, ensuring that the plan is actually implemented, monitored and evaluated - 
which depends on leadership and team work - are the most difficult and challenging parts 
of the whole process. Thus the last two questions in the course focused on leadership 
and team work: How can we develop our effectiveness as leaders? How can we build 
teams? 

A number of perspectives on leadership and team building proved useful in the course 
(Rutherford 1992). First, there are the personal qualities and associated actions that 
followers look for in leaders if they are to give exceptional performance. Kouzes and 
Posner (1987) identify four such qualities: honesty; competence; vision; and enthusiasm. 
They emphasise that credibility is the foundation of leadership. Second, there is the work 
of Adair (1990) which focuses on three complementary functions that the effective leader 
must attend to: achieving the task; building the team; and developing the individuals in 
the team. Adair (1990) and Belbin (1981) have much to say about building teams that 
the course members appreciated. Third, there is the work of Blanchard et al (1987) 
which emphasises that followers have different needs, which depend on the particular task 
in hand, so that leaders need a variety of styles and the ability to match the appropriate 
style to the follower's needs. Fourth, the work of Argyris (1982) and Schon (1987) on 
iheory-of-action underlines the difficulty of practising what we preach, of ensuring that 
our behaviour really does match our stated values. 

CONCLUDING REMARKS 

The course is interesting in that it aims both to prepare the participants to engage in 
quality development in their own schools, and to be an exemplar of a "quality course". 
A great deal of care went into planning and delivering the course, and into providing 
tutorial support. Visiting teachers in their own schools proved extremely beneficial in 
helping to build and cement good working relationships, and to talk through problems. 
"Keeping close to the customer" was one of the author's priorities. A strong sense of 
group identity and commitment developed during the course which was reflected in the 
performance indicators. Oakland's framework for total quality management has provided 
ihe theoretical basis and inspiration for the course and, in the author's opinion and 
experience, these principles can be applied very successfully in education. They underpin 
a way of working described in this paper that can empower individuals to deliver quality 
in education. However the larger task remains: introducing total quality management 
throughout the whole school. 



112 



106 QUALITY IN COURSES 

References 

Adair J (1990) Not Bosses but Leaders 2nd Edn Kogan Page London 

Argyris C (1982) Reasoning, Learning and Action Jossey-Bass San Francisco 

Beibin R M (1981) Management Teams Butterworth-Heinemann Oxford 

Berg B and Ostergren B (1979) Innovation Processes in Higher Education Studies in 
Higher Education 4 pp 261-268 

Blanchard FC, Zigarmi P and Zigarmi D (1987) Leadership and the One Minute Manager 
Fontana Glasgow 

Hall J Ed (1989) Local Education Authorities Project Management in Education BBC 
Milton Keynes 

Hall J Ed (1990) Local Education Authorities Project Locally Managed Schools BBC 
Milton Keynes 

Hargrcaves D and Hopkins D (1991) The Empowered School: the management and 
practice of development planning Cassell London 

Hopkins D (1989) Evaluation for School Development Open University Milton Keynes 

Kouzes J M and Posner B Z (1987) The Leadership Challenge Jossey-Bass San 
Francisco 

Lindquist J (1978) Strategics for Change Pacific Soundings Press Berkeley CA 

Oakland J S ( 1 989) Total Quality Management Buttervvoth-Heinemann Oxford 

Rutherford D (1992) Appraisal in Action: a case study of innovation and leadership 
Studies in Higher Education 17 pp 201-210 

Schon D A (1987) Educating the Reflective Practitioner Jossey-Bass San Francisco 

Whale E and Ribbins P (1990) Quality Development in Education: a case study of 
supported self -evaluation in Birmingham Educational Review 42 pp 167-179 



1 1 9 



18 Quality in course design: empowering 
students through course processes and 
structures in a professional development 
scheme 

Eric Roper, feeds Metropolitan University, Leeds, LS6 3QS, UK 
INTRODUCTION 

The PGDip/MEd in the Faculty of Cultural and Education Studies at Leeds Metropolitan 
University (formerly Leeds Polytechnic) is a professional development scheme for 
educators working in all sectors of the education system. It is argued that the quality of 
participants' experiences in the scheme depends on a flexible and coherent scheme 
structure which is consistent with and responsive to clearly articulated scheme processes 
The scheme will be examined in relation to three perspectives on quality: fitness for 
purpose, how «t meets participants' needs and how it empowers participants to take 
charge of their own professional development. 

In 1990 the MEd was re-designed as a linked, modular Postgraduate Diploma/Master 
of Education Scheme. What had been a three year, cohort-based, part time MEd in 
applied research in education, became a flexible professional development scheme which, 
although it retains a commitment to using research findings in exploring educational 
practices and to the preparation of participants to undertake naturalistic research in the 
MEd dissertation, also offers a wide range of elective modules which enable participants 
to examine substantive areas of educational practice. 

The PGDip consists of 12 modules, 6 core and 6 elective; satisfactory completion of the 
PGDip permits entry to the Master's stage, which is based on a 16,000 wo;i dissertation 
(equivalent to 8 modules). As well as Curriculum Lssues and Assessment Issues, which 
provide frameworks for conceptualising key elements of the educational process, the core 
PGDip modules are Approaches to Research in Education, Collaborative Action Research 
and Recording. Reviewing and Planning (double module); for participants intending to 
proceed to the MEd. two of the six elective modules must be Case Study Research I and 
Case Study Research II. There are currently 32 validated PGDip elective modules. 
These include generic, subject specific and sector specific modules, for example Human 
Relations, Mathematics Education and Early Years modules, and participants may take 
management-focussed electives from the MSc in Education Management. 

SCHEME PROCESSES 

The PGDip/MEd has been designed as a professional development scheme and, as such, 
seeks to empower participants to take charge of their own professional development. The 
flexibility stemming from its modular design facilitates this, but flexibility of this kind 
has been tempered by a commitment to certain scheme processes which constrain 
participants' freedom of choice, at least at the structural level, for example by limiting 
the number of PGDip electives to six (four if participants intend to proceed to the MEd), 
Because the scheme seeks to promote in participants the ownership of change as a basis 
for professional development and professional learning, the methodology of all modules 
seeks to enable participants to articulate their professional experiences, re-examine and 
make explicit their professional values and goals, reflect on and analyse taken for granted 



114 



108 



QUALITY IN COURSES 



arrangements and practices, and identify any felt unease about their day to day 
professional activities in terms of mis-match between aspiration and present experience. 
Professional development is grounded in such activities, for it is through them that 
participants come to understand the nature of and need for chaw "^cognise what it is 
in schools, classrooms and themselves that makes change diffcu.c to accomplish, feel 
some control over the situation, construct and own agendas for personal action, and 
develop a motivation for change that is personally founded (Rudduck 1988). 

All scheme modules focus attention on how individuals and groups, including scheme 
participants, see their situations, and on how these different perspectives are influenced 
by prevalent educational theories, ideologies and personal value systems, and all make 
use of active teaching and learning strategies which sharpen participants' skills to act in 
these situations. Moreover, ail modules must address the generic objectives of the 
scheme: to enhance participants' self- awareness, insight, cepacity for constructive 
criticism, capacity for rational action based on reflection, interpersonal skills and active 
professionalism. 

The double PGDip core module Recording, Reviewing and Planning underpins the 
whole scheme and is central to its process. It involves participants in reflecting, 
reviewing, recording and target setting, collaboratively as well as individually. These 
activities enhance participants* understandings of themselves, and support their search for 
identity as professionals. They enable participants to feel some sense of individual power 
at the centre of the action, not only in terms of the scheme itself, but also in relation to 
their professional lives as the scheme activities impact on their day to day work. 

The research modules in the PGDip are derived from the original MEd and are intended 
to provide participants with a thorough grounding in the naturalistic research methods 
they are expected to use in their dissertations. Furthermore, experience in the original 
MEd confirmed the value of a course process grounded in naturalistic methodology for 
promoting both the personal and professional development of practitioners: participants 
were ao. to identify ways in which their interactions with colleagues had been 
transformed by their understanding of, for example, the need to explore the meanings 
invested by different individuals in events and practices in the workplace, by their 
articulation for themselves of ethical principles of procedure, and by their development 
of the interpersonal skills needed by the naturalistic researcher. 

The research modules arc also intended to empower participants to make enquiries into 
the learning which they facilitate. Priority is given to equipping all participants in the 
scheme to approach in a more informed and skilled way the situations and problems 
which they encounter, or might expect to encounter, during their daily professional lives. 
As a means to this end, it is thought to be of fundamental importance that teachers and 
other professionals in education who are responsible for actions within professional 
settings should be equipped to research the consequences of such actions and 
progressively identify and focus attention on significant and important issues and 
problems in the specific settings in which they occur. In this sense the scheme attempts 
to bridge the gap between practitioner and researcher as a basis for participants* 
professional development. 

SCHEME STRUCTURES 

Scheme structures should be fit for these purposes and, ideally, take their form from 
scheme processes. 

The flexibility inherent in a modular scheme, together with the range of electives, has 
certainly assisted in the marketing of the scheme which, now in its third semester of 



115 



QUALITY IN COURSE DESIGN 



109 



operation, has over 50 participants. Unlike the original MEd, participants may join the 
scheme in any semester (starting in October or February), and may take semesters out or 
vary the number of modules taken in any semester, according to personal and professional 
circumstances. The Scheme Leader manages a staffing budget and, because most elective 
modules have been designed to be offered as taught or tutored modules (with contact time 
adjusted pro rata according to numbers of participants) or by independent study, it is 
possible to offer participants their preferred elective modules by adjusting the mode of 
delivery to take account of available resources. 

Because most scheme modules generate activities and investigations that participants 
complete in the workplace, they are delivered as long, thin modules, one in each half of 
the PGDip/MEd evening, using 10 or 12 of the sessions in the 15 week semester. Taught 
modules have c 18 hours of contact time out of a total workload of between 45 hours 
(minimum) and 75 hours (maximum). 

The Recording, Reviewing and Planning double module is completed in action learning 
groups (ALGs), consisting of approximately six participants and an ALG adviser, meeting 
throughout the period of its members registration for the PGDip. (ALGs are reconstituted 
for the Master s stage of the scheme). In the PGDip these groups meet on six occasions 
each 15 week semester, for periods of between one and three hours, giving a total of c 
10 hours each semester. This arrangement is a deliberate attempt to counteract what 
could in a modular scheme become a fragmented experience, by enabling participants to 
identify with a cohesive group. 

Action learning groups are mutual support groups, which provide help and advice for 
members, who are engaged on similar practical tasks. In the PGDip the task is to 
complete the Recording, Reviewing and Planning module, while at the MEd stage ALG 
members are preparing their dissertations. The ALG adopts an action frame of reference, 
and members question each other about progress and blockages in exploring what has 
been done and what needs to be done. Each member is challenged to verify facts and 
interpretations, and to justify proposals. By reflecting on what they are doing, members 
of the action learning group see how to do it better, and consult others in the group in 
the act of getting it done. In helping each other, participants see more clearly how to 
help themselves. The variety of participant professional backgrounds in the ALG 
(typically reflecting all sectors of the school sys f em, further and higher education and 
nurse education) is an asset, for example h challenging members" definitions of their 
situations. 

Scheme tutors act as advisers to action learning groups, and are responsible for 
providing participants with both academic and pastoral support. In the ALGs participants 
explore their reactions to the scheme, their professional development concerns and goals, 
and identify ways in which the scheme might be used to meet individual needs and 
achieve personal goals. In the PGDip, the tasks participants complete with the support 
of their ALG are in practice: 

♦ log/diary/journal keeping, recording their experiences in the scheme and making 
links between the scheme and the workplace; 

♦ the preparation of programme proposals for the next semester, which state 
professional development targets and identify strategies for achieving them, and 
which can irclude proposals for Independent Study modules (written by the 
participants themselves to meet identified professional development needs) and 
requests for advanced standing based on previous study or professional 
development experience; and 

♦ the preparation of the Statement (the module assignment for Recording, Reviewing 
and Planning submitted at the end of the last PGDip semester, and requiring 



lie 



110 



QUALITY IN COURSES 



participants to identify their learning and development during the PGDip stage of 
he scheme, grounding their claims in data extracted from the log/diary/joumal, 
programme proposals and module assignments). 

The style of working in the ALG and the production of the Statement also contribute to 
participants' understanding of naturalistic research in preparation for the MEd dissertation. 
Work in the ALG makes participants aware of the diverse "multiple realities" of ALG 
members as they give meaning to their experiences in the scheme, while the Statement 
is a kind of case study of each participant's experience of the PGDip stage of the scheme, 
and is focussed on issues which are explored and grounded in data. The delivery of the 
module Case Study Research II, which is focussed on data analysis and presentation, is 
able to draw on participants' experiences of preparing their Statements. 

Participants' programme proposals are examined by the scheme's Approvals Board. 
The scheme sets out to embody maximum flexibility and deliberately imposes few 
restrictions on participants* decisions about the order in which to take modules, because 
individual participants will start from different points in terms of previous experience and 
initially perceived needs. ALGs help participants to articulate their growing awareness 
of their own professional development, and tG develop programme proposals which match 
their needs. The scheme process is an enabling one, supporting participants in developing 
programmes which have coherence and progression in terms of individual professional 
development needs. The headings to be used in writing programme proposals, 
independent study module proposals and requests for advanced standing, together with 
the criteria to be applied by the Approvals Board in considering them, are stated in the 
scheme handbook. 

Because the PGDip/MEd was validated as a professional development scheme, rather 
than as a course in the traditional sense, an Approvals Board was considered essential for 
examining participants* proposed programmes of study; its existence also enables modules 
to be revised and new elective modules added to the scheme in response to the emerging 
needs of participants. The criteria to be applied by the Approvals Board have an enabling 
function; they are written to support participants* in arguing the case for individual 
programmes of study which meet articulated professional development needs. 

In the scheme documentation, attention has been given to the information needs of 
scheme participants and tutors. As well as stating the criteria to be applied by the 
Approvals Board, the scheme handbook includes protocols < which set out scheme 
requirements) for the log/diary/joumal, action learning groups (identifying the 
responsibilities of both advisers and ALG members), the Statement, MEd dissertations, 
and procedures for negotiating a research contract in educational settings; guidelines for 
module assignments, programme proposals and for independent study module proposals; 
glossaries of scheme and research terminology; and module syllabuses. 

Within the scheme, module syllabuses (other than those which describe research training 
experiences) are seen as frameworks upon which participants and tutors can negotiate 
module programmes which meet the professional development needs of participants, as 
identified in their programme proposals. The scope of the negotiation involved not only 
covers matters of content in relation to objectives, but also of learning and teaching styles 
and assessment. The topics, issues or enquiries which participants pursue for assessment 
purposes may arise from programme content or contribute to it, and module assignments 
may be produced in a form suitable for a specific audience or use in the workplace (as 
long as they address the scheme's three assessment domains: literature and theory, 
reflection and analysis, and application and rcflexivity). 



117 



/ 



QUALITY IN COURSE D.2SIGN 111 

QUALITY IN COURSE DESIGN: SOME EMERGING ISSUES 

It can be argued that the quality of course design is best gauged by the quality of 
participants' experiences, and that in a professional development scheme the latter 
depends on developing a flexible and coherent course/scheme structure which is 
consistent with and responsive to clearly articulated course/scheme processes - processes 
which empower participants to take control of their own professional development by 
identifying and meeting professional development needs. That this has been achieved to 
some degree is evidenced by participants' perceptions of the strengths cf the scheme, and 
of the opportunities it offers them. Nevertheless, weaknesses and threats as perceived by 
participants and tutors have to be addressed. 

The commitment of the scheme as a whole to the naturalistic inquiry/research paradigm, 
and the implications of this for participants' programmes and activities both within the 
scheme and in their own work situation, is made clear to all prospective participants in 
advisory seminars and interviews. Nevertheless, participants understand and/or are 
committed to the scheme process to different degrees, particularly in the early semesters. 
Thus in a SWOT analysis of the scheme after the first semester of operation, groups of 
participants identified the following as "threats": "people who are pressuring for an old 
style MEd and won't engage in active learning (and the danger of tutors going along with 
them even if it goes against the ethos of the course)"; "getting to know you activities" 
(presumably the structured trust and climate building activities used in the first sessions 
of the semester prior to participants forming themselves into ALGs); programme 
proposals; and the "journal - for some". 

As the scheme grows, participants who have overcome the "general insecurities which 
come with starting a new course" will be able to reassure new entrants, but in a sense 
participants will only come to understand and value the scheme processes through 
engagement in them. The quality of their early experiences in the scheme will be critical, 
and the perceived "disparity between the quality of leadership of the ALGs", and 
participants' "not being absolutely clear", and the "the number of new elements" they 
have to cope with, are perceived "threats" that ALG advisers will be able to help 
participants deal with only if the expanding team of ALG advisers understands and shares 
the ethos of the scheme and the way this is translated into processes and structures. 

In the first semester of operation, the ALG advisers were those tutors who had 
developed the use of ALGs in the original MEd, and so had experience of the ALG 
adviser role. As the team has expanded not all ALG advisers have had previous 
experience of the role. Even for those tutors who have had this experience, the role has 
expanded: as expressed by one group of participants, the "limited amount of time 
available presents a threat when considering the complexity of professional issues" to be 
addressed; and the range of tasks to be completed with the support of the ALG has 
increased, as has the range of skills participants are required to develop. 

That some participants have found the log/diary/journal keeping a threat is 
understandable, because it involves the identification of feelings (diary) in relation to 
events (log) and, standing back somewhat, the acknowledgement of the issues, threads 
and themes that they reveal and that are of significance in the participant's professional 
understandings and actions (journal) (Holly 1989). In the ALG participants are 
encouraged to share from these records, and although they control the release of this data, 
this sharing requires the disclosure of personal information. If they are going to reveal 
themselves to others in the ALG, trust in other group members is essential. Issues 
relating to the timing and duration of ALG meetings are significant here, as participants 
have identified: "ALGs - too infrequent to be the driving force of the programme - 



IIS 



112 



QUALITY IN COURSES 



sessions too long - more frequent, possibly shorter, sessions (needed)". 

In the participants' first semester, much of the time in ALGs is taken up with coming 
to understand the scheme requirements, particularly in relation to log/diary/journal 
keeping and programme proposals, and although these are intended to be enabling devices 
for identifying and working on professional development targets and strategies, and 
participants are encouraged to find their own best ways of completing them within the 
requirements and guidance provided, not all participants "own" them in this way. It takes 
a minority of participants considerable time to see their value for their own purposes 
rather than as externally imposed "threats", and either concerns about what is expected 
are expressed and have to be dealt with or they are completed in a mechanistic fashion, 
with little investment of self in the process and with little evidence that professional 
development targets and strategies are grounded in log/diary/joumal work. There is a 
tension between using time in the ALG to promote the scheme's agenda in terms of 
requirements, and giving participants the opportunity to create their own agendas, which 
is perhaps reflected in the view that there is "not enough 'informal' time in the ALG". 

Participants are encouraged to use the ALG, log/diary/joumal, independent study- 
opportunities, modules and module assignments strategically in order to achieve the 
professional development targets identified in their programme proposals. Ideally 
participants will take modules in the scheme in order to work towards these targets, and 
will know enough of what they want from a particular module to negotiate appropriate 
content and delivery strategies. This negotiation may, however, be too early for optimum 
decision making and not allow for the clarification and refining of targets or the 
identification of more significant targets as the module unfolds. Although most modules 
are validated for delivery in different modes in order to make them more available, some 
participants prefer taught inodules and are disappointed that modules they wish to take 
are not available in this mode of delivery; on the other hand, many participants prefer 
modules to be delivered as tutored modules, giving them more time to work 
independently or in syndicates and to focus on their own particular issues and concerns. 

The documentation provided in the scheme handbook has provoked the following 
comment: "scheme outline difficult to understand, jargon laden - needs to be written in 
user- friendly English". While not wishing to dismiss this criticism, it reveals the problem 
of how to initiate new entrants into the culture (including the language) of the 
PGDip/MEd scheme so that, through their own activities within it. they can begin to own 
and shape it. 

ASSURING QUALITY: WAYS FORWARD 

Participants' and tutors* experiences within the scheme are reviewed each semester in 
formal scheme monitoring meetings. One source of data for these meetings is the 
University's Student Evaluation Questionnaire, designed for use across all courses but 
permitting course-specific items to be added. The challenge faced by scheme tutors (and 
participants) is to develop items for this instrument which reflect the culture of the 
scheme and the specific issu > identified by participants in SWOT analyses of the scheme 
as a whole and in module reviews. ALG advisers have regular contact with participants, 
and given that one of the tasks of the ALG is to help participants to review their 
experiences in the scheme, detailed feedback is available to tutors. The Statements 
produced by participants at the end of the PGDip will provide valuable additional data. 

Given the commitment of the scheme to naturalistic methods of inquiry and research, 
it is intended to apply these in evaluating the scheme itself. Currently the scheme leader 
is using action research methods to investigate his own role as an ALG adviser, and this 



QUALITY IN COURSE DESIGN 



113 



will produce materials for use in ALGs and in staff development activities with tutors 
(Roper 1992). There is scope for ALG advisers to undertake collaborative action research 
into this centrally important tutor role within the scheme, and for ALGs themselves to 
explore their different cultures in inter-group meetings and activities. 

Team building and staff development activities are a priority, now that the scheme is 
entering its third semester and the tutor team is expanding. Some of these activities will 
be grounded in data and documents produced within the scheme: for example, reports of 
action research undertaken by tutors into key roles within the scheme, participants* 
programme proposals and Statements. Time has been found to enable potential ALG 
advisers to observe established advisers in ALG meetings, and for ALG advisers to meet 
together to share experiences and provide mutual support. Module leaders need to share 
their experiences of getting modules off the ground, of negotiating module content and 
delivery with participants and of delivering modules in different modes. 

It is also intended to return to basics and for tutors to identify and debate issues and 
approaches in teacher professional development, drawing on the growing body of teacher 
development literature (Hargreaves and FuIIan 1992), which may offer conceptual 
frameworks for appraising the PGDip/MEd as a professional development scheme and 
suggest possible activities which can be developed for use within the scheme (e.g. the 
writing of collaborative autobiography by members of ALGs). Similarly, literature on 
collaborative enquiry may help tutors to appraise the delivery of scheme modules and the 
operation of ALGs - posing questions, for example, about die extent to which the latter 
have transcended contrived collegiality and developed genuinely collaborative cultures 
(Little 1990; Hargreaves 1992). An important reason for the success of the scheme in 
attracting participants is its ethos and vision, and its commitment to process, and these 
need to be reviewed and reconfirmed as the scheme expands. 

References 

Hargreaves A (1992) Cultures of Teaching: a focus for change in Hargreaves and 
FuIIan (Eds.) 1992 

Hargreaves A and Fullan M G (Eds.) (1992) Understanding Teacher Development 
Cassell London 

Holly M L (1989) Writing to Grow: keeping a personal - professional journal 
Hcinemann Portsmouth. NH 

Little J W (1990) The Persistence of Privacy: autonomy and initiative in teachers' 
professional relations Teachers College Record 91 pp 509-36 

Roper E H (1992) How to Intervene? - Facilitating action learning groups in Bruce C 
S and Russell A L (Eds.) Transforming Tomorrow Today (Proceedings of the Second 
World Congress on Action Learning) Action Learning, Action Research and Process 
Management Association Incorporated Brisbane 

Rudduck J (1988) The Ownership of Change as a Basis for Teachers' Professional 
Learning in Caldcrhead J (Ed.) Teachers ' Professional Learning Falmer Lewes 



120 



19 Quality horses for quality courses: 
matching students with courses in music 



Louise Gibbs, Goldsmiths' College, University of London, London, SE14 6NW, UK 
INTRODUCTION 

Selecting suitable candidates for courses in further and higher education is undoubtedly 
crucial to the quality of course outcomes and thus the perceived quality of a course. Poor 
or unsuitable students are as likely as poor teaching provision to undermine the evaluation 
of a course's quality. While we may have little control over the quality, academic or 
otherwise, of those who present themselves as prospective students, nonetheless, the 
responsibility for deciding upon what are quality students and selecting them remains 
ours. In the cycle of educational activities and thus of quality procedures, student 
recruitment and interview marks a decisive point. Misjudged selection or mismatching 
of courses and students may make any further evaluations of course quality redundant. 
In this paper I wish to argue that: 

1 the perception of course quality is individual as well as general; students' 
involvement with quality assurance procedures starts with course advising 
which should be seen as a case of matching rather than selecting 

2 the conditions of adult and continuing education offer examples of client- 
sensitive course advising, the analysis of which can offer insight into how 
student-based quality procedures may work and thus be improved 

3 like any other aspect of a service, course advising needs clearly defined 
procedures but also requires the application of "intuition" to be sensitive to 
the interpretation and judgement of individuals and their circumstances; a 
quality assurance system requires much the same approach if it is to be useful 
analysing and recommending on the particular as well as the general. 

ASSESSING STUDENT SUITABILITY 

In the cycle of course design, planning, teaching and assessment we, as providers of 
courses, have some direct control and thus responsibility for ensuring quality in terms of 
"fitness for purpose". If the fitness for purpose of a course can be me;isured by its ability 
to graduate students who, as a consequence of the course: 

• maintain attendance and do not "drop out" 

• achieve high examination marks 

• perform well 

• find employment quickly and maintain it satisfactorily 

• obtain personal satisfaction 

• acquire deeper knowledge and appreciation of subject 

• are motivated and equipped to pursue further study 

then we attempt to choose those students who we feel have the potential to last the course 
and achieve well in relation to these outcomes. However, we often have no more 
sophisticated an instrument for assessing student suitability to study than the absorption 
of a body of knowledge and the ability to pass examinations; we often judge and admit 
students to further and higher education on the basis of GCSE or A level examination 



MATCHING STUDENTS WITH COURSES 



115 



results even without seeing or interviewing them (Ramsden 1986). 

When a course is designed to provide for a clearly specified need, in the way of specific 
training rather than a general educational aim, it is not difficult to formulate a model of 
the kind of person suitable for such a course and potentially able to fulfil its originating 
purpose. If you want to train an orchestral musician, for instance, you select a competent 
player, with good sightreading skills, who can blend musically, work well with other 
players and take direction. As the selector you are the "expert" (Williams 1986). You 
know the standards required. You have formulated some picture of the ideal orchestral 
musician, the skills, experiences, knowledge and attitudes which are likely to make not 
only a potentially good player but also a good student who would benefit from the kind 
of training that you offer, and you select accordingly. The prospective student's role in 
this is to accept an offer or reject it - not usually to negotiate the basis upon which it is 
made. Entry to most further and higher education and training is controlled in this 
apparently autocratic way. The institution sets the entry standards, the curriculum, and 
the methods of assessment. However, in the end it is student performance which 
vindicates or dwnns institutional provision. 

ISSUES OF QUALITY IN MATCHING STUDENTS WITH 
COURSES 

But what is a "quality" student? Are not those who genuinely wish to study, potentially 
"quality" students? And if there can be unsuitable students for courses, can there not also 
be unsuitable courses for students? Lack of fit must also be an aspect of lack of fitness 
for purpose. And while fitness for purpose can address itself to the outcomes of 
educational provision does it not also incorporate the idea of quality of the means in 
achieving the ends? Effective initial course advising and consequently better fitted course 
placement. I argue, is the first step in the student- institution relationship of a quality 
assurance kind. But only if the student is involved and takes responsibility for quality 
assurance, as does the institution itself by example. Because the relationship between 
institution and student is essentially unequal, it requires the institution to invite the 
student to participate. The recent "market-speak" description of our students as "clients" 
may indicate not only that institutions see themselves as offering a service, but also that 
the relationship between student ana institution is less paternalistic, and more negotiable 
in the way of a partnership (Jarvis 1983). The relationship between student and 
institution as far as quality assurance is concerned must also surely be one of partnership. 

The lessons now being learned in the new generation, market-led higher education have 
probably been encountered by adult continuing education some time ago. Higher and 
further education could have something to learn from the adult continuing education ethos 
if not its method. Adult and continuing education (ACE) has traditionally attempted to 
anticipate or respond to the perceived usually nonvocational needs of the local community 
based around a concept of personal enrichment or liberal education (Jennings 1981). 
Students are involved by choice and usually towards no particular extrinsic goal other 
than the personal satisfaction of involvement, study or learning. However, with ACE 
moving increasingly into accredited and award-bearing courses preparatory to further and 
higher education or continuing professional development, teaching and learning is 
becoming as goal-oriented as that within further and higher education (FHE). 

The greatest contrasts between liberal ACE and qualification seeking FHE-type courses 
would be in the more obvious autonomy and agency in their education that students have 
in ACE (see Jarvis 1985). ACE students are used to paying directly out of their own 
pockets for their courses and thus they often feel as if they have greater control over how 



122 



116 



QUALITY IN COURSES 



they are taught. They may also have a hand in proposing or designing what they learn 
along community education lines (Allen et al 1987). And in my own experience of AE 
teaching I have more than once felt the need to alter my course plan if it has seemed 
inappropriate to the requirements of the majority of students or lose the course altogether. 
These students support what they feel to be good courses by "voting with their feet 1 '. 
They can walk right out again if it is not to their liking, and as students used to making 
moves for themselves they are less hesitant about demanding their money's worth in the 
way of adequate resources and relevant and effective teaching. The point is that adult 
education students tend primarily to select their courses rather than being selected for 
them, and because of their relative autonomy, their support or lack of it is a potent 
indicator of the quality of a course certainly from the student' perspective. 

The apparent autonomy of the ACE student makes for a dangerous life in adult 
continuing education planning but sharpens the mind to quality issues. When students 
abandon ACE courses in large numbers the course closes but this loss alerts you to a 
possible quality problem. When you lose a few individuals you may never see them 
again. The trouble is that issues of quality or lack of it often surface when it is too late. 
The usual student contribution to course quality assurance is the course evaluation 
questionnaire handed out towards the end of a course. While they can provide very 
useful suggestions to the fine tuning of quality course provision, the answers are fairly 
predictable because favourable evaluations tend to come from those who are satisfied 
enough to stay the course. The most telling evaluations I would suggest come from those 
who leave before the end, as this brief report may illustrate. 

To give background to a quality evaluation exercise within the music programme in the 
Department of Continuing and Community Education, Goldsmiths* College, an enquiry 
was made into the reasons why students dropped out of four music classes. Tutor 
permission is needed to enrol for all these classes and course advice is available. Some 
figures briefly describe the results: 

Course: (A) (B) (C) (D) Total 

Enroled numbers: 15 18 12 12 57 

Left early: 4 4 10 9 

Reasons for leaving course: 

-domestic or employment commitments: 3 

-travel problems: 1 

-course level too demanding: 2 (1 found another DCCE course) 

-course level not demanding enough: 2 (2 found another DCCE course) 
-course not quite what was expected: 1 v' 1 realised course would need some 

commitment") 

These figures in themselves are not significant and certainly on the basis of the sample 
not general is able to the whole music programme which caters for approximately 800 
students. What is important to me at least is the fact that out of nine ex-students five left 
because of issues to do with the quality of the course. Although three subsequently 
received course advice and found another DCCE course, two out of the five slipped 
through, did not take advantage of the course advice service at all and left the DCCE 
music programme altogether. While student perceptions are important indicators of the 
quality in terms of teaching effectiveness and communication, institutions and the teachers 
and "experts" that constitute them have the ultimate responsibility for determining the 
relevance and content of what is taught and thus the quality of overall provision. 

For students to assume some of the responsibility for the quality of their education they 
need to be involved on a personal level and they also need to feel that they arc receiving 



MATCHING STUDENTS WITH COURSES 



117 



personal attention. Students feel more inclined to ask your advice if they know your 
name and face, though from my own point of view this kind of input to quality is a 
double-edged sword. More informative prospectuses and precise course descriptions 
help, but experience shows that face-to-face course advice is what prospective students 
are after. Many of you will recognise the situation where the answer you are giving to 
a telephone enquiry is exactly that given by the prospectus held in the hand of the 
enquirer at the other end of line! 

ENHANCING THE QUALITY OF COURSE ADVISING 

There's a saying that if you wish to leam about something you should teach it. In 
teaching some classes on course advising on the newly-launched Certificate in Music 
Teaching to Adults course, I became aware of the value not only of having clear course 
advising procedures but also of the use of "intuition" in applying them. I became more 
explicitly aware of this in having to engage my students in simulated student-course 
advising activities in the way of role play. Here is an example of a role play question: 

"You are about to retire and you will now have time to devote to learning music. You 
own a number of chord organs and electronic keyboards and have taughi yourself to 
play by 'fiddling about*. You now feel that it's time to learn to play 'properly'." 

One student takes the role of the prospective student and the other the role of the course 
adviser. 

While it was not too difficult to coach or point out to my students the general lines of 
enquiry to be made, or even in what manner questions could be fielded, unanticipated 
answers were received which they were nevertheless able to respond to when prompted 
or led. We also framed the same role play game into the form of questions submitted to 
an adviser and requiring a written reply. My 14 students gave some very individual 
answers to the same question, despite making a generally similar analysis in class after 
the activity had been done of the information needed by the counsellor and prospective 
student. The class answers were analysed against my own answer given as the "expert". 

At the start of this particular class, the quality of the advising done by the students was 
surprisingly variable. However, once presented with a system in the way of a checklist 
of questions, there was noticeable improvement all round with the questioning being less 
haphazard and the process taking a shorter time to reach some conclusion. Here is a 
somewhat abbreviated form of the procedures used in the course advising training 
sessions described above: 

1 Find out what and why the student wants to study and preparedness for study 

• by direct questioning 

• by indirect questioning 

• by audition 

• by examining portfolio 

• by diagnostic test 

What you need to know about the prospective student 

• Musical interests 

• Musical experience, if any 

• Music educational background, if any 

• Reasons for study 

• Level of seriousness, commitment to study 

• Appropriate home study or practice support/instrument 



124 



118 



QUALITY IN COURSES 



• Other commitments - family/employment 

2 Suggest suitable course(s) or programme of study 

• programme prospectus 

• course descriptions 

What you yourself need to know 

• Scope of music programme on offer 

• Entry requirements, content and level of individual courses 

• Other courses which your own programme accesses 

• Musical activities and experiences to which your courses can contribute 

• Further study in music 

• Employment opportunities in music 
What in general you must have 

• A wide knowledge of musical genres, styles and practices and the kinds 
of skills, knowledge and attitudes that are part of these musical traditions 

• An ability to listen carefully 

• A genuine interest in placing prospective students within suitable courses 

• An ability to analyse suitable levels of musical knowledge and experience 
allied with a scheme of progressed instruction in music 

3a) If suitable course(s) found 

• outline course commitments and expectations, agree them, and ask if there are 
any further queries 

3b) If no suitable course(s) found 

• suggest further lines of enquiry or other advisers 

While it was not to difficult to spell out the general kinds of questions an adviser could 
field, the particularity of responses require a flexible approach. There seemed to be times 
when the checklist had to be abandoned or not followed in an evidently systematic way. 

THE PLACE OF INTUITION 

Earlier I proposed the notion of the use of "intuition" in advising prospective students. 
The way that it is used here is to do with openmindedly interpreting and making a 
"creative" guess or suggestion about a particular situation or set of circumstances you 
may not have encountered before. I would suggest that what I call "intuition" in this 
context is not just a flash of insight coming from nowhere and without antecedent or 
preparation. The basis of Intuition is often unfathomable at the time but an aspect of what 
has been called "tacit knowledge" or of knowing more than you can tell (Polanyi 1966). 

You apply intuition when you creatively and imaginatively manipulate your own 
knowledge and experience to solve a problem that is new to you or has unique 
circumstances, often when formal systems or procedures cannot give guidance. Intuition 
may have an unarticulated basis, it may also be "untidy" in the way that it operates, but 
is flexible, sympathetic to the particular, and thus personal. Because of this, I would 
advocate that intuition is an important part of human interaction. If we are evaluating 
other human beings and their experience and potential as we inevitably do as teachers and 
educators then to be fair we need to remember that we apply both systematic procedures 
and intuition. 



MATCHING STUDENTS WITH COURSES 119 

References 

Allen G et al (eds) (1987) Community Education: an agenda for educational reform 
Open University Press Milton Keynes 

Jarvis P (1983) Professional Education Croom Helm London 

Jarvis P (1985) The Sociology of Adult and Continuing Education Croom Helm London 

Jennings B (1981) Adult Education in the United Kingdom Newland Paper No 3 
University of Hull 

Polanyi M (1966) The Tacit Dimension Routledge & Kegan Paul London 

Ramsden P (1986) Students and Quality In Moodie G C (ed) Standards and Criteria 
in Higher Education SRHE & NFER-Nelson Guildford 

Williams G (1986) The Missing Bottom Line In Moodie G C (ed) Standards and 
Criteria in Higher Education SRHE & NFER-Nelson Guildford 



20 Recognising quality in engineering 
education 



D C Hughes and R G S Matthew, Dept of Civil Engineering, University of 
Bradford, Bradford, BD7 1DP, UK 

INTRODUCTION 



Quality can only be considered when both the purpose of education and the course design 
features associated with its attainment have been absorbed. This paper considers both 
elements with regard to engineering education at degree bvel. The course features are 
considered from the perspective of what the students should be experiencing rather than 
how the academic is performing. 

Quality of higher education is commonly expressed in terms of the teacher's 
performance. End of course questionnaires handed out to students often require comment 
upon the clarity of overhead projection transparencies, quality of handouts and whether 
lectures started and finished on time. This seems to miss the point somewhat; after all, 
learning is something which learners do. rather than something which is done to them. 
It would be more productive, if more difficult, to assess how well our efforts helped the 
students to learn. 

We shall take the definition of "quality" to be fitness for purpose. This begs the 
question as to what is the purpose of an engineering education? A universal answer is 
that we aim to improve the quality of thinking of our students. This has been elaborated 
(Hawkins et al 1965), within an engineering context, by the American Society for 
Engineering Education (ASEE) to be: 

• mastery of the fundamental scientific principles and a command of the basic 
knowledge underlying a branch of engineering, and 

♦ thorough understanding of the engineering method and elementary 
competence in its application. 

This paper explores two questions: 

1 What are the components of the "engineering method"? 

2 How would students recognise that they were on a quality course which 
allowed them to develop skills in using the engineering method? 

It is not our intention to discuss different teaching methodologies but rather their 
characteristics by which quality may be judged. 



WHAT ARE THE COMPONENTS OF THE ENGINEERING 
METHOD? 



In engineering we wish students to recognise that there is not a single correct solution to 
most problems. Hence, participants at the workshop defined the engineering method as 
a process of design to produce a workable solution within constraints imposed by society. 
Although intellectual skills are involved it is important to recognise that the products of 
engineering are implemented through people. Hence, two sets of skills are required. 
Intellectual skills of creativity, analysis, synthesis and evaluation allow understanding 




QUALITY IN ENGINEERING EDUCATION 



121 



and manipulation of knowledge. Problems may thus be solved and effective decisions 
made. In doing so, deficiencies of knowledge and understanding will be recognised and 
strategies to overcome them implemented. 

Engineers work in teams and require an awareness of group processes and a repertoire 
of appropriate skills. Harrisberger (1984) claims the working day comprises 80%" -ings", 
namely planning, communicating, listening, persuading, discussing, informing, leading, 
delegating, managing etc. The various intellectual skills may also be included in this 
grouping. It is also important that young engineers develop a self- awareness of their 
strengths and weaknesses. 

The remaining part of the day is taken up with the "-ics" - physics, hydraulics, 
electronics, mathematics etc. These form the first of the two ASEE goals. Whilst higher 
education may not feel that an 80/20 split is appropriate for degree courses, it is 
undeniable that the current balance is heavily weighted the other way. 

What is immediately apparent is that the passive receipt of information from a lecture 
is inadequate to develop skills in the engineering method. The University of Bradford's 
mission statement is "Making Knowledge Work". This implies movement and effort. 
Thus, students must be actively involved in their own education. It is also apparent that 
a reductionist view of engineering is inappropriate. Engineering is a complex whole with 
which students must be confronted, not only within a single topic but also across the 
complete discipline. 

HOW WOULD STUDENTS RECOGNISE QUALITY? 

If we wish to improve students* thinking it is first necessary to ask how they approach 
their learning. Research suggests that students adopt either a surface or a deep approach 
to learning. This is largely a conscious intention dictated by circumstances. The surface 
approach is characterised by focussing on the details and information in a learning 
environment. Learning is by memorising facts in a linear manner with no interpretation. 
In contrast, students adopting a deep approach focus on the meaning of the information 
presented to them. They will relate facts, possibly to their own experiences, to develop 
a coherent picture. It is obviously the latter approach which we should encourage and 
which is implicit in our definition of the engineering method. It is unfortunate to note 
the evidence suggests that it is the surface approach which is all too common in HE 
(Gibbs 1992). 

Four characteristics of course design and delivery have been identified which arc 
associated with the deep approach to learning: 

• learner activity 

• student interaction 

• a well structured knowledge base 

• motivational context 

We have already identified that the engineering method requires active involvement in 
the learning process. Activity is not enough; it must be planned and time allowed for 
reflection upon the learning so that it may be integrated with existing concepts, 
knowledge and experience. One of the key features which drives students to the surface 
approach is a heavy workload which does not allow the time for reflection. This must 
be resisted, whatever the demands from professional institutions for more and more 
content. Proficiency in engineering comes from method or process, not facts. 
Not only is the active involvement of other students necessary for the "-ings" but also 



128 



122 



QUALITY IN COURSES 



for clarifying concepts. Learning is more profound if the subject matter is discussed, 
chewed over or presented. After all, the best way to learn something is to have to teach 
it. There are profound implications for the architecture of HE institutions. With the 
dominant delivery mechanism in engineering degrees being the lecture we are reasonably 
equipped with raked lecture theatres. These are of little use for small group discussions 
etc. Flat floored rooms with moveable furniture, socialising areas and the like will 
promote human interaction. Arrangements for laboratory classes need to flexible, and the 
technical support staff kept informed, so that the students have access when required, not 
when timetabled. Library staff need to be kept informed and involved in developments 
so that they can ensure that the necessary support is available. Similarly, support for 
computer hardware and software needs to be available to support the learning. 

The engineering method requires an holistic approach to design which also aids the deep 
approach to learning. However, the struclure of knowledge and its inter-relationship with 
other knowledge and concepts must be clear to students. We have found that problems 
directly imported from industry, whilst true to life, can contain ambiguities or avenues 
which detract from the required learning experience. The use of such problems provides 
the context in which learning is motivated. Students recognise a need to know for 
themselves and experience ownership of the learning. It is incumbent on the academics 
to provide an open and supportive environment for the students to experiment, question, 
make mistakes and develop. 

High level skills take time to develop. Thus, high class contact and excessive course 
material should be avoided to create the appropriate learning environment for the 
engineering method to be refined. Additionally, a lack of choice of subjects learned and 
little opportunity to study a subject in depth reinforce the tendency to adopt the surface 
rather than deep approach. 

These features should be associated with the complete course from start to finish. The 
temptation to say "Let's get the basics into them in the first two years and then ease off 
in the final year and do all this experimental stuff" has to be avoided. Students have an 
open mind as to what HE is when they first walk through the door. That is the time to 
set the academic agenda, which, once set, will become the accepted norm. Confusion and 
anxiety will be the result of changing for the last year and a surface approach will be the 
inevitable result. 



We have identified several fe^ures of degree courses which should be in place in order 
for students to adopt both a deep approach to learning and follow the engineering method. 
Indeed, the two are inextricably linked. A quality course should be judged against these 
criteria rather than simplistic measures of lecturing performance. 

References 

Gibbs G (1992) Improving the Quality of Student Learning Technical and Educational 
Services Ltd Bristol 

Harrisberger L (1984) Curricula and Teaching Methods in Engineering Education 
Education for the Professions Nelson Walton-on-Thames 

Hawkins G A, Pettit J M and Walker E A (1965) Goals of Engineering Education - A 
preliminary report American Society for Engineering Education Washington DC 



CONCLUSIONS 




Section 4: Quality in Learning and 
Assessment 



The theme that runs through all the papers in this section is that the quality of learning 
and assessment is enhanced when learners themselves are taken seriously: when they are 
helped to understand how they learn, when their preferences for and reactions to different 
learning strategies are acknowledged, when they are empowered to take ownership of 
their own learning and assessment, when they are seen as partners in evaluating the 
quality of learning outcomes. Among the papers in this section are two workshop reports 
which describe the activities completed, activities which were designed to draw on the 
collective experience of participants in generating useful insights and products. 

Dave Muller and Peter Funnell argue in their keynote paper, "Exploring learners* 
perceptions of quality", that quality requires the learner to be at the centre of both the 
learning process and the process by which such learning is evaluated. They make a 
compelling case for obtaining regular qualitative feedback from learners, indicate how this 
process is closer to the tenets of total quality management and can have an immediate 
impact on the quality of individual learning, and identify implications for teaching and 
learning strategies. Although the context is vocational education and training, their 
insights have general applicability in the search for quality in learning and assessment. 

"Concept mapping, post-questioning and feedback in distance education", by Robert M 
Bernard, Som Naidu and Karin Lundgrcn-Cayrol, presents the findings of a field 
experiment to compare the strengths and weaknesses of mathemagenic (eg. post- 
questioning) versus generative (eg. concept mapping) instructional strategies. A 
significant difference in cognitive gains by nursing students who completed at least 11 
of 1 2 concept mapping exercises was found. It was also found that feedback is a key 
element for instructional gain in both strategies. The paper also discusses students' 
preferences for and reactions to different learning strategies. 

The uext two papers address different but complementary themes in the context of 
workplace-based learning. Caroline Bucklow in "Comparing chalk and cheese - quality 
in assessing work-based learning", outlines a workshop designed to enable participants 
to identify priority issues in, and build quality into, the process of assessing students' 
workplace-based learning. The workshop is written up in a form that enable? it to be 
replicated as a staff development activity for course teams. "Using learning contracts to 
enhance the quality of work-based learning", by Iain Marshall and Margaret Mill, 
describes a strategy for involving learners in negotiating learning plans and goals that are 
responsive to their needs, and also in negotiating what is to count as evidence that 
learning outcomes have been achieved. They also show that the accreditation of 
workplace-based learning can be cost effective in terms of staff time. 

In "Quality of assessment", Phil Race identifies four main processes of learning and 
shows how traditional assessment methods are at cross purposes with them. He discusses 
ten "worries" about assessment, and makes suggestions for avoiding the problems 
associated with each of them. Practical ideas for improving assessment are illustrated, 
with emphasis on self- and peer- assessment strategies which enable learners to take 
ownership of and learn through the process. In "BS5750 for assessment" he presents the 
products of the workshop associated with his paper, in which participants consider the 
fitness for purpose of different forms of assessment, review their own experiences of 
being assessed, and "lay down the law" by identifying criteria for quality in assessment. 



130 



21 Exploring learners' perceptions of 
quality 



Dave Mutter and Peter Funnell, Suffolk College, Ipswich, IP4 lLT t UK 
INTRODUCTION 

In response to the needs of external funding agencies and in order to provide information 
for managers of education and training institutions, considerable emphisis has been 
placed in recent years on the collection of quantitative learner feedback m the form of 
questionnaires. This data has often been used to support course evaluation processes and 
contribute to the creation of institutional performance indicators. It would not be 
unreasonable to suggest that those of us with responsibility for the management and 
delivery of vocational education and training have been subject to much tighter control 
than those engaged in academic education and research or the delivery of the traditional 
sixth form work. The Department of Employment, in particular, has placed emphasis 
upon data driven planning and the evaluation of the effectiveness of vocational education 
and training through the use of employer and learner feedback using questionnaire 
methods. 

However, in exploring "quality" in the context of vocational education and training, we 
have found ourselves focusing on qualitative issues. In our first work in this field 
(Muller and Funnell 199la) we explored quality from the perspective of learners, 
employers and as a managerial issue. We became committed to "total quality 
management" as referred to by leaders in the field (such as Oakland 1989a, 1989b) both 
as a concept and as a form of operational practice. In subsequent papers (Muller and 
Funnell 1991b, 1992a 1992b) a total quality approach to managing vocational education 
and training was put forward emphasising the centrality of the learner in the learning 
process. 

Nevertheless, it has become clear to us that a managerial commitment, however 
important it is, is not the sole prerequisite to a move towards total quality management. 
Indeed we now believe that to be faithful to the major tenets of the total quality approach 
we need to consider the extent to which we are really close to our customers. Put simply 
it now seems to us that in following the total quality approach we need not only to 
commit ourselves as educational and training managers to a customer orientated approach 
but also recognise that we need also to explore the perceptions of our customers through 
their own eyes. This paper sets a framework from within which we seek to achieve this. 

WHAT IS QUALITY? 

There are clearly multiple and varied definitions of quality as applied to vocational 
education and training and multiple definers of the concept. We have sought to argue 
against a competence based approach which suggests that quality is delivering a product 
which is "fit for purpose". Rather, we have explored quality in terms of a wider range 
of factors leading to a notion of "value-addedness". We consider it important that the 
notion of quality be defined in terms of the longer term impact vocational education and 
training will have on the learner. In exploring quality from this perspective we have 
argued that the role of educational providers is to ensure: 

"that learners fully participate in, and contribute to, the learning process in such a way that 
they become responsible for creating, delivering and evaluating the product." (Muller and 
Funnell 1991a, p 175) 



131 



126 



QUALITY IN LEARNING AND ASSESSMENT 



Following a detailed review of the literature and the exploration of a number of case 
studies, we have proposed five defining criteria which contribute to the delivery of quality 
vocational education (Miiller and Funnell 1991a). All of these focus specifically on the 
learner and the role which he or she plays in determining the product: 

a) quality recognises the central ity of the learner in obtaining successful learning 
outcomes; 

b) to some extent the learner must take ownership of the learning process; 

c) the learner has some responsibility for determining the style and mode of delivery of 
learning; 

d) learners participate outside the formal learning context to the process of learning; 

e) learners innovate, experiment and learn from failure as well as success, a notion 
which challenges the concept of "zero defect" prevalent in total quality 
management. 

The conclusions we have come to have been derived from a managerial focused approach 
to investigating quality. The case studies we have employed in our own work: have 
tended to reflect different approaches to managing the delivery of education and training 
and highlight good educational practice. However it was left to Barnett (1991) to 
articulate more precisely the limitations of numerical performance indicators and the need 
to place the learner at the centre of the educational institution as a whole. He proposed: 

"What we need, in short, are indicators that a concern for the quality of student 'learning 
is built into the internal life of the institution. In the previous section, I gave some 
examples of what these might be. If we find an institution which has iastituted learning 
weeks, has a staff development unit concerned about student learning and the student 
experience, gives tangible rewards for high quality teaching, clearly looks to evidence of 
the commitment to teaching in its promotions procedures, picks out examples of good 
teaching in its newsletter, and encourages staff to explore teaching and learning intentions 
within its course review procedures, then we have grounds for believing that here is an 
institution in which the character of the student experience really matters and is not just 
a piece of rhetoric" (p 50). 

There remains, however, a need to go beyond this and to recognise that organisations 
through managerial approaches alone cannot place the learner at the heart of the process 
without ensuring that the tutor has fundamental devolved responsibility for creating the 
product in partnership with the learner. We now believe that it is essential to place the 
learner at the centre of the evaluation of learning processes and outcomes by focusing on 
qualitative approaches to exploring learners* perceptions of quality. To achieve this the 
responsibility for evaluation must be built in to the learning process itself and be owned, 
in partnership, by learners and tutors. The rest of this paper will explore some first steps 
in fulfilling this commitment to identify and act upon learners * perceptions of quality. 



To date the major rationale for exploring and evaluating quality from the perspective of 
learners has been to provide information for curriculum managers and those with broader 
institutional responsibilities. This has also included external validating bodies and in 
particular the Council for National Academic Awards (CNAA) which has placed 
particular emphasis on the course review process. In further education, those external 
agencies with Responsibility for funding have required the collation of student feedback 



EVALUATING QUALITY 




LEARNERS* PERCEPTIONS OF QUALITY 



127 



data, which impacted more upon the political context within which vocational education 
was funded rather than the actual experience of the learner. Nevertheless, thete have 
been a number of well tried and tested approaches to evaluating quality all of which have 
been primarily concerned with providing information to be used by others in designing 
the curriculum and in the planning of vocational education. It is notable that a great deal 
of the data collected has been retrospective and has been of greater benefit to future 
generations of learners rather than current learners. 

Quantitative Approaches 

The questionnaire approach has long been dominant in vocational education and training 
as the means of obtaining feedback from learners. There are many quite sophisticated 
questionnaires developed both in the UK and elsewhere (Bedford LEA 1989; County of 
Avon 1989; and Thomson 1988). The basis for quantifying learners* perceptions of 
quality is usually through some means of scaling. For example, a paper by Okun et al 
(1986) reported on the development of a psychometric approach to investigating USA 
College students' affective reactions to their academic lives. Students were asked how 
they feel about major areas such as the classes they are taking, what they are learning, 
their instructors and what is available to them in the way of services, guidance and 
assistance. The response options were "delighted, pleased, mostly satisfied, mixed (both 
equally satisfied and dissatisfied), mostly dissatisfied, unhappy and terrible." Data of this 
kind can be valuable in that it offers quantitative feedback to tutors whilst collectively 
providing an overall evaluation of the learning environment. 

The need to collate feedback gathered through questionnaires has led to a number of 
highly sophisticated commercial systems coming on the market. ScanTech Systems Ltd, 
Dunstable, UK, are promoting a means of collecting and collating questionnaire responses 
which captures data and processes it for presentation in graphical and tabular formats. 
The information provided is of high standard and is readily available to those with front- 
line curriculum responsibility as well as to institutional managers. The kinds of areas 
explored are not dissimilar to other questionnaire approaches. For example, students are 
asked for their views on college facilities such as buildings and the refectory and on 
course/programme content and teaching, and response options are very good, good, below 
standard and poor. The use of such systems can be invaluable if the major emphasis is 
to be placed upon generalised data and a broader approach to evaluating learners ' 
perceptions of quality. However, it is important to recognise that learners are not in 
ownership of this information nor is it built in as an integrated part of the learning 
process. 

Qualitative Approaches 

Simultaneously with the prepaiation of this paper, the proceedings of a recently convened 
CNAA conference on "Evaluating the Quality of the Student Experience" were published 
(Burrows 1992). A major theme emerging from the conference has been the need to 
place the student more at the centre of the evaluation process. A number of the papers 
argued that giving students more autonomy and greater responsibility within a quality 
assurance system must be seen as part of the educational process itself, and that greater 
emphasis should be placed on the need to develop a dialogue between students and tutors. 
Clearly in this context the quantitative, questionnaire derived, approach fails to place 
ownership with the learner or create a dialogue which is in itself customer orientated. 
Interestingly whilst a number of the papers presented explored qualitative rather than 



133 



128 



QUALITY IN LEARNING AND ASSESSMENT 



quantitative approaches to gaining feedback (eg. Baldwin and Carter 1992) it was a 
student contribution which argued the case for building self and course assessment into 
the programme of study itself (Rostron 1992). 

In our view qualitative approaches to learners' perceptions of quality are closer to the 
basic principles of total quality management and should have a greater impact upon the 
teaching and learning process. From a managerial perspective the total quality movement 
places particular emphasis on getting as close as possible to the customer and on building 
quality in to the production process rather than excluding deficient products at the end. 
If we apply this to vocational education and training, and recognise that the learner is 
both customer and product, the need to find means of building quality into the learning 
process becomes self evident. 

Those closest to learners are either fellow learners (eg. those following the same 
programme of study) or those with teaching responsibilities. In our view qualitative 
approaches need to bring together learners and tutors in such a way that their explorations 
of the learning process can have an immediate impact upon the curriculum. At the heart 
of vocational education and training lies individuals with shared responsibility for 
delivering a quality product in terms of a qualified learner outcome and a well developed 
curriculum process. The encouragement of learners to share with tutors the ways in 
which the process is developing can be seen to be of greater value than the retrospective 
recording of quantitative data which may or may not be acted on by managers. 
believe that it is necessary to give greater currency to the constructive dialogues which 
learners and tutors have and to build this into the process of learning through devolving 
responsibility to those who have the knowledge to transform the product at its formative 
stage. 



IMPLICATIONS FOR TEACHING AND LEARNING 
STRATEGIES 



In placing greater emphasis on regular qualitative feedback the process of teaching and 
learning is likely to become influenced in different ways. There are immediate 
implications both for learners and their tutors in coping with the responsibility of 
managing their own curriculum. Many of these issues need greater exploration and it is 
our intention to confront some of these during the next phase of our own work, which 
will be an empirical investigation of learners' perceptions of quality. However, it is 
important to identify some of these issues and challenges at this stage: 

a) A major challenge will be the devising of means to support and teach learners how 
to evaluate quality. This will require both objective explorations of learners* 
perceptions of quality as well as an analytical approach to curriculum design by 
tutors. 

b) Learners will need support in understanding their own role in constructing the 
products of vocational education and training. This is a particularly difficult concept 
to grasp in the context of a customer purchasing a product from a provider. Rarely, 
as purchasers, do customers have to participate in creating the product they are 
buying. 

c) Evaluation will need to become part of the teaching and learning programme. By this 
means qualitative evaluation will become integral to the delivery of the curriculum 
and will not be simply an adjunct to it. This too will challenge many existing 
assumptions about the nature of curriculum design and the resource allocated to 
traditional teaching. 




LEARNERS' PERCEPTIONS OF QUALITY 



129 



d) There will be a need to formalise in a constructive, and non- threatening and non- 
bureaucratic, manner the ways in which learners and tutors talk about the curriculum 
and learning processes. It is a common assumption that good interpersonal 
relationships between tutors and learners is an insufficient condition for quality 
assurance procedures. In our view this assumption needs to be challenged. Indeed 
in the process of challenging this assumption we may find that for too long we have 
ignored good practice in this area. 

e) There may emerge a need to give greater importance to peer review. Learners are 
often the best judge of the curriculum and its delivery, and if engaged not only in the 
process of comment but in taking action, are likely to recognise issues and to respond 
positively to them. 

f) Finally, qualitative evaluation without action is of no value. We will need to explore 
and encourage learners and tutors to evolve informed strategies and to take actions. 
Responsibility through constructive conversations will be shared and should lead to 
a more sophisticated approach to the delivery of quality in vocational education and 
training. 

THE CHALLENGE - REALISM OR RHETORIC? 

Concerns have been expressed that the "language" of qualitative approaches to evaluation 
is both derived from, and only appropriate to, higher education. We feel uncomfortable 
with this argument. Clearly it is important to recognise the different needs, and skill and 
experience bases, of the consumers of vocational education and training. However it is 
our view that all learners can be empowered to take responsibility for their own learning. 
Indeed there are excellent examples of this operating in the compulsory education system 
where one of the major strengths of the educational process is the partnership which has 
emerged between parents, pupils and their teachers. 

Clearly there is an idealism here. However at a practical level few of us with 
ixperience of working in vocational education and training are unable to engage young 
trainees or returning adults in conversations about their hopes, plans and current learning 
experiences. Increasingly, the narrow views of teaching specific skills which lead to 
competence without capability, is coming under critical attack (Marshall 1991). Few of 
us would disagree that we want learners who are both competent and capable and, 
whatever the teaching methods employed, that we need to develop entrepreneurial and 
leadership skills. All work situations in the end require the individual to make decisions. 
The starting point is placing some ownership for learning with all of those fully- 
integrated and engaged with the educational system. What can be forgotten is that the 
young trainee of today is the mature learner of tomorrow, complete with family and 
community responsibilities. Life teaches us how to make decisions and to own 
challenges, opportunities and problems. We must be wary as educators of failing to 
recognise the power of the education process itself in supporting this. 

THE NEXT STAGE 

As part of our own process of learning and teaching about quality we are now planning 
to mount a series of empirical studies into qualitative approaches to investigating learners* 
perceptions of quality. It is our intention to collect data and to work with colleagues in 
the more traditional areas of further and vocational education. We have plans to try and 
compare both craft and technician courses as well as mature and younger learners. As 
part of this empirical exercise we wish to bring together the learner and the tutor with a 



135 



130 



QUALITY IN LEARNING AND ASSESSMENT 



view to empowering both partners not only to evaluate quality but to improve it. We also 
hope to demonstrate that the emphasis we have placed on learners owning the product is 
appropriate and in doing so to be able to support the tenets of the total quality approach 
to vocational education and training. In summary, we believe that: 

a) it is important for tutors to get close to the learner and hence the product; 

b) for quality to be built in to this process; arid 

c) to encourage the learner and the tutor to take responsibility for constant 
improvement. 

This will legitimise the place of the learner at the centre of the evaluation process, and 
is in line with the theoretical principles derived from the total quality approach. As such 
it places responsibility on learners and their tutors for identifying the factors which 
determine quality and for acting upon them. 

References 

Baldwin J and Carter B (1992) Workshops and Direct Comment, an Alternative to Staff 
Questionnaires in Burrows A (ed) Evaluating the Quality of the Student Experience 
CNAA London 

Barnett R (1991) Delivering Quality in Student Learning in Miiller D J and Funnell P 
Delivering Quality in Vocational Education Kogan Page London 

Bedford LEA (1989) The Bedfordshire Quality Project An Evaluation Pack for Use by 
Course Teams Bedford LEA Bedford 

Burrows A (1992) (ed) Evaluating the Quality of the Student Experience CNAA 
London 

County of Avon (1989) County of Avon: Ensuring Quality Course Review Profile 
County of Avon Avon 

Marshall K (1991) NVQ's: An Assessment of the "Outcomes" Approach to Education 
and Training Journal of Further and Higher Education 15 pp 56-64 

Miiller D J and Funnell P (1991a) Delivering Quality in Vocational Education Kogan 
Page London 

Miiller D J and Funnell P (1991b) Realising Potential Through Quality Vocational 
Education in Winterburn R (ed) Realising Human Potential Kogan Page London 

Miiller D J and Funnell P (1992a) Initiating Change in Further and Vocational 
Education: the quality approach Journal of Further and Higher Education 

Miiller D J and Funnell P (1992b) An Exploration of the Concept of Quality in 
Vocational Education and Training Education and Training Technology International 

Oakland D N (1989a) (ed) Total Quality Management IFS Bedford 

Oakland D N (1989b) Total Quality Management Heinemann Oxford 



13G 



LEARNERS 1 PERCEPTIONS OF QUALITY 



131 



Okun M A, Kardash C A, Stock W A, Sandler I N and Bauman D J (1986) Measuring 
Perceptions of the Quality of Academic Life Among College Students Journal of College 
Student Peisonnel 27 pp 447-451 

Rostron J (1992) "Consumer Opinion" or "Action Research"? An evaluation student 
speaks out in Burrows A (Ed) Evaluating the Quality of the Student Experience CNAA 
London 

Thomson C (1988) (Ed) Client Satisfaction: Monitoring Quality Further Education Staff 
College Bristol 



137 



22 Concept mapping, post-questioning and 
feedback in distance education 



Robert M Bernard, Som Naidu and Karin Lundgren-Cayrol, Department of 
Education, Concordia University, W Montreal, PQ, Canada 



The aim of this paper is to disseminate findings from a field experiment in a Distance 
Education setting, where the main goal was to compare mathemagenic and generative 
instructional strategies for cognitive gain on both a short term and a long term basis. A 
secondary goal was to look at students' preferences of learning style, and willingness to 
acquire a new strategy. Mathemagenic and generative learning strategies are here defined 
according to Rothkop (1970) and Jonassen (1985). Mathemagenic strategies refer to 
instructor imposed activities which direct the learner to the basic content of the text (ie. 
inserted or post-questioning techniques, behavioral objectives, advance organizers etc.) 
as opposed to generative learning strategies, where the learner creates his/her own 
representation of the content (eg. concept-mapping, note-taking, elaborations etc.). 

It has been argued that mathemagenic devices are efficient time-wise in guiding the 
student to specified content (Hamaker 1986). However, it has also been revealed that 
students in distance education settings appear to skip these devices, unless they see an 
immediate benefit. Therefore, this method might be less effective in terms of recall on 
a long term basis (Clyde et al 1983; Marland et 'A 1990). 

On the other hand, generative learning strateg es, such as concept mapping and note- 
taking, are aimed at activating students to build on their previous knowledge and 
construct a meaningful graphical and/or conceptual representation of the material to be 
learned. Essentially, generative methods activate intellectual capabilities, such as the 
ability to make observations through the use of inference (ie. to synthesize, predict and 
explain events), thus moving beyond rote-learning and hopefully promoting meaningful 
and reflective learning (Ausubel, discussed in Novak and Gowin 1984). 

A survey of the literature provided the following points as guidelines for the hypotheses 
tested in this study: 

• although attrition is usually high, some students tend to leam generative strategies 
with more ease than others; they either do not have a workable strategy (Novak 
and Gowin 1984; Clark 1990; Dean and Kulhavey 1981) or they have a preference 
for visual aids in learning (Feldsine 1987; Okebukola and Jegede 1988); 

• students being presented with mathemagenic strategies tend to leam basic content 
as well as students being presented with generative strategies (Novak and Gowin 
1984; Miccinati 1988); 

• students using generative learning strategies tend to do better on conceptually 
hierarchical content (eg. biology) once the strategy is well practised, than students 
being presented with mathemagenic learning devices (Moreira 1977; Dean and 
Kulhavey 1981; Novak 1990; Feldsine 1987; Schmid & Telaro 1990). 



INTRODUCTION 




CONCEPT MAPPING 



133 



METHOD 

Design, Subjects and Procedure 

This study was carried out through the Department of Nursing at St Xavier University 
in Antigonish, Nova Scotia, giving Nursing 200: Community Mental Health as a distance 
education course in a four year programme. The participants were 141 female nursing 
students in an obligatory 20 week distance education course, consisting of weekly 
assignments, a midterm and a final exam. 

The midterm and final exams were the dependent measures (DV). Both dependent 
measures were separated into a set of multiple choice questions, and a set of essay 
questions broken down into key-idea and related-idea units, both including mapping 
versus non -mapping items. This technique produced five midterm and five final measures 
- 10 dependent measures in all. 

Treatment groups were assigned to geographical clusters, keeping the largest group for 
the concept-mapping condition, because of the expected attrition from this condition. 
Distribution of subjects was as follows: the control group having post-questions without 
feedback (n = 22); the post-questions with feedback condition (n = 36); the combined 
group (questions and mapping exercises) (n = 32); and the concept mapping with 
feedback condition (n = 50). The concept-mapping group naturally separated itself into 
high persistent mappers (n - 22; M=l 1 maps; SD=1.5) and low persistent mappers (n = 
28; M=3.5 maps, SD=1.6). This bifurcation was used for statistical analyses and is 
discussed in depth elsewhere (Bernard and Naidu 1992). 

Materials 

All students received the same instructional package, with an additional study guide that 
differed according to the instructional strategy employed for that treatment condition. 

Control Group 

The control group received self-assessed tests with no feedback at the end of each 
module, which was the normal procedure. 

Post-Questions with Feedback 

This group received the same self-assessed tests as the control group, but got feedback 
on 12 of the 20 occasions. 

The Combined Group 

This group received the same training guide as the concept mapping group and the same 
self-assessed tests as the two other groups. They were recommended to do 6 maps with 
feedback and 6 self- assessed tests with feedback. 

Concept Mapping 

A training guide on concept mapping was provided. Students were encouraged to 



139 



134 



QUALITY IN LEARNING AND ASSESSMENT 



construct 12 maps during the course of the treatment, and received feedback on each map 
completed. 

Attitude Questionnaires 

All students received a questionnaire evoking their views of the learning experience, as 
well as a 5-point Likert type questionnaire consisting of 15 statements eliciting 
information about their predominant style of studying. 

RESULTS 

Group Equivalence and Covariate 

Group equivalence, including the two concept-mapping groups, was established through 
an analysis of variance using a combined score from four previous exams (F(4,135)=.89, 
p >.05). The covariate was found significant and was used to adjust the means for the 
10 cognitive measures, both at the midterm and in the final exam. All scores were 
converted into z-scores. 

Cognitive Measures: Midterm and Final Exams 

The omnibus MANCOVA produced a significant main effect for treatment, 
F(40,494)=2.13, p<0.001. The Roy-Bargmann stepdown approach was chosen, since the 
sequence of the tests were of importance (Bernard and Naidu 1992). This approach 
produced four significant variables. At midterm, multiple choice items F(4, 134) =2.90, 
p<.05) and key non-mapping items, F(4 ,131)=2.82, p=.G05 were significant. In the final 
exam, key-mapping items F(4,128)=3.97, p<,01 and key non-mapping items 
F(4,126)=2.98, p=.02 were significant. 

Post- hoc analyses were conducted to determine which groups differed and these results 
are shown in Figure 22.1. Interestingly, the high persistent concept-mapping group 
performed significantly lower than all other groups on multiple choice items at midterm. 
However, in the final exam this difference disappeared. This phenomenon is supported 
in the literature and could partly be explained by the fact that students find it hard to 
learn a new strategy, and therefore struggle with the strategy more than the content. 

It is worth noting that the post-questions with feedback group performed better than the 
control group, suggesting that feedback is a beneficial element with both mathemagenic 
and generative strategies. 

Furthermore, a significant difference was revealed in favour of the high persistent 
concept mappers at both midterm and final exams on key non-mapping items. This 
suggests that even for non-mapping items these students were enabled to relate more 
information correctly than students in the other groups. For the key-mapping idea units, 
the high persistent mappers produced better results than all other groups both at midterm 
and in the final exam. This implies that recall of main ideas is learned more thoroughly 
with concept mapping, maybe because students elaborate and spend more time on the 
material, than when using mathemagenic strategies. Moreover, the combined group 
outperformed the control group on key mapping items in the final exam, further 
confirming the idea that concept mapping nelps recall for these types of cognitive 
measures. 



140 



CONCEPT MAPPING 



135 



Measures 



Kay-Non (F) 
Qr3 > Grl/445 



Qr4 > Grl 
KtyMap (F) 

Gr3> Gr4* Gr1 



Below the mean 



Key NonWap(M) 

Gr3> Gr4,5 A Grl 



Mult. Ch.(M) 
Gr3< ali groups 



Above the mean 




-1 OA -0.6 -0.4 -0.2 



FINAL 

CoflM Group 1 
UwCU Group 2 
HghCU Qrou P 3 
****** Group 4 
GrOcp5 



MIDTERM 



0.2 0.4 0.6 0.8 1 



Z-Scor»s (M410; SD-+-1 .0 



Figure 22.1 Significant results from the cognitive measures at midterm and final 
Attitudes after Experience 

The factor analysis on the study method questionnaire included six factors that together 
explained 64.8% of the variance. The variance accounted for by each of the six factors 
is shown in Figure 22.2. 

Factor scores (regression approach) were saved and a follow-up analysis revealed a 
significant main effect for factors, Hotellings F(20,454)=2.02, p<.01 . Univariate contrasts 
indicated that this difference resides in attitudes between the low and high persister 
mapping groups (t=-2.79, p<.01) on Factor 2. None of the other factors or groups 
produced any significant results. Figure 22.3 demonstrates these findings. 

Six questions were constructed to elicit information from all students about their 
attitudes towards self-assessed tests, since this is a mathemagenic device often used in DE 
as well as traditional settings. No significant differences were found among groups. 
However, it is interesting to note that 94% reported to have done most or all of these 
exercises. Most (83%) verified all the answers, however only 63% reported that they 
always reviewed incorrect answers, suggesting that post-questions are not always used as 
intended by the instructional designer. Moreover. 48% of students found them difficult 
in the beginning compared to 25% at the end. Only 36% found them time-consuming 
and 67% found them useful to help them study. 



136 



QUALITY IN LEARNING AND ASSESSMENT 



<**) 




Pi*) 





Fwtarl: 


■ 




□ 








□ 


F**ort: 


m 









Factors (**•»%> 

DWculty ramembertno; facts and (totalis 
Preference for visual repraseotaioo of content 

Ukos to understand content thourougty 

Preference for k»y concepts 

Ukas no la taking and memorization 
Preference tor existing order of presontafon. 



Figure 22.2 Variance explained by attitude factors about study methods 



Factor s cor 9 (M-0;3d 

07 -j 




Facti 



Fact.2 



Fact.3 



FacU 



Fact.5 



0 



* significant differences 



Groups 
Control 
H Low CM 
| | High CM 



Factors 



Fifture 22.3 Significant difference between high and low mappers: preference for 
visual aids in education (factor 2) 



142 



CONCEPT MAPPING 



137 



High and Low Mappers 

As shown in Figure 22.3 low persistent mappers appear more negatively inclined towards 
visual and graphical aids in learning new materials. 

A closer look at these responses revealed that 65% of the high persisters found maps 
provided by the instructor useful whereas only 45% of the low persisters thought so. 
When asked how difficult concept-mapping was at the beginning of the experiment 62% 
of the low persisters thought it was very difficult compared to 35% of the high persisters. 
By the end of the experiment only 20% of the high persisters still reported it difficult, 
with an encouraging 45% reporting concept mapping as quite easy. 

The questionnaire also indicated that most low persisters found the exercises "very 
difficult to do" (62%), "very time-consuming" (83%) and "not very useful"(75%), 
whereas the high persisters still found them "difficult" (35%) and "time-consuming" 
(45%). However, 50% of the high persisters reported that they thought of them as a 
useful aid in learning new material compared to only 25% of the low persisters. When 
asked whether they mostly preferred well tried out approaches to learning, 46% of the 
low persisters agreed to this compared with 30% of the high persisters, indicating a 
higher probability of willingness to acquire a new learning strategy. 



Concept Mapping Group: High Low 

Questions 



Maps by instructor are helpful 65% 45% 

Mapping very difficult in beginning of experiment 15% 50% 

Mapping easy in end of experiment 45% - 

Mapping is time-consuming 45% 83% 

Mapping is useful when learning new material 50% 25% 

I mostly prefer well tried out approaches to learning 30% 46% 



Table 22.1 Summary of attitude questions 

DISCUSSION 

Perhaps the most important finding in this field experiment is that it is possible to teach 
distant learners generative learning strategies that are beneficial for cognitive outcomes 
on a long term basis. It also showed that mathemagenic and generative strategies are 
beneficial for different types of outcomes, since high concept-mappers performed better 
at key-idea items, whether they mapped them or not, than all the other groups. No 
differences were found for multiple choice items at the final exam, suggesting that factual 
or declarative knowledge can be gained by any strategy, but conceptual and procedural 
knowledge such as that found in the sciences might be enhanced by concept-mapping as 
a visual aid in learning (Moreira 1977; Feldsine 1987; Novak 1990; Schmid & Telaro 
1990). In this study the mapping exercises were matched to elements in the course that 
lend themselves to concept-mapping (ie. conceptually hierarchical knowledge as opposed 
to declarative or fact remembering knowledge, Beukhof 1990). It could be speculated 
that course content in general consists of both types of knowledge. Its implication for 
material design might be to analyze content in order to pinpoint these differences. 



143 



138 



QUALITY IN LEARNING AND ASSESSMENT 



The other outcome confirms what the literature says (Clark 1990), that all learners do 
not want to or can not adopt a self- activating, generative strategy. This conclusion is 
supported by the high attrition rate of 56% in the concept-mapping condition as compared 
with the mathemagenic conditions (post-questions .with feedback and combined group) 
where no attrition at all was noticed. The results from the attitudinal instruments further 
demonstrate that the difference for preferences of study methods seem to lie in the fact 
that some students prefer visual and graphical aids (the high persisters), and a tendency 
by the low mappers to prefer well tried out approaches. All other factors and groups did 
not differ. 

Since the aim was to examine the distance students* reactions to mathemagenic versus 
generative learning strategies, it could be concluded that mathemagenic learning methods 
can adopted by anyone, but that they do not necessarily optimize learning. Beudoin 
(1990) expresses concern over the dependence that mathemagenic strategies create and 
contends that distance learners must be helped to develop learner controlled learning 
strategies. By contrast, a generative learning strategy such as concept-mapping appears 
desirable in this respect, but has drawbacks when the fact that only some learners were 
willing to acquire it, however successfully, is considered. 

F jth in traditional and dis:ance education there is a tendency to view the learner as a 
reactive rather than a proactive participant in the educational process. Expository 
learning, mathemagenics (*othkop 1970), teaching machines/programmed learning 
(Pressey 1927; Skinner 1958 . to mention only a few, are orientations that place emphasis 
on the constructive- interpre* alive role of the teacher-designer. There is little doubt that 
these perspectives are valuable, but they represent only one side of the educational 
equation. The other side argues that empowering students with a set of learning tools (ie. 
study skills and strategies) can provide both short term learning gain and long term 
learning potential (Bruner 1966; Rigney 1978; Weinstein and Meyer 1986). This study 
contributes to the idea that distance educators can design materials that will improve 
learners study skills and strategies. 

References 

Bernard R M and Naidu S (1992) Post-Questioning, Concept Mapping and Feedback: 
a distance education field experiment British Journal of Educational Technology 23(1) 
pp 48-60 

Beudoin M (1990) The Instructor's Changing Role in Distance Education The 
American Journal of Distance Education 4(2) pp 21-29 

Beukhof G (1990) Learner and Program Controlled Text Processing in H Mandle, E 
De Corte, S N Gennett and H F Friedrich Learning and Instruction European Research 
in an International Context (Vol 2:2) Analysis of Complex Skills and Complex 
Knowledge Domains New York Pergamon Press 

Bruner J S (1966) Towards a Theory of Instruction Norton New York 

Clark R E (1990) When Teaching Kills Learning: research on matheinaihantics in H 
Mandle, E De Corte, S N Gennett and H F Friedrich Learning and Instruction European 
Research in an International Context (Vol 2:2) Analysis of Complex Skills and 
Complex Knowledge Domains Pergamon Press New York 




CONCEPT MAPPING 139 

Clyde A , Crother H, Patching W, Putt I and Store R (1983) How Students Use 
Distance Teaching Materials: an institutional study Distance Education 4(1) pp 4-26 

Dean R S and Kulhavey R W (1981) Influence of Spatial Organization in Prose 
Journal of Educational Psychology 73(1) pp 57-64 

Feldsine J E Jr (1987) The Construction of Concepts Maps Facilitates the Learning of 
General College Chemistry: a case study Unpublished doctoral dissertation Department 
of Education Cornell University Ithaca NY 

Hamaker C (1986) The Effects of Adjunct Questions on Prose Learning Review of 
Educational Research 56 pp 212-242 

Jonassen D H (1985) Generative Learning Versus Mathemagenic Control of Text 
Processing in D H Jonassen (Ed.) The Technology of Text Educational Technology 
Englewood Cliffs NJ 

Marland P W, Patching W, Putt I and Putt R (1990) Distance Learners' Interactions 
with Text while Studying Distance Education 11(1) pp 71-91 

Miccinati J L (1988) Mapping the Terrain: connecting reading with academic writing 
Journal of Reading 31(6) pp 542-552 

Moreira M A (1977) An Ausubelian Approach to Physics Instruction: an experiment in 
an introductory college course in electromagnetism Unpublished doctoral Dissertation 
Department of Education Cornell University Ithace NY 

Novak J D (1990) Concept Maps and Vee Diagrams: two meta-cognitive tools to 
facilitate meaningful learning Instructional Science 19 pp 29-52 

Novak J D. and Gowin D B (1984) Learning How to Uarn CUP Cambridge 

Okebukola and Jegede (1988) Cognitive Preference and Learning Mode as Determinants 
of Meaningful Learning through Conceptmapping Science Education 72(4) pp 489-500 

Pressey S L (1927) A Machine for Automatic Teaching of Drill Material School and 
Society 25 pp 549-552 

Rigney J W (1978) Learning Strategics: a theoretical perspective in H F O'Neil Jr (Ed.) 
Learning strategies (pp 165-205) Academic Press New York 

RothkopEZ (1970) The Concept of Mathemagenic Activities Review of Educational 
Research 40 pp 325-336 

Schmid R F and Telaro G (1990) Concept Mapping as an Instructional Strategy for 
High School Biology Journal of Educational Research 84(2) pp 78-85 

Skinner B F (1958) Teaching Machines Science 128 pp 969-977 

Wcinstein C E and Mayer R E (1986) The Teaching of Learning Strategies in M C 
Wittrock (Ed) Handbook of Research on Teaching pp 315-327 MacMillan New York 



145 



23 Comparing chalk and cheese - quality in 
assessing work-based learning 



Caroline Bucklow, Coventry School of Art and Design, Coventry University, CV1 



Assessing industrial placements as an integral part of an undergraduate degree is a 
contentious issue. It is clear that industrial placements provide students with valuable 
experiential learning which complements academic study, but the issues relating to 
assessment of this type of learning are not easy to resolve. What aspects of students' 
learning can we reasonably assess? Who should perform the assessment? What type of 
assessment is appropriate? How do you define quality in assessment and how do you 
ensure it? These are vital issues which affect the significance accorded to assessed 
workplace-based learning, by students and employers alike. 

The process of assessment can be critical in shaping a student's perception of the value 
of workplace-based learning in relation to academic study. Assessment can increase the 
credibility the student accords a range of skills developed through workplace-based 
learning and this can deepen the student's involvement with their own learning process. 
However, the beneficial results are only achievable if all the parties involved are 
committed to maintaining quality, both in the learning experiences offered and in the 
assessment process. 

The workshop described in this paper provides an opportunity to explore issues of 
quality in assessing workplace based learning through the medium of case study. The 
cases are based on our experience in assessing workplace based learning for students on 
the BA (Hons) Technical Communication course at Coventty University. The as*.essment 
scheme used on the course has been developed in the light of research sponsored by the 
Training Agency (as it was) and the CNAA. Participants in the workshop undertake two 
tasks to help develop their own responses to the issues involved in maintaining quality 
when assessing contrasting industrial placements. 



The idea for the workshop grew out of discussions among members of the course team 
for Coventry University's BA (Hons) Technical Communication degree, as they 
developed the assessment scheme for an industrial placement programme. Although the 
team was agreed on the need for objective assessment based on measurable outcomes, it 
was only by engaging in a series of discussions about issues of quality that we were able 
to agree on an assessment process. 

During the discussions it became clear that the assessment process would have to 
accommodate huge differences in the workplace context of individual industrial 
placements and in the expectations different organisations had of the students. 
Additionally, the team recognised that students, academic staff and employers might be 
looking for different outcomes from the placement and felt it was important that the 
assessment process would be sensitive to different viewpoints. 

The process the team went through is represented, in a concentrated form, in the 
workshop tasks. The role of the workshop facilitator is similar to that of the Training 
Agency funded project officer who put together the assessment scheme currently in use 
on the course. The workshop involves participants in two main tasks. The first task is 



5FB, UK 



INTRODUCTION 



WORKSHOP 




ASSESSING WORK-BASED LEARNING 



141 



to agree on the key quality criteria appropriate to assessing workplace-based learning. 
The second task is to apply these criteria in developing an assessment scheme based on 
case studies of student industrial placements. These activities offer an opportunity to 
explore different aspects of quality in assessment and provide an opportunity to devise 
different assessment schemes and to see how well they stand up to selected quality 
criteria. 

Task 1 - Identifying Priority Issues in Assessing Workplace-based 
Learning (30 minutes) 

For Task I the facilitator is probably best employed helping ail the participants to identify 
with their roles and helping the groups to reach decisions. 

Materials 

Pieces of card (in three colours) , one for each participant 
Writing materials 

Objectives 

By the end of the task participants should have: 

• formed syndicate groups for task two 

• discussed possible interpretations of quality from the viewpoint of student, 
industrial mentor/supervisor and academic supervisor 

• negotiated a set of quality criteria on which participants can draw for the main 



The success of the task depends on participants' willingness to engage in role play. 
Stage 1 (15 minutes) 

Participants are each given a piece of coloured card. The colour of the card indicates the 
role the participant takes in the discussion. There are three roles; student, academic 
supervisor, and workplace supervisor/mentor. As far as possible participants should adopt 
a role to which they are unaccustomed. 
Participants form a group with others in the same role. 

Individually, the participants list the key features they (as members of their role group) 
would wish to include in an assessment scheme embodying the principles of quality. 

The group members then exchange ideas and decide on a list of 'quality 01161™' they 
wish to include in the assessment process. 

Stage 2 (15 minutes) 

The participants form new groups. This time each group includes participants 
representing the three different roles. The task now is to negotiate a set of quality criteria 
that takes into account the varying concerns and perspectives of the different role groups. 
The intention here is to highlight the differences of opinion which may exist between the 
roles and to explore ways of reconciling the different perspectives. 



task. 




142 



QUALITY IN LEARNING AND ASSESSMENT 



At the end of the allotted time the group must have a set of agreed criteria. Any feature 
which is unacceptable to one or more of the roles must be modified to meet the original 
objections or be discarded. These criteria can be displayed on flipchart paper so that the 
groups can compare their own ideas with those of other groups. 

Task 2 - Building Quality into the Assessment Process (45 minutes) 

For this task the facilitator will probably need to be available to help the groups focus 
on the task but should not be tempted to supply solutions to the issues raised. 

Materials 

Case study material describing three industrial placement situations with contrasting 
characteristics (for further information about this material, contact the author). 

Source material on good practice in assessing workplace learning (see References) 

Writing materials 

Flipchart paper and pens 

Objectives 

By the end of the task each syndicate group should have: 

♦ identified key issues to be addressed when applying their agreed quality criteria 
to diverse workplace learning situations 

♦ decided on some approaches to assessment of workplace learning that meet their 
own criteria for quality 

♦ summarised their findings on a flip chart in poster format ready for presentation 
to other syndicate groups 

Introduction (15 minutes) 

The facilitator introduces the individual cases, drawing attention to particularly 
idiosyncratic features which might influence the assessment process. The facilitator 
suggests features that might be brought out in the assessment scheme. These should 
relate to the quality criteria identified in the first task. For example they might include: 
setting objective success criteria, encouraging students to develop transferable skills, 
emphasizing deep learning. The facilitator might provide examples of some approaches 
used in existing workplace learning programmes. This task includes opportunities for 
participants to raise questions about the case study material. 

Main Activity (30 minutes) 

Participants remain in the syndicate groups formed in stage 2 of the first task. 

Each group reviews its quality criteria from Task 1 in the light of the need to devise 
an assessment scheme that will work for each industrial placement described in the case 
study material. 

It is up to the group to decide how to approach the task. Some groups may wish to 
concentrate on looking at the nature of the assessment process, some may wish to define 
the appropriate outcomes of the process, some may wish to consider how to record the 
results of the assessment to implement the quality criteria. 




ASSESSING WORK-BASED LEARNING 



143 



It is not necessary for each group to develop a complete assessment scheme for this task 
to be successful. Some groups may only be able to reach agreement on the outline 
principles. Similarly, the case study material should be seen as suggestive of the range 
of problems that may be encountered by people attempting to devise or implement such 
schemes. It may be. appropriate for workshop participants to draw on examples from 
their own experiences, either to supplement case study material or in the place of the 
material. 

The group can choose which pans of its deliberations it wishes to share with the other 
groups. 

Syndicate Groups Report Back (20 minutes) 

Each group in turn shares its findings with the other groups. The time available will 
typically be between five and ten minutes for each group, depending on the number of 
syndicate groups. 

Workshop Conclusions (15 minutes) 

Each participant in turn is given the opportunity to comment on the success of the 
workshop from their own perspective. The facilitator may offer general concluding 
remarks if this is appropriate. 

References 

Coates H and Wright J (1991) The Integration of Work Based Learning with Academic 
Assessment, Guidelines for good practice Coventry Polytechnic Coventry 



1 



24 Using learning contracts to enhance 
the quality of work-based learning 



Iain S Marshall and Margaret L Mill, Napier University, Edinburgh, EH14 1DJ, 
UK 

INTRODUCTION 

Involving learners in the development of their learning contracts is proving a powerful 
means of enhancing the quality of learning for undergraduates on sandwich col. es, and 
for employees in small businesses. A learning contract is a formal agreement between 
the learner, assessor and mentor, about what will be learned, and how that learning will 
be achieved and assessed. 

Quality learning which takes place in sandwich placement, or work experience, is 
gaining full academic recognition at Napier University (formerly Napier Polytechnic) in 
Edinburgh, where undergraduates are collaborating with teaching staff and employers to 
develop learning contracts which earn academic credit towards their degree. The 
Department of Employment art funding the refining of a stude-.t-driven model, which can 
be used in a variety of institutional contexts (Figure 24.1). 




Outcomes 

Agreed Expectations 

Agreed Criteria 

Agreed Learning Outcomes 

Negotiated Assessments 

Credit for Learning Achieved 



Figure 24.1 Model of student driven three-way learning contract 

By encouraging the students to take a large measure of responsibility for identifying what 
they want to learn from the experience of sandwich placements, the motivation of the 
learner is harnessed to achieving agreed learning outcomes. The model encourages 
ongoing re-negotiation by all three partners. Students are keen to turn learning which 
they value into academic credit towards their honours degree. 

Academic staff tend to find the focus on the outcomes of learning an alien concept, and 
this identifies a need for appropriate staff development. 

The Napier project involves students in each of the stages identified in Figure 24.2. 



USING LEARNING CONTRACTS 



145 



Workshop meetings of students 
with staff to prepare individual 
Learning Plans 

1 

Student negotiates 
Learning Plan 
with Polytechnic Assessor 

i 

Students agree 
Learning Plan 
with Placement employer 

\ 

Student in placement 
organisation working 
towards Learning Outcomes 
Review and update Learning 
Contract 



I 

Student returns to Polytechnic 

with evidence of 
Learning Outcomes achieved 

T 

Polytechnic assessment of 
evidence of learning 
presented by student/employer 

Figure 24.2 Model of work-based learning process for students 

Employers of our sandwich students, especially small and medium sized companies, 
greatly appreciate students negotiating what they hope to learn from placement 
experience, in advance. It saves time and money by avoiding inappropriate matching of 
student and placement. Students are glad to have the chance to turn relevant learning 
from experience, into real academic credit towards their degree. 

Of course, this approach raises a whole range of staff development issues for the 
university, as it shifts the role relationship between staff and student, towards greater 
learner autonomy. However, a possible bonus to the academic institution is to 
legitimately shorten the overall length of a course, by offering academic credit for a 
previously "dead" period. 

There are three good reasons why systems and procedures must be developed, which 
allow students to ea' > credit which counts towards their degree, from work-based 
learning: 



146 



QUALITY IN LEARNING AND ASSESSMENT 



• In academic institutions there is a distinct tendency for resources to follow the 
award of credit. 

• Many of our student have an instrumental approach to course work. The award 
of marks or credit influences the effort they are prepared to put into an 
assignment. 

• Where employers are participating actively in the process they express the view 
that students should be rewarded for achieving relevant learning outcomes. 

ASSESSMENT OF WORK-BASED LEARNING 

Assessment of the evidence submitted by the student is central to the award of credible 
academic credit. 

In the learning contract model being developed in Napier University, the following 
principles and conditions are emerging. 

Some general principles: 

1 That assessment be collaborative rather than unilateral. 

2 That assessment be constructive and responsive to learners' needs. 

3 That the conditions necessary for good assessment must be given a priority in the 
allocation of time and other resources. 

Some necessary conditions: 

1 That there be clear learning outcomes which are agreed between learner and 
assessor to be relevant to the qualifications sought and valued by the learner. 

2 That the credit awarded for evidence of learning should fairly reflect the quality 
of that learning. 

3 That there be a policy regarding the nature of evidence of learning and a 
readiness to explore different instruments of assessment. 

4 That there should be mechanisms which take account of the views of the learner 
and the employer in arriving at a collaborative assessment. 

5 That there should be a policy for the professional development of teaching staff 
and of employer 'mentors', in support of the woik-tased learner and in the 
assessment of the evidence of learning which is submitted. 

6 That there be a structure within an institution where the academic Board of 
Studies can evaluate its success in achieving its own stated aims, and adjust its 
practices appropriately. 

The Learning Outcome 

The fundamental element of the Napier model of work-based learning is the learning 
outcome. By encouraging the learner to reflect on learning which they already have 
achieved, or identify learning they will attempt to achieve, attention is paid to specifying 
learning which is supported by adequate evidence. 

Students are encouraged to group learning outcomes they plan to achieve from 
placement under three headings: 

Job Related Outcomes 

Personal Development Outcomes 

Course Related Outcomes 



USING LEARNING CONTRACTS 147 

During the planning phase prior to going on placement, the developing list of learning 
outcomes is negotiated with staff in the university. The first major consideration is that 
the learning outcomes should be judged to be meeting the aims of the degree. The 
second is that the learning outcome is capable of being assessed. 

Each student negotiates with a member of the university staff who will be the university 
assessor of the evidence which the student will offer in support of the agreed learning 
outcomes. The list of learning outcomes proposed by a student is used as part of the 
agenda during placement interviews with potential employers. Invariably, further 
negotiations occur which have to satisfy all three partners - the student, the company 
mentor and the university assessor. 

Throughout the placement experience, the list of learning outcomes is regularly 
modified and renegotiated between all three partners to the learning contract. Before the 
student is due to finish the placement, they confirm with the university assessor the list 
of learning outcomes they wish to be assessed against. 

The company mentor receives a print out of this list in the form of an assessment grid, 
which asks the mentor to confirm whether or not the student in each case: 



a Has indeed accomplished the learning outcome. 

b Has provided evidence of understanding against a scale 1-4. 

c Has provided evidence of adding value. 

This assessment by the mentor is invariably discussed with the student, and sent to the 
student's university assessor, who has his/her own assessment grid which is asking 
similar questions of the portfolio of evidence the student submits in support of their 
claimed learning outcomes, and arrives at a considered judgement as to the credit to be 
awarded. 

By developing professional judgement through experience gained with colleague 
assessors and students, the question of quality and standards are continually reviewed and 
refined. 



New Pilot Project 

A spin off from this work has been a new development of the model, which is being 
supported through funding of another project by the Scottish Enterprise Company and 
Lothian and Edinburgh Enterprise Limited (LEEL). The idea is to enable the Work- 
Based Learning Unit to target a number of small business enterprises. The intention is 
to identify training needs within the small businesses, negotiate those needs into an 
agreed learning contract, and to use a variety of appropriate resources to meet these 
needs, through flexible delivery mechanisms. 

One of the innovative aspects of this new pilot project, is the involvement of small 
businesses, in partnership with a major company, which has these smaller companies as 
suppliers. A major benefit to the large company is the interest they have in the continued 
well-being of their supplier companies, and this approach to total quality management is 
enhanced by the customer/supplier partnership in identifying areas for improved 
efficiency. It offers an opportunity for the sub-contractor to participate in an analysts of 
their own training needs and to improve their efficiency by developing appropriate 
management skills. . 

All of this experience leads us to see the learning contract as a powerful mechanism tor 
holding together: what has to be learned/demonstrated, how this will be achieved, and 
how it can be assessed. 



153 



148 



QUALITY IN LEARNING AND ASSESSMENT 



ACCREDITATION OF WORK-BASED LEARNING 

Napier University is moving towards a credit-based system, which will incorporate work- 
based learning, but at present undergraduates still have to achieve a number of marks 
from a variety of instruments of assessment, which determines their honours degree 
classification. 

A danger in the move toward a credit-based scheme, is that where units of credit are 
based on units of time, ie 1 credit = 8 hours of appropriate activity, then the 
discriminatory power of the model is lost. To be regained, it seems to us that we must 
move on from the simplistic time-based unit of credit, toward an acknowledgement of the 
relevant learning outcomes which can be realistically achieved and demonstrated by a 
learner. This would allow students to earn more or fewer credits towards their degree, 
by the way of work-based learning. Where courses are explicit about the learning 
outcomes which are at present implicit within the syllabus, the issue of credit transfer, 
based as it would be on credit for learning outcomes achieved, is greatly facilitated. 

Moreover, the Napier experience is that this need be no more expensive than delivering 
university based C.A.T.S. credit - and at a stroke, takes away the widely held view that 
work-based learning has to be so resource intensive that it must be 'too expensive*. 

By calculating the total number of units of C.A.T.S. credit available in an academic 
year, and dividing by the total number of available academic staff hours, we arrive at an 
efficiency ratio which shows that on average each available staff hour can generate 1.2 
units of credit for students. 

Total F.T.E's x 120 Units of Credit = 1 .2 Units of Credit per 

Total number of available academic staff hours available staff hour 

When we look at the units of credit available from 'sandwich' placement, and use the 
figures from our project to estimate staff hours necessary to support and assess this 
number of credits, the resulting efficiency ratio is 1.9 units of credit per staff hour. 

Units of available credit x student numbers = 1.9 Units of Credit per 

Total staff hours in napier work-based learning model available staff hour 

Finally, the conclusion reached by the staff at Napier University is that the award of 
academic credit for evidence of relevant learning is the engine which drives the process 
for undergraduates. For people in companies, it is the recognition that what they are 
learning is of benefit to them in the work that they do. The challenge confronting us is 
to continue to refine appropriate systems and procedures for supporting relevant learning. 
Anyone interested in learning more about this approach is encouraged to contact the 
authors at the Work-Based Learning Unit, Napier University, 219 Colinton Road, 
Edinburgh EH14 1DJ, Telephone: 031 455-4390. 

References 

Boak G (1990) Developing Management Competences Pitman London 

Carr and Kcmmis (1983) Becoming Critical - Knowing Through Action Research 
Deaking University Press Geelong 

Knowles M S (1986) Using Learning Contracts Jossey-Bass San Francisco 



154 



25 Quality of assessment 



Phil Race, University of Glamorgan, Pontypridd, CF37 1DL, UK 
INTRODUCTION 

Much of this volume is about the assessment of quality - this paper is about the quality 
of assessment. What our students really pay for (or what society pays us for ) are their 
degrees or their diplomas or their certificates. Yet we put much more emphasis in trying 
to teach our students than we normally put into designing assessment - particularly the 
"formal assessment" that counts towards final qualifications. Assessment is where 
learners often get a raw deal. 

The 1991 AETT Conference had the theme "Developing And Measuring Competence". 
As one of the editors of the proceedings of that conference (Saunders and Race 1992), 
I became well acquainted with the many views on competence which were aired by the 
various contributors. It was heartening to see that many of the approaches to the 
development of competence focused on defining the evidence which would be sufficient 
to demonstrate competence. 

However, I was left with some concerns about whether competence would often end up 
as minimum competence rather than high competence. Figures 25.1 and 25.2 pose some 
questions about "shades" of competence, and about a range of descriptors which may be 
needed to provide more information than simply "can do". 




Figure 25.1 Shades of competence 



15 



150 



QUALITY IN LEARNING AND ASSESSMENT 



excellently 



distinguishedly 



innovatively 




Figure 25.2 Some competence descriptors 

Higher education in the United Kingdom is being moved steadily towards greater 
participation rates. This move includes not only rapidly increasing numbers of school 
leavers, but also many more non-traditional entrants. Everywhere, lecturers and tutors 
are seeking help in handling large classes. It is clear that the workload of staff in higher 
education is increasing, as they become responsible for more and more learners. 

In some ways, teaching can be "done to" large numbers almost as easily as to small 
numbers - it's almost as easy to lecture to 300 as to lecture to 100. However, teaching 
greater numbers may have little connection with learning by greater numbers. And when 
it comes to assessment, there are no short cuts. It usually takes three times the amount 
of time to assess 300 learners as it would to assess 100 learners. The main challenge we 
face as we move towards a mass higher education system, is merely maintaining the 
quality of assessment we presently have - not to mention improving it. 

HOW MOST PEOPLE LEARN 

First, consider how people learn - real people, that is. not educational psychologists or 
their learners! During the last couple of years, I've given thousands of people post-its, 
and asked them to write down the answers to the following questions on them. 



I. Think of something you're good at - something you do well. Write down a few 
words about how you became good at it. 



155 



QUALITY OF ASSESSMENT 



151 



2. Think of something ywfeel good about - something that gives you a "glow". Write 
down a few words explaining your evidence for this good feeling. 

3. Think of a learning experience that was not successful. Write down a few words 
explaining what went wrong. 

The answers to "1" are almost always along the following lines: 

♦ doing it 

♦ practice 

♦ trial and error 

♦ learning from mistakes. 

In other words, most people learn by doing rather than by sitting at the feet of masters 
or mistresses. 

The answers to "2" are usually along the following lines: 

♦ feedback from others 

♦ other people's reactions 

♦ because I can see the effect. 

In other words, to feel good about something (including learning) , people need feedback. 

The answers to "3" are more complex. Sometimes they reflect things that went wrong 
with the "learning by doing" stage - for example lack of opportunity to practice. 
Sometimes they reflect things that went wrong with the feedback stage - in other words 
lack of the opportunity to develop good feelings about what had been learned. Two more 
problems often emerge: 

♦ lack of motivation in the first place 

♦ lack of opportunity to make sense of the learning experience. 
LINKING TRADITIONAL ASSESSMENT TO LEARNING 



So how does assessment usually relate to the four fundamental steps in the learning 
process? Let's take the most common form of assessment - the written examination. 

♦ Wanting to learn (motivation) 

Not many people like exams! The fact that there is an exam coming along at the end 
of the road is not the strongest motivator for most people. The inevitability of 
traditional forms of assessment is a key factor in preventing many people from 
participating in learning. 

♦ Learning by doing (practice, learning from mistakes, and so on) 

Learning by doing can indeed happen during assessments, including written exams. 
Mistakes are indeed made during assessment - plenty of them! But learning by doing 
while being assessed is hardly the best way of using experiential learning. Besides, 
it is usually presumed that the learning should have taken place before the assessment 
event. 

♦ Learning through feedback (to develop positive feelings about the learning) 

All assessment results in some sort of feedback. However, it's often the absolute 
minimum of feedback - for example a mere score - and that's weeks or months after 
the event! There is little or no real feedback, and chances to learn from the feedback 
are minimal. 



157 



152 



QUALITY IN LEARNING AND ASSESSMENT 



• Digesting (taking stock, making sense of the experience, and of the feedback) 

Exams are better known for producing indigestion than for allowing people the chance 
to consolidate their learning. As I've said, the feedback is usually very limited in 
scope (and often delayed in time) and is not a useful means towards "digesting". 
Such feedback as there is, tends to be one-way feedback. There's or no little chance 
to discuss the details or negotiate what best to do next. 

At each step, assessment processes are at cross purposes with learning processes. Perhaps 
traditional forms of assessment have only one real contribution towards learning- people 
are frightened (shamed) into doing some learning so that they may minimise their chances 
of being shown to be "lacking". Much intensive learning is done just before exams - but 
most of it is of a superficial nature, and soon forgotten again. 

TEN WORRIES - AND SOME SUGGESTIONS 

I list below ten worries I have about assessment, outline my concerns, and offer some 
suggestions about how the problems may be minimised. Many of my suggestions point 
towards involving learners in their own assessment. This can be achieved by letting 
learners get their hands on the assessment criteria. It can be achieved even better by 
letting learners apply the assessment criteria, in self -assessment and peer- assessment. It 
is achieved best by helping learners to formulate the criteria, then apply them. An 
illustrated discussion of uses of self- and peer-assessment is included later in this paper. 

1 . Assessment is often done in a rush, to meet exam board deadlines. It's rarely done 
under the best of conditions! 

This is because assessment tends to be done to learners - not by them. Assessment tends 
to be done at the end of learning something, rather than as a means to help the learning 
processes. In public exams, examiners often face piles of some hundreds of scripts, all 
of which need to be finished within only a week or two. Some suggestions for avoiding 
the problem are as follows: 

• allow much more time for self-assessment, so that it can be done well 

• allow learners to use self- and peer- assessment, so they can learn by assessing. 

2. Assessment is often done by bored people, tired of reading the same answers to the 
same questions (and seeing the same mistakes). 

Examiners get thoroughly fed-up as they wade through hundreds of scripts. They get 
discouraged when they see things they hoped their learners would have mastered, only 
to find thai messages have not got across. Any tedious or repetitive task causes people 
to change their mood. If assessors' moods plunge, the objectivity of assessment is likely 
to be affected accordingly. 

• decrease the emphasis on traditional written exams altogether 

• allow learners to learn by reading their own mistakes, and those of their peers. 

3. Assessment tends to be governed by "what is easy to 3ssess u . Therefore, traditional 
written exams (relatively straightforward to assess) are used. These measure learners' 
skills at tackling traditional written exams. 



QUALITY OF ASSESSMENT 



153 



How much you know? 

How much, you don't know? 

How fast you can write? 

How good your memory is? 

How much work you did the night before? 

How well you keep your cool? 

How competent you are? 

How well you can read the questions? 

How good you are at answering exam 
questions? 

How practised you are at answering exam 
questions? 

How you perform under pressure? 

How good you are at time management? 

How well you can keep addressing the 
question? 

How often you've practised on similar I 
questions? I 

How well you read your own answers? | 
Figure 25.3 What exams measure? 

There is still not enough attention being paid to what should constitute the evidence upon 
which to base awards. Many important competences are simply not assessable by 
traditional methods. While it is perfectly possible to use traditional methods to measure 
recall of facts and information, it is not-at-all easy to use such methods to measure 
innovation, judgement, or personality. Figure 25.3 shows an overhead transparency that 
I use to alert students to the various agendas that may be served by traditional exams. 

• look carefully at what is being measured by each form of assessment 

• refrain from measuring the same things all the time - especially recall (people who 
can find and apply information are usually more valuable than people who simply 
happen to remember a lot of it). 

4. Learners rarely know the intimate details of the assessment criteria, and how we 
interpret them. 

There really is no excuse for this. The reason may be sinister - that those who design 
the assessment criteria are not sufficiently confident about them to show them to the 
learners! Assessors often fear that learners may demand to know "why did I get 65% for 
this, when my friend got 75%?" Surely, they have every right to ask this sort of question 
- and to learn from the feedback they should be given by way of a response to the 
question. 

• give learners the opportunity not only to see the intimate details of assessment 
criteria, but also to use the criteria. 



153 



154 



QUALITY IN LEARNING AND ASSESSMENT 



5. How should we develop learners' unassessable qualities? Should we refrain from 
developing them because we can't measure them? 

"Don't bother to learn anything, when you can't see how they can ask you about it at the 
end of the day": this is a perfectly rational view taken by learners, deciding what to learn 
and what not to learn. 

• bring the unassessable qualities firmly onto the agenda; explain to learners why 
they're important, and work out with learners what kinds of evidence can be 
linked to these qualities, and how the demonstration of that evidence can be built 
in to assessment procedures. 

6. Almost all assessment processes in common use foster learner competition rather than 
collaboration. No wonder our educated people are so bad at working in teams. 

Learners preparing for exams are often quite secretive about the work they do. No-one 
likes to be thought of as a "swot"! However, it's more sinister than this: we actually 
compound the competition by using norm-referenced assessment far too much. In other 
words, only a certain proportion of learners are allowed to receive "A" grades, or 1st- 
class honours degree classifications. Therefore, learners are in competition. 

• use criterion -referenced assessment only - abolish the use of norm-referencing 

• help learners to feel that they can help each other prepare to demonstrate their 
competence, without disadvantaging one another. 

7. What competences are measured by assessment anyway? Are they "can do" 
competences? Or are they simply "did do, once" ghosts? 

Exams tend to measure "did once" competences. At their worst, they still remember 
"knew once" competences rather than "did once"! 

• increase the proportion of assessment schedules allocated to continuous assessment 
- which measures "is doing" competences rather than "did once" ones. 

8. If we were to introduce "Quality in Assessment BS5750A": what should the criteria 
look like? What evidence of competence should assessors demonstrate? 

At present, it is automatically assumed that anyone appointed to a post involving teaching 
or lecturing is blessed with all the skills needed to design assessment schemes and 
implement fair assessment. People are appointed to teaching (and assessing) posts not 
on the basis of how well they can do either task, but often on the record of their own 
academic performance. 

• assess the assessors - have a system of "licences" to assess, and police the system 
thoroughly 

• increase the uses of self-assessment and peer-assessment, which depend far less 
on subjectivity of assessors, and allow far greater amounts of feedback to 
contribute towards successful learning experiences. 

9. "If you can't measure it, it doesn't exist. If you can measure it, it isn't it". What 
should we be trying to measure? 

It has been said that one of the main faults of our education and training systems is that 
we tend to teach people things that are already understood, instead of equipping them to 
understand new things. Assessment reflects this. 



QUALITY OF ASSESSMENT 



155 



• use self- and peer-assessment as an inherent part of learning processes, with the 
emphasis on learning rather than assessment outcomes. 

lO.Where stops the buck? Whose fault is it that assessment is so artificial? HoD? 
Employers? Assessors? Validators? The Government? Yours? Mine? 

If you imply that there is something suspect about people's abilities to assess, it is badly 
received! Assessment is something that is usually done privately rather than publicly, and 
people go to great lengths to ensure that they retain privacy. Is such privacy really 
needed mainly because of the "put down the number you first thought of" syndrome? 

LEARNING THROUGH ASSESSING 

In much of my discussion so far, I've been focusing on the dangers when assessment is 
"done to" people, and hinting at the benefits which can result when learners themselves 
are allowed to be intimately involved in their own - and each other's - assessment. Self- 
assessment, and peer-assessment may lack some of the precision of the best of "formal" 
assessment - where (some) assessors have a great deal of experience, and (sometimes) 
assess fairly and conscientiously. However, what may be lacked in terms of precision 
is more than compensated for by the benefits of deeper learning, which go hand in hand 
with the act of learners themselves assessing. 

Close Encounters with Assessment Criteria 

This is the crucial difference between formal assessment, self -assessment and peer- 
assessment. Learners find out a lot about any subject simply by applying assessment 
criteria to examples of work in that subject (whether the examples are self-generated, 
made by other learners or issued by a teacher). Previously, assessment criteria have 
seemed to learners to be the property of examiners. There has been a tendency for 
teachers to regard assessment criteria as quite private. For many years, I marked "O u - 
level and "A" -level scripts for two of England's public examination boards, and the 
marking schemes were invariably labelled "secret". Even where model answers and 
marking schemes have been required to be sent to external examiners or moderators, the 
vital information in such schemes has seldom been shared with learners, and until 
recently hardly ever applied by learners themselves. Yet when learners get the chance 
to get their hands on assessment criteria, they seem to develop a thirst for the information 
they can derive from them - leading to much deeper learning. 

Self-assessment and Peer-assessment Are Not Just Self-testing 

These forms of assessment when well -developed involve two processes: 

• involving learners in identifying standards or criteria to apply to their work 

• allowing learners to make judgements about the extent to which they have met 
these standards and criteria. 

Assessment Criteria: Black and White or Shades of Grey? 

In subjects like maths, science and engineering, things are often either right or wrong - 
and it is relatively easy to devise assessment criteria for tests and exercises. However, 



161 



156 



QUALITY IN LEARNING AND ASSESSMENT 



even in subjects such as law or social studies, there are identifiable hallmarks of a good 
answer or an unsatisfactory answer to a question. Such hallmarks can be turned into 
checklists of a flexible kind, which enable the characteristics of good and less-good 
answer to be compared and contrasted. Students can benefit by learning in the act of 
applying assessment criteria to their own work, and to each others* work. 

Benefits to Learners of Close Encounters with Assessment Criteria 

Learners can quickly find out about incorrect assumptions they have been making. They 
are able to find out the answer to the question: "What am I expected to become able to 
do?" There are of course many more benefits, depending on how we involve learners in 
using assessment criteria - including helping learners themselves to formulate the criteria 
(when this is possible or appropriate) - leading to the most obvious form of ownership 
of assessment. 

Some Examples of Self- and Peer-assessment Mechanisms 

Self assessment is not confined to the variety that is widely used in open and distance 
learning (though of course that is one powerful form of it). Self assessment processes 
can include any of the following: 

• providing learners with assessment criteria and a marking scheme and allowing 
them to mark their own work 

• as above, but then allowing learners the chance to compare their mark with that 
of a "professional" marker 

• as above, but also giving learners feedback about the quality of their self 
assessment 

• enabling individual learners to generate assessment criteria, and use them to assess 
their own work 

• enabling a group of learners to generate assessment criteria, and so on 

• allowing learners to use core criteria generated by a group, plus additional criteria 
specific to their own pieces of work, with an agreed weighting 

• groups of learners can be issued with criteria to apply to each others* work 

• groups of learners could produce criteria and apply them to each others' work. 

There are further combinations of these. There is also the additional matter of whether 
the grades or scores contribute in a formal way to the performance records of learners. 

Eliciting Assessment Criteria from a Group 

I have found that the following approach gives useful results for groups of 10-20. 

• I ask each learner to privately list (eg.) six things you would expect of a good V 
(where V could be an essay or presentation or handout and so on). 

• I ask learners to go to groups of 3 or 4, and discuss criteria. 

• I ask the groups to make a list of criteria, and to prioritise them. 

• I flipchart the most important criteria from each group, then the next most 
important, etc. 

• I ask the whole group whether anything important is missing from the flipchart 
list. 

• I tidy up the flipcharted items if necessary, and number them eg. 1-8. 

• I ask each learner to privately distribute (eg. 20) "marks" among the criteria. 



162 



QUALITY OF ASSESSMENT 



157 



• I write each learner's "mark" alongside each criterion on the flipchart. Then I 
either average them out, or allow each learner to apply his/her own weightings in 
the peer assessments to follow. 

• Learners then go off to perform the task (individuals or groups). 

• I then produce a grid with their criteria and weightings, ready for peer assessment. 

An Example 

I helped a group of students from a foundation course in Science, Engineering and 
Technology to generate and prioritise some simple criteria for a short presentation that 
each student was to give to the group. (The presentations were optional; the assessment 
for the "Learning Strategies" module of the course contained six equal elements, and 
students could choose which five they tried - in other words they could choose not to 
give the presentation if they really wished to). We agreed that anyone who had the 
courage to get up and start immediately deserved to "pass" - in other words they got the 
first 40 marks there and then. Seven criteria were then devised by the students to account 
for the remaining 60 marks, and the peer-assessment grids became as shown in Figure 
25.4. 



r 


A 


B 


C 


D 


E 


F 






\ Criteria 


Weight 


















1 confidence 


5 


















^ staying on 
topic 


5 


















3 well 

illustrated 


10 


















4 well 

researched 


10 


















^ handouts 
overheads 


1 0 


















® dealing with 
questions 


10 


















^ timing 


1 0 


















8 

starting 


40 






































^^Total 


100 



















Figure 25.4 Peer-assessment grid for presentations 



163 



158 



QUALITY IN LEARNING AND ASSESSMENT 



Each student filled in a grid for each of the presentations he or she witnessed (and 
students additionally self-assessed their own using the same criteria). For various 
reasons, some students were only able to be present for half of the time involved, and one 
or two students participated as assessors, but did not wish to give presentations 
themselves. I myself worked as one of the assessors, and was relieved to find that in 
general, the marks I awarded each of the presentations were close to the average mark 
for that presentation. 

Learners' Ownership of Assessment Criteria 

The sort of peer-assessment described above is suitable for tasks such as presentations, 
where many people can assess the same piece of evidence, and where scores can then be 
compared and discussed by the group. For individual tasks such as essays, reports, 
projects, dissertations and so on, it is likely that each piece of work will reflect slightly 
different criteria (or even very different criteria), and then it is often best to allow for 
some "agreed" criteria, and some "idiosyncratic" criteria so that each learner can exercise 
more ownership of the assessment criteria. 

The most important outcome of involving learners in the formulation of self -assessment 
or peer- assessment criteria, is that learners address the task with criteria in their minds, 
and the quality of their work seems to be much higher than it may otherwise have been. 

Throw Away the Numbers or Grades? 

I've often suggested to learners after a peer assessment exercise that the numbers or 
grades awarded (if not contributing to their overall assessment) were only a vehicle to 
help them do learning of a higher productivity. However, I've found they usually want 
to hang on to the numbers - good or bad. Perhaps this is evidence of the sort of 
ownership we* re aiming for? 

How Well Can Students Assess Themselves, and Each Other? 

In general, students are quite accurate in their assessing. I have found that when students 
are asked to "guess" their own performance scores just after completing an exam, around 
90% of students "guess" within 5% of their actual scores. It is useful to identify the 10% 
who had an inaccurate perception of how they had done - they usually benefit from a 
discussion to probe the causes. Those 10% may be over anxious, and underestimate their 
achievements, or over-confident and over-estimate their achievements. When 
discrepancies in self- assessment occur, they are usually due to one of the following 
causes: 

• there is some tendency for learners lo over- rate themselves in areas to which they 
are new - this tends to happen with the weaker members of the group 

• there is a tendency for some learners to under-rate themselves in areas in which 
they are experienced - this tends to happen with the more-skilled members of the 
group. 

Peer-assessment and self -assessment can be usefully combined. Peer-assessment can be 
conducted "blind" so that "arranged" scoring is avoided. If the peer- assessment mark or 
grade is equal to the corresponding self- assessment mark (e.g. within 5%) then the self- 
assessment marks go forward into the assessment system - possibly with a staff "scan" 
to ensure that fair play is in operation. Ot is far quicker to scan a piece of work to check 



QUALITY OF ASSESSMENT 



159 



whether the assessment is fair, than it is to mark the work from scratch). When self- and 
peer-scores differ, negotiation or staff intervention may be necessary (but this happens 
surprisingly rarely in practice), 

SELF-ASSESSMENT, PEER-ASSESSMENT AND LEARNING 

Throughout this paper, I've advocated the benefits of helping learners become intimately 
involved in processes of assessing, and I've pointed to the hazards of traditional 
assessment procedures. To conclude, lets look once again at the four main processes of 
learning, and see how self- and peer- assessment link to them. 

• Wanting to learn (motivation) 

Motivation can be improved by early success. Self-assessment in particular can be 
used with the comfort of privacy, and learners can gain confidence by finding that 
they are "doing alright" long before they need to prove so publicly or formally. 

• Learning by doing (practice, learning from mistakes, and so on) 

There's no better way to find out about one's successes and failures than by finding 
them out for oneself - or having a peer help one do this, rather than an "authority 
figure" like a tutor or examiner. The very act of assessing is intrinsically "learning 
by doing" - it involves the application of criteria, decision-making, judgement, and 
reflection. 

• Learning through feedback (to develop positive feelings about the learning) 
The showstopper of formal assessment has to be the dreadfully limited feedback that 
is the norm. Peer-assessment can allow for a great deal of feedback - far more than 
could ever be given by a tutor or assessor. In addition, the feedback gained in peer- 
assessment is usually far less threatening than that from "professionar 1 assessors. 
Indeed, peers will often argue and debate issues, further deepening the usefulness of 
the feedback exchanges they receive. 

• Digesting (taking stock, making sense of the experience, and of the feedback) 
Both self- assessment and peer-assessment can help learners make sense of their 
learning experiences - and of the feedback they gain. Furthermore, the time lag 
between the learning and the feedback can be much less than with traditional methods 
of assessment. Therefore, the feedback is much more actively received and the 
learning thereby enhanced. 

Reflecting on Assessment 

It can be argued that people who need a "tester" are inadequately prepared to be sent out 
into the world outside. Self-assessment and peer-assessment can both be an important 
part of the learning process. The learning experience resulting from such forms of 
assessment is more important than the result of the assessment. SeJf-assessment or peer- 
assessment do not necessarily have to lead to any "formal" (recorded) assessment. The 
aim can be purely as a learning experience, with the "marks" simply part of the process 
through which that experience is facilitated. 

Self- assessment and peer- assessment arc skills, and become more reliable with practice. 
Receiving feedback on the quality of these forms of assessment is vital if learners are to 
derive the maximum benefit from engaging in them. 

Self -assessment and peer-assessment should be introduced early - for example during 
the first term rather than being left until the final year. Late in a course, students may 
see little point in embracing new ways of learning. 



165 



160 



QUALITY IN LEARNING AND ASSESSMENT 



"Ownership" is the most crucial aspect of successful learning, and both self-assessment 
and peer-assessment are closely connected to the development of ownership of learning. 

Not all students warm to the "exposure" of self- or peer-assessment. They may begin 
their studies with expectations that they will be assessed by professionals. "What's in it 
for me?" they naturally may ask. They need to be convinced that self- and peer- 
assessment have direct benefits for themselves, and do not represent an abdication from 
duties on the part of tutors. Some tutors however feel it is dangerous to "lose control" 
of assessment. If such tutors try to employ self- or peer-assessment, but constantly 
safeguard their right to step in "should things go wrong", the whole concept of such 
forms of assessment is undermined. 

So what about teaching? Admittedly, our students learn from us. But they probably 
learn more on their own, and they probably learn even more from each other. Much of 
their learning occurs in the immediate run-up to assessment of one kind or another - so 
the role of assessment is an important factor in the circumstances which accelerate 
learning. Therefore, perhaps our biggest contribution to our students' learning is directly 
associated with the quality of the assessment our students encounter - and has less than 
we'd like to think to do with our teaching activities. I believe there is a strong case for 
using self- and peer-assessment not primarily to assess, but as processes to enhance 
learning. 

Finally, if and when we must resort to traditional, formal assessment, I believe that 
there is a great deal of room for improvement. The "ten worries" I expressed earlier may 
help to set an agenda for improving the quality of assessment. I end this paper with 6 
recommendations. 

WHAT CAN WE DO TO IMPROVE ASSESSMENT? 

• Extend and develop the competence-assessment approach, where people 
accumulate evidence of the things they do, and when they're ready, bring the 
evidence forward for assessment. 

• Use assess ment as a learning opportunity. Give people feedback on their 
performance, and help to improve it. Make the primary purpose of assessment 
that of helping people find out how their learning is going, so that they can adjust 
their strategies accordingly. 

• Use assessing as a learning process - help learners to self- assess their own work, 
and peer- assess their colleagues' work (whether or not the marks or grades are to 
"count"). Let learners gain familiarity with the nature of the assessment criteria 
which they need to live up to. Help them find out what sort of performance they 
need to be able to give. 

• Change the culture where any professional can sit and scribble down an exam 
qu* ion in a few minutes, to a culture where all important exams are made up 
entirely of questions which have been piloted, tested, evaluated, and proved to 
measure desirable things. 

• Develop team approaches to question setting, and marking. Introduce at the very 
least double-marking as a standard for exams - and when possible multiple 
m.vking. 

• Pay people well for assessing, instead of just paying them for teaching. Teachers 
often earn "holiday money" by doing extra exam-marking. It's all the wrong way 
round! Assessment is so important that perhaps it should be that assessors can 
earn "holiday money" by doing a little teaching! Then require them to 
continuously demonstrate their competence at the job of assessing. 



16 



QUALITY OF ASSESSMENT 



161 



References 



Saunders D and Race P (Eds) (1992) Developing and Measuring Competence Aspects 
of Educational and Training Technology XXV Kogan Page London 




26 BS5750 for assessment 



Workshop led by Phil Race, University of Glamorgan, Pontypridd, CF37 1DL, UK 
INTRODUCTION 

The following abstract had been published in the Conference documentation: "The 
workshop will be a participative event, sharing the good (and bad) experiences of 
participants - both of assessing - and of being assessed. The workshop will begin by 
extending the 'ten worries about assessment' presented in the associated paper (chapter 
25 in this volume). Syndicates will examine ways and means for overcoming the 
problems. The primary aim of the workshop is to generate a set of 'quality criteria 
which can be applied to assessment processes, practices, and instruments, to guarantee 
learners a fair and just assessment". 

The aims and objectives of the workshop were: 

• to pinpoint the weaknesses in assessment practices and devices 

• to explore alternative ways of assessing 

• to enhance the learning which can be derived from assessment 

• to produce "quality criteria" for assessment. 

PARTICIPANTS' EXPECTATIONS 

Participants were given a small piece of acetate and asked to write on it their answers to 
the question: "what do you personally hope to gain from this workshop?' . This aUowed 
the workshop to address particular issues which participants had brought to the woii^hop. 
The acetates were displayed to the group, and clarification encouraged where needed. 
Their expectations were as follows: 

• Where assessment fits into BS5750 (Nick Owen) 

• To get an idea of how BS5750 compares with other possible models (Chanid) 

• To use BS5750 creatively (Jill) 

• To highlight quality criteria (Alec Bickerton) 

• Relating BS5750 to assesn-nent, specifically quality manual documentation 

(Michael) . 

• How to improve assessment - ideas, experiences of others (David) 

• When and how it can go wrong (David) 

• Further ammunition and ideas for reforming assessment (Bob Sawyer) 

• Rationale for justifying democratic versus traditional assessment for use in the 
context of a traditional organisation (Jeremy Arter) m 

• Identifying appropriate means of assessing (non-teaching background) (fatncia) 

• Criteria for designing quality assessment; ideas on using computers for continuous 
assessment (Chin-Poon) . 

• Ideas to help colleagues to more effective assessment with less quantity ot work 
(Exam board activity can surely be just as effective with fewer numbers to 
crunch!) (Julie Clarke, a registrar) m 

• Why BS5750 is in the title but not mentioned in the abstract (Chris U Haganj me 

found out!) i e % * ( 

• Clear ideas about quality assessment; how to avoid unhelpful assessment ot 

students' work (Christine) 

• New insights into the use of assessment (Paul Ellis) 



16C 



BS5750 FOR ASSESSMENT 



163 



♦ Changing students' expectations to accept student-centred assessment (Larry 
Roberts) 

♦ How far should courses be "assessment driven"? The relationship between 
assessment and outcomes. How far should the teaching and learning strategy 
reflect the assessment strategy? (Steve Leary) 

♦ Ideas (Andrew) 

♦ Ideas to use in a course I am designing where I might be able to include diagnosis 
as a form of student learning (Rachel) 

♦ Some ideas on BS5750 criteria for assessment (Jill) 

♦ Ideas, concepts to be of use with FE colleagues wrestling with problems of 
assessment (Jack Oakley) 

♦ Find out a little about how FE/HE have initiated changes/improvements in 
assessment procedures, and how this is linked to 5750 (Peter Leckstein) 

These expectations are of considerable interest in their own right. They illustrate the 
many different ways that people view assessment. Several expectations mentioned 
BS5750 in particular. The facilitator "came clean" at once on this, and indicated that the 
purpose of the workshop was to generate information which would lead to criteria for 
high quality in assessment, rather than look at how existing BS5750 documentation may 
lend itself to applications to assessment. 

Many participants had already been present when Phil Race ran his earlier Conference 
session "Quality of assessment", and he issued further copies of his paper with that title. 
He summarised quickly how learning is dependent on the four processes of "wanting", 
"doing", "feedback", and "digesting" (as discussed in detail in the paper), and reminded 
participants that one of the intentions of the workshop was to explore how best 
assessment could be designed and implemented to assist these processes of learning. 

FITNESS FOR PURPOSE 

Participants were divided into syndicates, and briefed as follows to consider particular 
types of assessment, and prepare an overhead transparency addressing each of the 
following issues: 

SYNDICATE TASK I 

Choose a type of assessment 

♦ decide exactly what it actually assesses best 

♦ discuss how it helps the processes of learning 

♦ list some advantages and disadvantages of the type of assessment. 

Figure 26.1 consists of transcripts of the acetates produced by the syndicates, which add 
up to an interesting exploration of a number of different possible processes of assessment. 

INDIVIDUAL RECOLLECTIONS OF ASSESSMENT 

Participants were issued with post-it slips, and asked several questions about the last 
assessment they themselves had experienced as learners. 

Their replies are transcribed in Table 26.1. They show a fascinating range of feelings 
about assessment, attitudes towards the results of assessment, and views about what (if 
anything) was learned from the various experiences of being assessed. 



163 



164 



QUALITY IN LEARNING AND ASSESSMENT 



1 ROLE PLAYING 




What? attitudes 




procedural behaviour 




skills (Interpersonal) 


(problem solving) (self awareness) 


applying theory 




HOW it help$ • confidence boosting 


• practice 




• experiential learning (remembered) 


• time effective 


MU V a N Id y t?£> 


Prnhlomc 


• build a profile 


• student reluctance 


" yUlUc IU \AJ\ I IJjyiol IOC 


cyu iiiyj tut ouiiic 


• nper learnino 

UVwl IVUl 1 III IU 


• not taken seriously 


• offopfivo Ipprninn 

CIICUUVC ICCUIllliy 


ui ii ecu 


• instant feedback 


• traumatic, subjective 


2 ORAL PRESENTATION 

mm %^ 1 1 * \ mm I II IhWmI V 1 # 1 I 1 I ^ 


- on bphalf of s nrniin 

\j i J i^v* Mull \j\ ci yi ^Um 


What it assesses 




Group Work 


Individual 


• content against objectives 


• presentation style 


• design and organisation 


• voice 


• time management 


• use of audio-visual aids 


• visual aids 




• use of sources 




• ability to answer questions 




How it helps learning 




• immediate feedback 




• group learning 




• peer tutoring 




• peer assessment 




Advantages 


Problems 


• validity 


• time consuming 


• good learning environment 


• free riders 




• design consistency 


3 SELF-ASSESSMENT 




What it assesses 




An example: bricklayers and total quality management programme 


How it helps learning 




ownership 




involvement 




Advantages Disadvantages 


• involvement leads to motivation 


how will they judge for themselves 


• realisation of responsibility 


their ability to implement their learning? 


• focuses responsibility 


cost of establishing competence criteria 


• another ethic throughout 


being adhered to? 


the organisation 


policing outcomes 




abdication of responsibility? 




honesty? cover up mistakes? 



BS5750 FOR ASSESSMENT 



165 



4 STUDENT SELF-ASSESSMENT 

Examples: portfolio of work, exhibitions, 

microteaching, teaching a lesson 

What it assesses 

• students' perceptions of their own learning 

• ail areas (Bloom) 

How it helps learning 

• leads to learner autonomy 

Advantages Disadvantages 

• reflective • abuse? 

• diagnostic • subjectivity 

• developmental • false humility 

• load taken from tutor • subversive 



5 PERFORMANCE ASSESSMENT 

What is assessed? 

conformity to norms: job-related criteria 

How it helps learning 

it can help learning, but it aims: 

• to qualify, 

• to certify competence, 

• to verify. 

Advantages Disadvantages 

• clarity • time to establish 

• a working tool 



6 SKILLS ASSESSMENT 

• skills being assessed 

• conditions specified 

• standards of achievement identified 

What is assessed - ability to carry out a task or perform a skill 

How it helps learning 

• sequences / prioritises the learning process 

• identifies difficulties 

Advantages Disadvantages 

• Indicates skills • resource demands heavy 

• "1 can do" / "feel good" factor • resource driven 



Figure 26.1 Fitness for purpose of different types of assessment 



It should be remembered, however, that the participants themselves represented learners 
who had succeeded. Their views would therefore be expected to be considerably more 
positive than an average group of learners, whose future careers may still depend upon 
what happens next time they are assessed. 



ERJC 



171 

* 



QUALITY IN LEARNING AND ASSESSMENT 



OO 

oo o> 



.2 

CQ 

"5 



o 



fc oo 



CO 



•55 po 



e 

00 

Si 

8:2 



0) 
0) 

e 

o 

oo 



i§.I 

oo - £ 

Q) oo t 

oo to oo 

oo gj oo 

CO O Q> 
CO 



o ^ 

CO 



0) 
o 5 

IS 



o 



co o 
cp 

> CO CO 

III 



^ QQ O Q UJ 



5 
1 

i 



CO 

ml 

oo 

Q> 

CO 

1 



1 



o 
c 

oo 

C 

"5 



Q> 



CO u. 



li 

"S o 

° E 
I J! 

> ed 




C O 




I a, 



E 

.S 



Hffi 



5 2 



M 1 



3 «U 

a. 8 



"8 



g 

a. 



■8 

US 



3 S! 

TO 3 

il 

■a 8 

l a 

3-S 

lit 



o 



1 

1 1 



> _ 



3 

o 
Z-S 



2 



BS5750 FOR ASSESSMENT 



0) 

s 
I 

£ 



q> 
£ 

Q S 



o 
c 



S 
fi 

«) 

I 

"5 

Q> 

0) £ 



1 

V) 



22 



3 



JJ GO 



PS25 



> C3 4J 

.a 8 ° 
is 

II § 



c 

o 

e 
8. 



o >> 
2 E 



5 



-a 

I 

to 



I 



2*53 
2.S 

ill 

cq5 



Iff s 

fill 

3 » o § 

■gall 



S3 

C 



168 



QUALITY IN LEARNING AND ASSESSMENT 



Uj 



5 

CtJ 

§ 

CCi 

i 



<U 

tfi 
Q> 
V) 
V) 
CCi 

</> 

i 

CO 



o 

C 

<) 
£ 

OUL 



S 
O 

c 

CJ> 



E 

V) 
CD 
V> 
V) 



3 



o ^ 
o c 



> 

. . <-t-l 

4=1 o . 



7 



8 



gx: 
o 

w c 

i o 

o "o 
«S £ 

ST" 

Is £ 
§ « 2 



«3 § H 

"Bio 
ex 0 «2 

JC > 3 
O O ~ 



■5 

s 

^ £ SS 

on w C 




| 3 

~ 3 

O o 

f = 
II 



c 
o 



1 



B.-8 

C/} (/! 



OS -5 



< O 

- -a 

{'§> 

w> S3 
2 

2^ 
-5 o 



ex. 



•8 
"S 

X) 

o . 
i— > <u 



5 *Sb 
0£ 




Ho 

8| 

•2 g 8 

2 *o 8 



3 32 
o <5 

c/J 



73 



JO 01 



It 



BS5750 FOR ASSESSMENT 



169 



5 
S 

I 

a 
3 



5c 
o 
c 



5 
S 

& 
"5 

CO 
OQ U. 



O 2 



s 

n 

.1 



■5 
o 



i 



c 

3 

£ 
•5 
o 



c 

l™ 1 




■fl 

=a-s 
3 "2 

"S'S 

si 

OS 



ea 

5 



■8 



S3 

CX. ' 



5 3 
as 

q o 



c c 



c 
a> 

CO 

co 
x: 
o 

3 
CO 



c 

CD 

E 

to 

CO 
CD 



-J) 
CO 

E 



cO 
c 
o 



03 



O 

CD 

CD 
JZ 



CO 



cn 
c 

S 

E 
l_ 

CD 

C 
3 

X> 
CO 
X) 
O 

Q. 

CD 

CO 

CD 
i_ 

O 

CD 

x: 
"O 

c 

CO 

> 

CO 

o> 

CD 



to 

IS 

> CO 
CD C 

© JE 

co — 

3 C 
CO CO 

o _ 

CD CO 

■? co 

>s CD 

bI 

CO © 
CD 

.52 o 

. CD 
CD .- 
> 

O CP 

■D 0) 
CD Q_ 



CD 
X: 



CD O 

cn*— 

"O CD 

© 5 



s_ E "Z 



o 

CO 



iS c ^ to 
« O w P 



CD ** 
3 £ 



s 



3 

X5 
CD 
O 
O 



0) 



o 



CD 

£ O Q- 



CD 
CO 



"O 

c: 



CD 
> 
CD 



to 

CO 
CD 
CO 
CO 

O CD 



tj to 



o S> 

x: g 

to E 

5 c 

o © 

X) 

CO o 
CD 



S2f 

CD 

E 

to 



to CO tO 



E S 

CO »_ 
tO m 
CD = 

o x: 
to o 

CD ^ 

a. E 
*co 

c >> 
0) CO 
J- 
CD 



O 

T3 

CD 
CD • 

£ 8 
E 8 
a>.° 
•|T5 

ai'g § 
g> go 
a) c 

" £ "co 
« S x: 

r E I 

co £ • 

a 2 to 
I? 8 

C c O 
CD CD X 

o-r 3 

to 



o i2 

S. 5 

c E^ 

> £ CD 

> o c 
£ to 

III 



CD o 

e £ 



- CD 



co -Q to 

\_ Q) C 

CD 2s to 

5>5 S- 



CD CD 

ii 

CD ^2 
CO 3 

"co .E 
c c 

% CO 



CD " 
X) 



8 S « 

w .£ ® o 

.2, E oj - 

2, Q) CD 

S?— >- T3 

.£ oj « r 

I* « S 

a o •« 



O « (0 

« 8 « 

§ ° 3 
o to <5 

sli 

H to • 

CO ^ £ 

C 5 - 
0) CO - 



2 tD 

«fc x: 
I c- 

O tD 

o ~ 
o 7* 

O) tD 
c > 

.= CD 

co £. 

3 ^ 



3 CD 

CO kl 

CO CO 

£ CD 

^ c 



a> 

to . 

is 



CD 



CD CO 

x: x 



co 

CD 



to $ w 

CT CD 

£ H co O co 



CD 
O to 

i to cS 

- CD CO 

© s * 

o co 

C CO CO 

o c > 

J- Q.£ 

(D O m 
t: tO 

P CL> 

m «> 



T3 CD 
CD C 

Q. CO 

E © 



o .t; 
E § 

CD <P 

■o o 

CO Q. 

Z q- 

H ° 

• O) 

CD •- 
CO c 

W CO 



CD 



CO ^ 
CD * 

if 



CD 



CO _ 
CO T3 

tD 

O) CO 

c to 

'tp p 



170 



QUALITY IN LEARNING AND ASSESSMENT 



LAYING DOWN THE LAW 

Participants were again divided into syndicates, and asked to produce and overhead 
transparency giving their recommendations for "criteria for quality in assessment". Each 
syndicate presented its acetate briefly to the whole group. The following are transcripts 
of the acetates produced. 

1 Recommendations fur Assessment 

1 Of both process and product 

2 Emphasis on verification/certification of product 

It must: 

• be standardised - as a process 

• give practical feedback 

• be standardised against established criteria/objectives 

• be valid/content/predictive 

• be reliable 

• be appropriate to the task 

• be easy and practical to use 

2 Laying Down the Laws 

a) Never use just one method to assess anything - use as many as possible. 

b) Avoid numbers (or letters) as "marks" where possible. Use words. 

c) Be diagnostic where possible, not normative. (Act on the diagnosis). 

d) Be student-centred where possible (Profiles, not single shots). 

e) Explain all criteria used beforehand to the students. If possible, negotiate them 
with the students. 

f) Spread the lead on both students and staff. 

g) If judgemental, be gentle. 

h) Failure -> diagnosis counselling -> opportunity -> success. 

3 Criteria for High Quality Assessment 

• Exams scrapped! 

• Honest 

• Fast feedback 

• Concise, no jargon 

• Fair 

• Encouraging, confidence-building 

• Achievable 

• Continuous 

• Not threatening 

• Learning experience 

• Flexible 

4 Criteria for Quality Assessment 

1 Link assessment to outcomes of teaching - ie. what did you plan for students to 



ISO 



BS5750 FOR ASSESSMENT 



171 



know, understand and become able to do? 

2 Use assessment to: 

• identify what students may need to learn 

• celebrate what students have achieved 

• diagnose students' problems 

• suggest how students can improve (formative) 

• summarise what students know at the end of a course 

• give information to future employers. 

3 Use different assessment techniques to match different course approaches. Don't 
rely on one method of assessment. 



valid 



Aims of 
rse 

ermine 



When? 

During? 
After? 
Mix? 




Why? 
(see 
Phil's 
paper) 

not necessarily 
the easy option 



\ 



knowledge 

Itt/tSdes Product? 
attituaes p rocesS es-a 



student input 
involvement 



Contract 



students 
understand 



<D Rachel, Julio, Andrew 
and David 



Figure 26.2 Rules for assessment 



172 



QUALITY IN LEARNING AND ASSESSMENT 



5 Laying Down the Law: Asking the Right Questions about Assessment? 

• Is it necessary? 

• Is it relevant to the course objectives? 

• Is it valid? 

• Is it accurate and reliable? 

• How does it build confidence? 

• How does it lead to learning? 

• How does it reinforce learning? 

• Does the assessment drive the curriculum, or the curriculum drive the assessment? 

• Is it fair to different students? 

• Can the students practice? 

• Can students know the criteria? 

• Can students influence the criteria? 

• Can quality feedback be given to students? 

• Can the results of the assessment be used to redesign the assessment strategy and 
the teaching/learning strategy? 

6 A Visual Approach 

Figure 26.2 shows one syndicate's visual approach to "laying down the laws". 

CONCLUSIONS 

I had fun running this workshop, and I think participants rather enjoyed it as well! It was 
a "busy" workshop; participants produced a great deal of material in the 90 minutes 
involved. Although delegates to an international conference on education and training 
are somewhat self-selecting in that they are people whose education and training 
experiences lend to have been highly successful, I hope that this collection of their 
wisdom and experience may be useful in helping to make assessment processes in general 
better-suited to their purposes. I hope that in particular, we may move further towards 
using assessment as a valuable part of the learning process, rather than a way of 
measuring how far the horse may have bolted after leaving the stable door unlocked! 

Acknowledgements 

I am grateful to all the participants who took part in this workshop: 

J Arter, A F Bickerton, Brian Canniford, Chu-Poon Yap Ching, Julie Clerk, Paul 
Ellis, Chahid Fourali, J Greenacre, Rachel Hudson, Steve Leary, Peter Leckstein, Jill 
Lloyd, Patricia McCarron, Chris O'Hagan, Jack Oakley, Michael Owen, Larry 
Roberts, Bob Sayer, Michael Shoolbred, Prof A R Sykes, Christine Tan. 



1 fie 



Section 5: Quality in Teaching and Training 



The papers in this section fall into two groups: three address the quality of teaching in 
higher education, and two of these consider the role of teacher appraisal in the process 
of developing quality. The clear message conveyed is that quality will be enhanced only 
if those seeking to bring about change understand and acknowledge the meanings teachers 
give to their professional activities and experiences. The remaining three papers describe 
applications of technology to training and total quality approaches to managing the design 
and development of CBT. 

In "Eating frogs and bridging gaps - post-Warnock conditions for teaching quality", Clive 
Colling explores the implications of the five conditions for teaching quality in higher 
education identified in the report of the Committee of Inquiry set up by the Polytechnics 
and Colleges Funding Council. His observations are important and timely, given the 
context of recent increases in participation rates in this sector. 

David Jones and Janet Hanson take up the challenge of the Warnock Report and 
demonstrate in their paper, "Teaching quality - 'I know it when I see it!'", how one 
higher education institution has set about identifying, supporting and developing good 
teaching practice. They outline a scheme for the appraisal of teaching practice, report the 
opinions of participants, identify proposed changes to the scheme in the light of the pilot 
and address wider concerns expressed about teaching appraisal in the education literature. 
In "Upward appraisal and its implications for higher education", Gerard McElwee and 
colleagues report the use of a questionnaire based on a ten-dimensional model of quality, 
developed for the service sector, in appraising lecturer performance. Although they focus 
mainly on the reactions of a sample of lecturers in a UK Business School to the use of 
students as appraisers, they also address the consequences of upward appraisal from the 
student point of view and indicate how upward appraisal data can be used to enhance the 
quality of teaching and learning. 

In "A total quality approach to managing CBT development", Steven Shaw and David 
Shaw describe techniques and approaches for reducing cost, improving quality and 
assessing quality in CBT. They provide a useful and detailed guide to quality 
management methodology that could be implemented at the level of a training division 
or department, even if a commitment to TQM has not been embraced at a higher, 
strategic level by the organisation as a whole. Meurig Williams and Graham Carr 
show in their paper, "Towards quality management in training design", how an integrated 
quality management policy is applied throughout the training development cycle, and 
across all training media, by the Training Development Group at Lloyds Bank. They also 
present a case study which shows how quality management tools are used in CBT 
courseware production. 

Feter Willis and John Eary, in "Achieving quality in networked interactive video", 
describe an award-winning multimedia system for training police supervisory officers in 
crowd control procedures at major spectator events such as football matches. They 
outline the quality approach in the design of the system, describe the equipment, 
functional specifications and the system -in -use. and identify its potential for management 
training. 



1*3 



27 Eating frogs and bridging gaps - post- 
Warnock conditions for teaching 
quality 

Clive Colling, University of Northumbria, Newcastle-upon-Tyne, NE1 8ST, UK 
THE QUALITY DEBATE 

People have very different views about what constitutes "quality" in higher education. 
Some believe the provision of material resources/equipment, buildings, etc to be a major 
criterion; others highlight the qualifications of staff and the examination results of 
students' Evidence of innovation, staff development, employer liaison and the pastoral 
care of students are also considered to be important indicators of quality. 

But at the point of delivery of any higher education system must be high quality 
curriculum content, effective and meaningful assessment procedures, and high quality 
teaching. Good teaching promotes good student learning, and exceptionally good 
teaching encourages students to be independent learners who know how to apply their 
knowledge and skills in practice, and consequently how to leam for themselves. 
Promoting learning by enabling students to "learn how to learn" will be the major 
criterion for successful professional teaching activity in higher education in the 90's. But 
achieving efficiency of teaching with effectiveness - for example, coping with high 
numbers of students for less time in restricted space - will be a major dilemma. Rarely 
has a decade started with so many challenging prospects for managers and teachers m 
higher education in the UK. 

Eighty-seven thousand more students, 41.560 of them full-time, are projected over the 
next three years: a remarkable scenario, exceeding even the Government's expectations 
on greater access. Annual rates of growth of up to 33% are predicted by 24 former 
Polytechnic and Colleges Funding Council sector institutions, including five former 
polytechnics. Against this backdrop, doing nothing about teaching quality is not an 
option. There is an imperative for higher education institutions to accommodate students 
with a wider range of academic and practical experience than before, many of whom will 
not have the traditional qualifications for entry. Not only will entry requirements and 
procedures have to change, but teaching methods and design of courses will have to meet 
the needs of the new tvpes of student. The Government believes that increased 
participation in higher education need not be at the expense of academic excellence; but 
it will be for the HEI's themselves to find ways of meeting the difficult challenge of 
maintaining (and enhancing?) quality standards in the 1990s. 

Universities and Colleges therefore face a fundamental change, in the next decade, if 
the Government's aim of increasing access to higher education is to be achieved. All 
institutions need as a matter of urgency to find ways by which they can ensure that an 
influx of more students will not adversely affect the quality of the education that is 
offered. "Teaching quality" thus takes on a new significance. 

FOCUS ON TEACHING 

In 1988, the then Secretary of State, in a letter to the Chairman of PCFC, confirmed the 
Government's concern that teaching quality should be taken into consideration by tin* 
Council. He wrote "I look to the Council to develop further indicators of both the quality 
and quantity of institutions' teaching and should be grateful if it would ... consider how 



18* 



176 



QUALITY IN TEACHING AND TRAINING 



these might be used as an input to its funding policies and decisions". While research 
has for generations been evaluated in higher education funding decisions, and its quality 
and efficiency both subject to review, teaching had not been subject to this kind of 
scrutiny. Now it was quite clear that the quality of teaching was to be one of the criteria 
employed in the allocation of funds. 

The Committee of Enquiry into Teaching Quality, chaired by Baroness Warnock, was 
set up by the Polytechnic and Colleges' Funding Council in January 1989. The terms of 
reference were: 

a) to identify characteristics of effective and efficient teaching; 

b) to identify which of these characteristics can be developed as indicators of 
teaching quality; 

c) to suggest the means by which institutions could demonstrate the effectiveness and 
efficiency of their teaching and the promotion of students' learning; 

d) to advise on possible strategies for the Council which would serve to raise the 
quality of teaching. These might include procedures for monitoring and evaluating 
outcomes, and funding mechanisms (PCFC 1990). 

The Warnock Committee met in April 1989, and held nine further meetings. In May 
1989, 84 PCFC institutions were invited to respond to three questions: 

a) how does your institution identify efficient and effective teaching? 

b) what is your institution doing to preserve high quality work once it has been 
identified? 

c) what is it doing to improve quality in those areas which have not been so 



Responses were received from 57 institutions. The Committee analysed the responses, 
and followed them up with a programme of visits, covering modular course arrangements, 
mature students, the teaching of engineering and modem languages. Enterprise Initiative 
developments, initial teacher training and the teaching of humanities. Institutions visited 
included a specialist arts institution, an institution that works with more than one 
validating body, and several which had educational methods or development units. 

A review of relevant teaching and learning literature was commissioned. A series of 
consultations and discussions took place with representatives from industry, commerce 
and the professional bodies. The key points in the Committee's report (PCFC 1990) are: 

a) that teaching must be interpreted broadly, as the initiation and management of 
student learning by a teacher; that it must be responsive to student needs; and that 
the conditions necessary for good teaching must be a priority at every level of the 
institution; 

b) teaching must be judged "good" by whether it contributes to the purposes of 
higher education - the life-chances of the students; excellence in teaching could 
be traced to the ethos of the whole institution; 

c) five necessary but inter-related conditions must be fulfilled before teaching can 
be judged good: 

• clarity of aims and objectives related specifically to teaching, and confidence 
that they are worthwhile and appropriate to students' needs; 

• a policy regarding curriculum organisation and delivery, including a readiness 
to consider different methods of promoting learning; 

• a policy for professional development of teaching staff, including appointment. 



identified? 




185 



CONDITIONS FOR TEACHING QUALITY 



177 



induction, appraisal and development; 

♦ means by which the views of students and employers can be used in 
judgement of the curriculum, its delivery and outcomes; 

♦ an identifiable framework within which an institution can evaluate its own 
success in meeting its objectives and adjust its practice accordingly. 



John F Kennedy said, "Change is a way of life. Those who look to the past or present 
will miss the future". Translating the recommendations of the W a mock Committee into 
practical action in classrooms, labs and studios across the former PCFC higher education 
sector will require change. Change is constant in many enterprises but, as somebody 
once said, in education, interpreting change is rather like trying to grasp fog. A head 
teacher friend once likened managing educational change to trying to build a ship - at sea, 
from the keel up! 

It is important that any change to professional teaching practices in higher education is 
seen to be constructive and positive; supporting and encouraging existing good teaching 
practice, rather than trying to revolutionise processes that have been going on for a long 
time. Certainly, some old assumptions may need to be tossed out - does one hour's 
lecturing automatically produce one hour's learning? - but there should be no pig-headed 
insistence on change for "flavour of the month" purposes. A lot of excellent educational 
practice exists across student learning outcomes. It is a matter of: 

♦ examining and knowing what we actually do, as teachers and managers of learning 
resources; 

♦ learning from the past - both the good and the bad; 

♦ working together to promote growth (do we really collaborate effectively, as 
professionals?); 

♦ knowing and using our limited resources effectively; 

♦ assessing our own performance; 

♦ using the corporate aim to put students first, and keep our thinking focused on the 
institutions' performance. 

A first scan of Warnock's five conditions may produce the response "We've been doing 
this for years!" from some teaching colleagues. But "doing this" sometimes equates to 
symbolic action, undertaken and written about in order to satisfy a plethora of 
committees. Literally interpreting the tasks encompassed in the five conditions requires 
a professional engagement with difficult tasks, and the raising of hard questions. A brief 
"Cooks Tour" may illustrate my point. 



Teaching aims and objectives should be clear, worthwhile and appropriate to 
students* needs. 

What do "clear", "worthwhile" and "appropriate to students' needs" mea.i for individual 
teachers, their courses, their students and their particular employers? One key 
recognisable characteristic of quality must be relevance. A key quality process must be 
to do with listening. Listening is strategic! The student perception is a reality. We need 
deliberate self-critical analysis of teaching practices and procedures in order to continually 



RHETORIC INTO REALITY 



CONDITION 1 




178 



QUALITY IN TEACHING AND TRAINING 



seek to improve what we provide for students. Does our teaching promote learning as 
part of a high quality student experience? Do we listen to what students say? Do we hear 
what they say. when they say it? Do we do anything as a result? There will need to be 
a partnership in deciding what is "clear", "worthwhile" and "appropriate". 



Curriculum policy should encourage new methods of promoting learning. 

There are extraordinarily innovative educational practices going on across the former 
PCFC sector. The problem in the present competitive climate is finding ways for 
professional teachers to share good practical ideas. The point is, many institutions are 
hotbeds of experimental activity in teaching and learning, and the Wamock 
recommendations are quite clearly supporting and encouraging constructive change and 
investigative educational development. To stand any chance of claiming extra cash for 
teaching quality, institutions will need to be able to demonstrate that the education they 
offer will enhance students' understanding and imagination, and "excite their intellectual 
curiosity and expand their knowledge and skills". Simply floating the usual papers at 
review and validation time won't be enough. It's not just a matter of what course teams 
feel, think and know about the curriculum, it's what they do. Are they promoting active 
learning? Live projects? Independent, autonomous learning? Peer assessment? 
Collaborative group work? etc, etc. 



Professional development policy should include appointment, induction, appraisal 
and development. 

For many of us, tiiis is the really good stuff. The Wamock Report recommends that 
induction courses for teaching staff should be compulsory for all teachers in their first 
year and that induction should continue beyond an initial course. Teaching methods, 
strategies and tactics should be all built in here. The Report is giving a hefty shove to 
the whole idea that teaching should be way up on institutional agendas. It is saying that 
there should be visible reward systems for good teaching and that senior management 
should support strategies aimed at raising the general level of teaching. The message is 
that full-time and part-time teaching staff should have access to support for their own 
professional development. And we are not talking about some weekly "Mary Poppins 
good time hour" here, but properly structured, organised, funded and delivered staff 
development programmes. (For the checklist of important criteria, for interpreting 
condition 3, take a look at page 49 of the Wamock Report.) 



Students and employers should contribute to the judgement of curriculum design, 
delivery and outcomes. 

Many courses in many institutions have for years actively involved students and potential 
employers in educational development decision-making. Perhaps others have simply 
talked or written about involving them. The message from Wamock is quite clear: 
institutions should commit resources to actively seeking the views of employers, 



CONDITION 2 



CONDITION 3 



CONDITION 4 




CONDITIONS FOR TEACHING QUALITY 



179 



professional bodies, staff and students about the needs, requirements "and imperatives" 
for each group. (Again a checklist of possible criteria can be found on page 50 of the 
Report ) 

No one would claim that there is a "handy dandy" quick-fit way of involving students 
and employers in judgements about curricula, delivery and outcomes. We really do need 
innovations and intelligent thinking in this area. It just isn't feasible to produce 
thousands of feedback questionnaires for example, which nobody ever analyses or 
interprets, and which have no status beyond being "draft" excluders in small overcrowded 
staff rooms. The views of students and employers need to be used to "actively improve 
the planning and delivery of academic programmes. 

CONDITION 5 

Institutions should evaluate their attainment of their objectives, and adjust their 
practice accordingly. 

Throughout the years of CNAA rule, courses, departments and institutions have stated 
objectives. The "objectives" business has become a fine art. Those colleagues who can 
string the right fine words together for course documents are in much demand at review 
and validation times. "Look at this Linda! 'Collaborative, activity-based, student-centred, 
formative assessment procedures will underpin course^ principles for high quality 
pedagogic action'. It's a wrap, they've got the lot there!" 

But promoting and supporting institutional self -evaluation is about making objectives 
operational, and negotiating what they mean in action. What are the unintended outcomes 
of stated objectives? How should they be reviewed and redefined? What are the key 
criteria and indicators of true progress and improvement? What sort of evaluation 
evidence will serve as a basis for claims of "outstanding" quality m the next funding 

round? . ~ f 

HEIs have fulfilled their obligations for stringent quality assurance to the Council tor 
National Academic Awards for years. Peer validation review procedures have done an 
excellent job in sustaining and monitoring quality for students across the former PCFC 
sector Wamock's fifth condition is, however, seeking to build on existing review 
practice, by recommending to the Funding Council that it looks for specific evidence of 
commitment to teaching quality enhancement not just at course and departmental level, 
and not just on paper, but throughout the management and practice of the institution. 
This is a harder edged perspective, and is underpinned by the principle stated in the 
Report that the conditions necessary for good teaching must be given priority at every 
level of the institution. 



References 

Polytechnic and Colleges Funding Council 
committee of enquiry PCFC Bristol 



(1990) Teaching Quality: report of the 



EJJC 



183 



28 Teaching quality - "I know it when I see 
it!" 



David Jones and Janet Hanson, Bournemouth University, Bournemouth, BH125BB, 
UK 

INTRODUCTION 

In October 1990 the Wamock Report on teaching quality threw down the gauntlet, 
sheathed in a velvet glove, challenging higher education institutions to demonstrate how 
they could identify, support and develop good teaching practice. This contribution 
demonstrates how one institution, Bournemouth Polytechnic (now Bournemouth 
University), took up this challenge. An Appraisal of Teaching Practice scheme was 
developed and piloted in two departments in Autumn 1991. 

The scheme devised is both developmental and evaluative. It aims both to enhance 
teaching quality by identifying good practice and facilitating the development of 
appropriate staff development, and to allow the institution's Academic Board to obtain 
an aggregate view of the teaching quality of each department. It involves the judgement 
of a peer and the individual lecturer's head of department. Both observe two teaching 
sessions and give verbal and written feedback to the lecturer concerned. 

This case study provides an insight into the main issues involved in the observation of 
classroom practice in a higher education environment and discusses the practicalities 
surrounding its implementation. 

DESIGN OF THE PILOT 

At the start of the pilot we suggested three aims for the exercise: 

♦ all panics to emerge from the process with their dignity and integrity intact; 

♦ that the experience should lead to a continuing improvement in teaching; 

♦ that we should be in a better position to advise on the design and implementation 
of a future system which would achieve the first two aims. 

The overall response emerging from the review of the pilot suggested that we were not 
far away from achieving the first two goals and we also think that we have found ways 
to improve on the initial appraisal of teaching quality pilot. 

The pilot involved staff from the departments of Business Information Systems and 
Product Design and Manufacture and took place over a period of three weeks between 
7 and 25 October 1991. Involvement was voluntary and 27 out of 29 staff from the two 
departments took part as appraisers. 

The appraisers comprised the five heads/associate heads of the two departments 
(A/HODs) and 13 experienced members of the Course Evaluation Group (CEG). This 
group acts as a peer review body for course approval and monitoring and was therefore 
the natural place to look for peer appraisers of teaching quality. Indeed, members are 
frequently appointed to this group on the basis of recognition of their teaching expertise. 
The head of department also seemed a natural choice for appraiser. As the lecturer's line 
manager, he has responsibility for assisting the individual to make use of staff 
development opportunities. 

Two observation sessions per member of staff were planned, both to be observed by the 
appraisce's A/HOD and a member of the CEG. One session was selected by the 



TEACHING QUALITY 



181 



appraisee and the other by the A/HOD. We decided that, if possible, each appraisee 
should have the same CEG appraiser observe both sessions. 

Following early discussion with the departments involved, we varied the pilot to include 
a group who would be appraised for one of the sessions by a subject specialist instead 
of the A/HOD This group was selected on a random basis and those selected were asked 
to nominate a subject specialist appraiser. Four of these appraisers were eventually 
nominated. In total, 54 sessions were arranged and the responses from the evaluation 
questionnaire covered 35 sessions. 

The observation criteria agreed by Academic Board were: 

1 Organisational and structure of session 

2 Clarity of presentation and use of resources 

3 Presentational style and classroom management 

4 Monitoring of effectiveness 

5 Whether the aims of the session have been met 

Three staff development workshops were held for the appraisces and two for the 
appraisers. These were designed and managed by the authors. 



The appraisal of teaching process involved four key stages for each session appraised: 

1 Briefing meeting 

2 Observed session 

3 Verbal feedback 

4 Written feedback 

1 The briefing meeting between appraisers and appraisee which took place before the 
observed session was regarded as an integral part of the appraisal process. The 
appraisee was able to set the scene in terms of such matters as the stage of 
development of the course, the objectives of the session, and the student group. The 
appraisee was also able to suggest particular features which he/she would like the 
appraisers to focus on during the observation. Guidelines on observing teaching 
behaviour and on giving and receiving feedback were provided to all taking part. 

2 The observations then took place and the appraisers' comments were recorded on the 
observation record designed for the pilot. 

3 Verbal feedback and discussion of the session with the appraisee took place as soon 
after the observation as possible. 

4 The written feedback record provided the formal record of the process of 
observation and feedback and included suggestions for future action. It was 
designed lo be completed jointly by the appraisers and sometime after the verbal 
feedback sessions. A copy of the record was then sent to the appraisee who signed 
it and if appropriate added further comments before sending it to their head o f 
department. 



At the end of the three weeks of observation activity a review of the exercise was carried 
out. The review was based on a questionnaire citculated to all participants who provided 



APPRAISAL PROCESS AND DOCUMENTATION 



REVIEW OF THE PILOT 




182 



QUALITY IN TEACHING AND TRAINING 



individual and anonymous comment on a range of matters. When the results of the 
questionnaires had been collated, meetings were held with groups of appraisers and 
appraisees to check the findings. 

The questionnaire asked for specific information about the details of how many sessions 
were observed, the type of sessions and how long the process took. It also sought the 
opinions of appraisers and appraisees on several aspects of the process, including their 
experience of observing and being observed, the adequacy of the preparation for the 
appraisal scheme and any suggestions for changes. 

The response rate was high. 21 replies were received from 27 appraisees (78%) and 
18 replies were received from 18 appraisers (100%). 

Comments About the Briefing and Feedback Sessions 

We had stressed that these were an integral part of the appraisal process so we wanted 
to find out how well they had been carried out and to what extent the appraisee had been 
encouraged to take the initiative in asking the appraisers to focus on specific features of 
the session. In the event, the problems of matching the timetable of the appraisees and 
the appraisers meant that only 20% (4) of the appraisees had the opportunity of holding 
a briefing session prior to both observations though 70% (14) had one briefing session. 
66% of the appraisees felt they had been given an opportunity to ask the appraiser to 
focus on a specific feature of the teaching session. 

Comments About the Immediacy of Feedback and Follow-up 
Action 

The literature on feedback underlines the value of a quick response and over 95% of 
those appraisees who responded said they received verbal feedback quickly after each 
session. The written feedback was not so immediate and frequently not provided. 

Giving Feedback 

Most appraisers appear to have followed the guidelines on giving feedback in that 75% 
(15) of the appraisees who were observed reported that they had an opportunity to put 
their ow>? views first; 80% (16) said they were given an opportunity to seek clarification 
and all those observed felt that their views were valued. 

Comments on the Observation Criteria 

The observation criteria were found to be helpful by 68% of 1 13/19] appraisees and 71% 
[12/17] appraisers in preparing for the session. 

Comments on the Validity and Utility of the Appraisers' 
Observations and Feedback 

During the preparation sessions some concerns were expressed about the relative value 
of specialist as opposed to non-specialist feedback. Indeed the pilot design was varied 
to allow a small group of observations to be undertaken by subject specialists nominated 
by the appraisee. 




TEACHING QUALITY 



183 



The results of the questionnaire showed little variation between the rating of the Course 
Evaluation Group members, the HOD/AHOD and the subject specialists. This perception 
is confirmed by the appraisers, in 66% (12/18) of the cases appraisers reported that their 
comments always matched those of the second appraiser. 

Comments on the Overall Quality of the Appraisers 

Despite the overall endorsement of the validity and the utility of the appraisers* 
observations, some concern was expressed by the appraisees about the quality of the 
appraisers. The concern centred on the need for objectivity. Many other characteristics 
and qualities emerged in response to the question which was asked of appraisees and 
appraisers: "What are the qualities of a good appraiser of teaching practice?" 

There was encouragingly a good deal of overlap between the set of responses. 

Sample of appraisees* comments: "Willingness to discuss with humility not desire to 
impose own style preferences" and "Someone who doesn't try to impose own teaching 
methods on a colleague". 

Sample of appraisers* comments: "Promote sessions as exchange of methods" and "To 
give constructive feedback". 

Comments on the Preparation Sessions 

The questionnaire data indicated that the training was successful in explaining the concept 
of teaching appraisal; 65% (13/20) of appraisees and 83% (15/18) appraisers agreed this 
to be the case. 

Personal Concerns 

The pilot scheme inevitably aroused much uncertainty among lecturers, so both sets of 
respondents were asked to say what concerned them most about the process and also to 
identify the principal benefits to counteract the feelings of uncertainty. 

Some clustering of views emerged from the individual comment of the appraisees (21 
respondents). These covered concerns about the increased administrative load connected 
with the scheme; the quality of the appraisers; the need for staff development support to 
underpin the development of teaching competence; the representativeness of the sample 
of observations; and the use to be made of the information collected about an individual's 
performance. 

Overall the comments of the appraisers (18 respondents) did not group as well as those 
of the appraisees, but concern was also expressed about the eventual use of the 
information and the additional administrative load. We return to these concerns when we 
consider the modifications we propose to make to the pilot scheme. 

It was clear that both appraisees and appraisers placed a high value on the outcomes of 
the appraisal process and saw its contribution in terms of improving practice, personal 
benefits and reflection. In answer to the question: "What do you think might be the 
benefit for your personally of the appraisal of teaching practice?" Appraisees* replies 
included: "Thinking and talking about my teaching with colleagues" and "Nothing but 
improvement in my teaching skills". 

It was also clear from individual comments that appraisers felt that the process was 
valuable in terms of their own learning: "Exchange of ideas and different perspectives" 
and "I leam for my own teaching". 





184 



QUALITY IN TEACHING AND TRAINING 



PROPOSED CHANGES TO THE APPRAISAL OF TEACHING 
PRACTICE SCHEME 



As a result of the pilot experience and the feedback provided by the review, we decided 
to make several changes to the appraisal process and the documentation should we run 
it again. 

In view of the relative agreement of views between the CEG appraisers and the 
A/HOD' s we decided that we could relax the requirement for both sessions to be 
observed by the A/HOD. A/HOD will only observe one session. 

We also felt that our choice of the CEG members as peer appraisers was justified since 
there was little difference in the validity and usefulness of their observations compared 
with those of the subject specialist observers. 

We felt that the observation criteria had not been sufficiently detailed to provide specific 
enough feedback to the appraisees or to be very helpful to the appraisers when planning 
the observation. A tool which some had found very useful was the list of "effective 
teacher behaviours" developed at Newcastle Polytechnic (Colling 1990). We felt it would 
be useful to extend the use of this tool but we did not wish it to be regarded as a detailed 
checklist. We decided to remove our existing criteria and invite appraisees and appraisers 
to negotiate criteria for each session. We suggest that the list of effective teaching 
behaviours can be used as a way of identifying specific teacher activity to be observed, 
which may differ in each session and that the list of effective teacher behaviours provides 
a common vocabulary during the verbal feedback session. 

We also felt that the list as published covers teaching behaviour which can be observed 
mainly at lectures and seminars but over a period of time we expect to build up a bank 
of additional activities and effective behaviours which apply to the teaching specific to 
this university. 

Very few appraisees received any written feedback and we realised that our method 
was not designed to encourage this, so in future we will advise appraisers to give verbal 
and written feedback during the feedback session. We have revised the written feedback 
form to encourage appraisees to use it first to reflect on their performance, and their 
comments are then followed by appraisers' comments. This means that only one piece 
of paper is in existence and it remains with the appraisee to pass it on to their HOD. The 
appraisers will also be urged to leave their observations records with the appraisee. We 
feel that these procedures should help to give the feeling to the appraisees that they are 
responsible for the flow *> c information, and to encourage ownership of the system. It 
should also help to prevent breach of confidentiality. Advice on these procedures will 
be gathered together to form a code of practice governing appropriate conduct for the 
appraisal of teaching practice. 



It is useful to draw some conclusions about this pilot in relation to the wider concerns 
expressed about teaching appraisal in the education literature. 

It is becoming recognised that general criteria for effective teaching can be identified, 
(Ramsden 1991) yet it is important to realise that it is not wise to import wholesale 
criteria from one institution to another. Each institution has to develop its own specific 
criteria and on this matter we agreed wholeheartedly with Seldin's comment that "the 
success or failure of the colleague observation programme is contingent on the common 
acceptance by the observer and teacher of the appropriateness, reasonableness and fairness 
of the rating instrument and its implementation" (Seldin 1980, quoted in Weimeret al 1988). 



CONCLUSIONS 



9 

ERLC 




TEACHING QUALITY 



185 



We feel that observation can be used as a developmental process in encouraging good 
teaching practice but that the observation records should not be the only source of 
information about teaching practice considered in the appraisal interview. We urged 
appraisees to collect information themselves about their teaching from their students and 
colleagues. As with any evaluation system, there should be multiple sources of input. 

We gained much good advice and examples of good practice from the literature on 
teaching appraisal in schools (Bollington and Bradley 1990), from management literature 
on the ways to give and receive feedback (Wood and Scott 1989) and from existing 
sources of advice on appraisal in higher education (Gibbs et al 1989). We recognised 
that in order to be' valid, comment must be based on observation not inference and 
furthermore, based on what was specifically observed during the two sessions. It was 
important not to make generalisations about the quality of teaching from these two 
sessions. 

We also felt that we proved that it is possible and useful to use appraisers outside the 
discipline areas of the appraisees and that their observations, although maybe different 
to a subject specialist's, were as valid. We felt that our system of monitoring and 
evaluation of courses provides us with other ways of checking the subject content of 
teaching. 

There are also features of the quality of the feedback which we found to be important. 
Feedback must be based on behaviour rather than personality, or what the teacher did 
rather than what the teacher is. Feedback should also be specific enough to allow the 
appraisee to formulate an action plan for self development. 

It is suggested that it is difficult to achieve inter-reliability between observers when 
many are used but we feel that it is preferable to disseminate good practice by using large 
numbers of appraisers rather than restricting the observations to a small group of 
appraisers, who may achieve high inter-reliability but at the expense of the feeling of 
ownership of the system which we want to encourage. 

References 

Bollington R and Bradley H (1990) Training for Appraisal: a set of distance learning 
materials Cambridge Institute of Education Cambridge 

Colling C (1990) Teaching Quality Matters Bulletin of Teaching and Learning Issue 
4 July pp 10-17 

Gibbs G, Habeshaw S and Habeshaw T (1989) 53 Interesting Ways to Appraise Your 
Teaching TES Bristol 

Polytechnic and Colleges Funding Council (1990) Teaching Quality: report of the 
committee of enquiry PCFC Bristol 

Ramsden P (1991) A Performance Indicator of Teaching Quality in Higher Education: 
the course experience questionnaire Studies in Higher Education 16(2) pp 129-150 

Seldin P (1980) Successful Faculty Evaluation Programs Coventry Press Crugers 

Weimer M G. Kems M M and Parreit J (1988) Instructional Observation: caveats, 
concerns, and ways to compensate Studies in Higher Education 13(3) pp 285-293 

Wood R and Scott A (1989) The Gentle Art of Feedback Personnel Management April 



194 



29 Upward appraisal and its implications 
for higher education 



Gerard McElwee and Tom Redman, David Eadson, Adrian Evans, Leeds 
Metropolitan University, Leeds, LSI 3HE and Teesside University, Middlesbrough, 
TS1 3BN, UK 

INTRODUCTION 

A number of potential appraisers are available for an organisation to appraise its 
professional staff. These include self, peers, managers, internal and external customers, 
and subordinates. In this paper we examine the potential for the use of upward appraisal, 
that is the use of data generated by and from students, in the assessment of lecturer 
performance. It remains an issue for others to decide, posing some interesting questions 
for educational philosophy along the way, whether upward appraisal of lecturers should 
be classified as appraisal by customers or subordinates! 

Upward appraisal is claimed to have many benefits for organisations. These include 
improving management style and people management, facilitating personal development, 
increasing productivity (Bernardin and Beatty 1987), enhancing the recruitment 
attractiveness of the organisation, and facilitating "voice" (Bernardin 1986) within the 
organisation. It is also claimed to have many advantages over traditional appraisal 
schemes. Firstly, by the very nature of their relationship students are usually in much 
greater contact with the lecturer than the line manager. They are thus in a good position 
to observe directly a large volume of lecturer behaviours. Secondly, such observations 
are from a unique vantage point - "the receiving end". Thirdly, it offers a multiple source 
of appraisal data. Lecturers have many students but usually only one direct line manager. 

Performance appraisal generally in the education sector has been particularly 
underdeveloped compared to industry. However, it seems that upward appraisal is one 
area where education has clearly led the field. For example, one study of the appraisal 
of lecturers in a UK university found some 29% of lecturers had experienced the use of 
information from students as part of an appraisal of their course/teaching (Rutherford 
1988). The use of student evaluations of lecturer performance has a long history of 
attracting the researcher's attention (eg Breed 1927). 

Although many aspects of student appraisals have been researched, there appears to be 
a number of "blind spots". Firstly, the studies have neglected the effects of upward 
appraisal of lecturing performance on the commitment of the students to their studies, 
course, institution etc. Secondly, the effects of upward appraisal on the lecturer's 
commitment and morale remain unexamined. Finally, most of the research that has been 
conducted is for research purposes only, yet it seems that appraisals differ markedly 
between the laboratory and real life (Banks and Murphy 1985). 

Our study of student appraisal of lecturer performance within a Business School seeks 
iO address some of these issues. We have examined, via the use of semi -structured 
interviews with lecturing staff, their experiences and perceptions of the use of appraisal 
data generated by students. We seek here to answer a number of questions about the use 
of upward appraisal for lecturers. How "acceptable" to lecturers is the use of student 
generated data in the appraisal process? For what purposes can student generated 
appraisal be used? Who should have access to the data? What are lecturers perceptions 
of students as fair and accurate raters of their performance? What are effects of upward 
appraisal on lecturer and student morale and commitment? 



UPWARD APPRAISAL 

METHODOLOGY 



187 



The QUALED pilot study, described below, identified the existence of gaps between 
students' and staff perceptions of attributes of Teesside Business School (TBS), and 
between staff and student expectation and perception of the activities of the School. It 
is argued that the existence of these gaps is of cause for concern as it is a source of 
dissatisfaction with the services provided. 

The objectives of the current study were to use the following criteria to measure the 
quality of educational provision: the concept of a gap analysis between perception and 
expectation of higher educational service; the ten dimensions of reliability, 
responsiveness, competence, access, courtesy, communication, credibility, security, 
understanding, and tangibles: the concepts of the SERVQUAL model as developed by 
Parasuraman and Zeithmal (1985). 

SERVQUAL was originally designed to assess the provision of services within the 
service sector. In effect it consisted of a 22-item, seven-point Likert scale. It was felt 
however that not all of the questions were directly applicable to higher education. A 35- 
item, seven-point scale was chosen. 

The questionnaire had three sections. The first section contains a series of questions 
relating to the normative expectations of quality provision in the TBS. The second 
section contains a series of questions relating to the perception of quality of services 
provided by TBS. The third section was designed to provide demographic and other data 
from the respondents. 

Although the items of the first and second sections were similar, as indicated above, the 
directions in each of the sections wert slightly different, in that in section A respondents 
were asked what they expected in regard to each item, whereas in section B respondents 
were asked to report their perception of the quality of service received on each item. To 
all intents and purposes this framework is not dissimilar to that chosen by Parasuraman, 
or that chosen by Saleh and Ryan (1991) in a subsequent adaptation of the SERVQUAL 
model to the hospitality industry. 

The sample consisted of 17 TBS staff and a cohort of 120 students on a BA (Hons) 
Business Studies degree. The questionnaires were completed by the respondents in the 
presence of, in the case of staff, student researchers, and in the case of students, the first 
author in a time allocated as a teaching period. The sample of 137 was comparable in 
size with other similar studies, Parasuraman and Zeithmal (1985) and Saleh and Ryan 
(1991). The survey was carried out over a two week period in February 1992. 

For this pilot study, our main concern was to provide a preliminary analysis that would 
test the hypothesised ten dimensions of quality derived from the SERVQUAL model. 
Factor analysis of the 35 items in the questionnaire (a principal factor analysis using an 
oblique (oblimin) rotation) revealed, however, only eight applicable factors for both 
expectation and perception of provision. It also indicated that these factors, in both cases, 
accounted for only 50% of the variance, and that the items relating to each of the 
SERVQUAL dimensions were spread across the factors. 

We need to investigate in more detail the measuring instrument. Considering that the 
questionnaire items were derived directly from the work of Parasuraman, it may be that 
they are not directly relevant to the higher education sector and will need further revision. 
Indeed, Parasuraman \s ten dimensional hypothesis of service quality may not be directly 
applicable to issues of quality in HE. As regards methodology, when applying factor 
analysis, and considering the number of items used, a much larger sample may be 
required and a "test/re- test" design may be more effective in distinguishing between 
expectation and experience. 



188 



QUALITY IN TEACHING AND TRAINING 



FINDINGS 

Firstly, it seems the majority of lecturers are in favour of using student generated data in 
appraising their performance. A number of these, perhaps to be expected in a Business 
School, expressed their approval couched in a language of customer care. For example, 
lecturers felt upward appraisal could measure "customer satisfaction", "client approval of 
the service provided", and provided a "key to delivering a quality product". One lecturer 
suggested that such a source of aopraisal data was more valuable than that provided by 
his line manager because "the customer is always right whereas the management here 
very rarely ever are". 

Particularly emphasised by those lecturers in favour of upward appraisal was the value 
to themselves of student feedback for personal development. A number of those 
interviewed stressed the value of student appraisals in providing a source of feedback on 
teaching behaviours they were completely unaware of. For example, one lecturer 
described how a student evaluation questionnaire had brought to his attention that he 
spent much of the time in lectures looking out of the window rather than at the class and 
that this was interpreted as a sign of his disinterest by the students. Thus upward 
appraisals possess the valuable potential to allow us a view of ourselves as lecturers as 
others see us. 

A small number of lecturers were on the other hand opposed to any use of such data 
in their performance appraisal and indeed to upward appraisals being conducted at all. 
A range of reasons were offered for this. Prominent here was a fear of what was 
described as the "get even factor". Lecturers were concerned with disaffected students, 
for example those who had scored badly in assessments, those who had personal grudges 
etc, using the vehicle of upward appraisal to give damaging and unfair ratings of their 
performance. One lecturer described how she felt upward appraisal would encourage 
students to become "over-litigious against some members of staff" and would thus "spend 
their whole time complaining rather than biting the bullet and getting on with it". 

A number of lecturers appeared to be "casualties " of the upward appraisal process. 
One lecturer described the damage to his morale and commitment to lecturing as a career 
that had been caused by previous "maulings" in student evaluations, so much so that a 
change in occupation had been seriously considered. Another lecturer described how he 
felt student evaluations he had personally experienced reduced to little more than 
"popularity contests 11 . They were considered as "providing an ego boost for some but the 
death knell to others". 

Another concern with using students as appraisers was that students have only a partial 
viewpoint and knowledge of the role of the lecturer and that upward appraisals should be 
carefully restricted to areas such as delivery, assessment, lecturer availability, 
approachability etc. A common view was the. the criteria used in student evaluations 
needs careful design, and it rarely receives this. Course tutors, those usually responsible 
for the design process, often used questionable evaluation instruments, usually obtained 
from prescriptive publications with very little consideration of customisation for the 
particular group/course to be surveyed A further worry was that many instruments 
measured little more than lecturer "entertainment value". 

Lecturer acceptance of the use of upward appraisal declined dramatically for purposes 
other than developmental activities, for example reward systems or promotion. When 
questioned on the acceptability of its use for forming a data source in formal "evaluation" 
type appraisal interviews support was much more limited. However, given such a 
position lecturers were generally fairly open about who should have access to upward 
appraisal data. Only a minority of lecturers thought it should by restricted to themselves 



JL K~' i 



UPWARD APPRAISAL 



189 



and their immediate line manager. Over half of our sample felt such information should 
be available to course boards and higher levels of management. 

Some concern emerged with how this information should be presented to course boards. 
One lecturer described what he felt was a rather insensitive handling of student 
evaluations where the results were aggregated and an overall rating was tabled as a 
standing item on the course board agenda. This practice caused particular concern as this 
very public arena was the first time that lecturers had access to what were occasionally 
very poor ratings. 

Interestingly from our findings, lecturers appeared to be more willing to share the data 
from upward appraisals with managers rather than students. A number of staff felt that 
none of the results of upward appraisal should be reported to students. Only half of those 
interviewed thought debriefing sessions between lecturers and students post appraisal was 
a useful exercise. On probing this issue, a prominent concern here was that staff would 
be put into an embarrassing position in such debrief ings if students had rated them badly, 
rather than a lack of their value. 

The reporting of upward appraisal results back to students was found to be very limited 
in practice. Generally it seems that although a substantial amount of upward appraisal 
existed in the Business School, on only one course were the results found to be reported 
back directly to the students. No examples of systematic briefing sessions with students 
using the data generated by upward appraisal were found. 



Over 70% of our sample had experienced upward appraisal by students within the last 
two years. A number of concerns emerge from our preliminary analysis. Firstly, it 
seems little thought is given to the design of the upward appraisal instruments used. A 
number of criticisms could be made about the psychometric and other properties of many 
of the instruments used for upward appraisal found in this study. 

Secondly, it would seem that the education sector could leam from industry's use of 
upward appraisal in a number of areas, not least in feedback to those who provide the 
data. Here industrial practice often encourages, in a few cases this is compulsory, those 
being appraised to conduct face to face debriefing sessions with the appraisers (Bernardin 
1986). Such debrief ings have the potential to be a very valuable developmental activity 
but need to be conducted with care. Here industrial practice emphasises the need for 
skilled facilitators in debriefing sessions. Upward appraisal, as practised in the higher 
education sector, seems to generate a rich source of data but organisations do very little 
with it. Our argument is that upward appraisal should be integrated into and inform 
decision making on the broader learning and teaching practices of the institution via 
incorporation into a student Personal Development Portfolio (PDP). Currently two of the 
authors are developing a model that will achieve this. 

Finally, although space has denied us the opportunity to examine the impact of upward 
appraisal on student commitment and morale there is some evidence that this is far from 
clear cut. Indeed, similar to the effects of other sophisticated interventions that are 
designed to increase commitment, student appraisal of lecturers may paradoxically 
actually decrease it (Bushardt et al 1991; Illes et al 1990). For example, one of our 
preliminary findings is that students who had been involved in a number of 
lecturer/course appraisal activities over the first two years of a three year part-time course 
were increasingly reluctant to participate in such activities in their last year and some 
refused completely. 

Two main reasons were given for this. Firstly, they felt strongly that it reduced to little 



CONCLUSIONS 





190 



QUALITY IN TEACHING AND TRAINING 



more than a paper exercise. No feedback to them was given on the results of previous 
reviews, nor did any positive action seem to ensue. One of their measures here was that 
the lecturers they had complained about and given very poor ratings to were still lecturing 
them. What is more, they continued with the same bad practices as before. No 
improvements could be discerned. Further, a number of lecturers had, in the words of 
one student, "got the hump" about the ratings they had been given and reacted 
"petulantly". 

A concern of some of the students was that following what were obviously very 
negative ratings (they could only assume this as no formal feedback was given to them), 
a number of lecturers were perceived as apparently retaliating by setting more work, more 
difficult assignments, and giving lower grades. Given that upward appraisal had resulted 
in no noticeable improvements and a perceived retaliation by lecturers it would be no 
surprise to find a reduced student commitment and morale. Again, the potential to 
prevent this negative impact of upward appraisal on students may derive from the use of 
a PDP mechanism that integrates them into broader decision making. 

Finally, one thing is evident from our preliminary analysis. Although there is a 
considerable amount of research on the upward appraisal of lecturers, much of this has 
been for research purposes only and concentrated on the more "technical aspects". Clearly 
there is a need for a broader based approach that examines the impact of upward 
appraisal on practice before whole-heartedly recommending its wider adoption as a 
mechanism to enhance quality in higher education. 

References 

Banks C G and Murphy K R (1985) Towards Narrowing the Research -Practice Gap in 
Performance Appraisal Personnel Psychology 35 pp 281-322 

Bernardin H J (1986) Subordinate Appraisal: a valuable source of information about 
managers Human Resource Management 25 pp 421-439 

Bernardin H J and Beatty R W (1987) Can Subordinate Appraisals Enhance Managerial 
Productivity? Sloan Management Review 28(4) pp 63-73 

Bernardin H J and Klatt L A (1985) Managerial Appraisal Systems: has practice caught 
up to the state of the art? Personnel Administrator 30 (ii) pp 79-86 

Breed F S (1927) Factors Contributing to Success in College Teaching Journal of 
Educational Research 16(4) pp 247-253 

Buhalo I H (1991) You Sign My Report Card, I'll Sign Yours Personnel May 23 

Bush G W and Stinson J W (1980) A Different Use of Performance Appraisal: 
evaluating the boss Management Review November 14-17 

Bushardt S C, Duhon D L and Fowler A R Jr (1991) Management Delegation Myths and 
the Paradox of Task Assignment Business Horizons March -April 37-43 

Crittenden K S and Norr J L (1973) Student Values and Teacher Evaluation: a problem 
in person perception Sociometry 36(2) pp 143-151 



UPWARD APPRAISAL 



191 



Feldman K A (1976) Grades and Student's Evaluation of Their Courses and Teachers 
Research in Higher Education 5(3) pp 243-288 

Feldman K A (1977) Consistency and Variability Among College Students in Rating 
Their Teachers and Courses: a review and analysis Research in Higher Education 6(3) 
pp 223-274 

Feldman K A (1978) Course Characteristics and College Students' Ratings of Their 
Teachers: what we know and what we don't Research in Higher Education 9(3) pp 
199-242 

Feldman K A (1983) Seniority and Experience of College Teachers As Related To 
Evaluations They Receive From Students Research in Higher Education 18(1) 
pp 3-124 

Feldman K A (1984) Class Size and College Students' Evaluations of Teachers and 
Courses: a closer look Research in Higher Education 21(1) pp 45-116 

Feldman K A (1986) The Perceived Instructional Effectiveness of College Teachers As 
Related To Their Personality and Attitudinal Characteristics: a «wiew and synthesis 
Research in Higher Education 24(2) pp 139-213 

Feldman K A (1989) Instructional Effectiveness of College Teachers as Judged by 
Teachers Themselves, Current and Former Students, Colleagues, Administrators and 
External (Neutral) Observers Research in Higher Education 30(2) pp 137-194 

Illes P, Mabe C and Robertson I (1990) HRM Practices and Employee Commitment: 
possibilities, pitfalls and paradoxes British Journal of Management Vol 1 pp 147- 157 

Parasuraman A and Zeithmal V (1985) A Conceptual Model of Service Quality and its 
Implications for Future Research Journal of Marketing Vol 49 pp 41-50 

Pctrini C (1991) Upside-Down Performance Appraisals Training and Development 
Journal July pp 15-22 

Saleh F and Ryan C (1991) Analysing Service Quality in the Hospitality Industry Using 
the SERVQUAL Model The Services Industries Journal 1 1(3) 

Stinson J and Stokes J (1980) How to Multi- appraise Management Today June pp 43-53 

Rutherford D (1988) Performance Appraisal: a survey of academic staff opinion Studies 
in Higher Education 13(1) pp 89-100 

Tsui A S and Ohlott P (1988) Multiple Assessment of Managerial Effectiveness: inter- 
rater agreement and consensus in effective models Personnel Psychology Vol 41 pp 
779-803 



200 



30 A total quality approach to managing 
CBT development 



Steven G Shaw and David R Shaw, Concordia University, Montreal and Curvet 
Information Systems, Ottawa, Canada 

INTRODUCTION 

A "total quality" approach to management of instructional development has many 
complex elements. Essentially, quality management involves (a) defining a quality-level 
bracket, (b) improving the quality of the product within that bracket, and (c) reducing the 
cost of production. 

Our full model of quality management for an organisation reflects four levels of 
information that are integrated horizontally and vertically between and across levels: 
strategic (used by senior management), tactical (used by middle management), supervisory 
(used by lower management) and functional (used by operational personnel). Originally 
developed for large publishing companies, it is now being adapted to CBT in an 
Instructional Design Quality Management Manual. 

Here we present techniques for quality manrgement that can be implemented at a 
tactical level or lower. That is to say, we provide a model and methodology that could 
be implemented at the level of a training division or department, even if a commitment 
to total quality management has not been embraced at a higher, strategic level by the 
organisation as a whole. 

The paper discusses techniques and approaches for reducing cost (including scheduling 
approaches, estimating, job and version control, developing benchmarks, reducing cycle 
time), improving quality (robust design, pre-design, concurrent engineering, design for 
production, rapid development) and assessing quality (quality indicators). Specific 
illustrations of the application of principles and methods of quality management to 
development of computer-based training are provided. 

IMPLEMENTATION 

There are as many models of total quality management as gurus. However, you can 
design your own program. In fact you will have to do this, anyway. Each organisation 
requires a unique solution. T e methods and procedures presented below are thus a 
shopping list, to help you get a quick start at the departmental level. They may not all 
be suitable to your circumstances. 

Another important consideration is that implementation may take years (typically two 
to five). Progress may wax and wane during this period. Initial results may even be 
confusing (Sutton 1991). To see a program through these difficult times, senior 
management must have total commitment. Some of the common reasons for failure are 
(Bellis-Jones and Hand 1989): inadequate resource levels; poorly designed procedures; 
inadequate documentation; lack of training; inadequate supervision; poor documentation; 
poorly motivated staff; changing priorities; and poor supporting computer systems. 

Other researchers have reported these barriers (Keys 1991): lack of use of Theory Z in 
human relations; lack of to'.al employee commitment; attempting to produce at capacity; 
excessive emphasis on short-term profits; and outdated cost-accounting systems. 

Most of the available models of implementation focus on manufacturing (Albrcgtsc et 
al 1991). Applying TQM to services is more difficult, nonetheless it is applied 
successfully in projects and service industries, including educational instruction. 



ERLC 



201 



MANAGING CBT DEVELOPMENT 



193 



Here we look at some methods for local implementation at the departmental level, based 
on SWOT analysis, an improvement and an implementation .strategy (Edosomwan and 
Savage-Moore 1991). 

SWOT Analysis 

Develop policy and procedures to prepare an annual departmental marketing audit, using 
an analysis of Strengths and Weaknesses, Opportunities and Threats, and Issues. Strengths 
and weaknesses are internal factors that may or may not be within your control. 
Opportunities are external factors frequently outside your control. Threats and issues are 
posed as questions to stimulate discussion. 

Improvement Strategy 

Develop an improvement strategy for the department based on: 

• activity analysis 

• customer identification 

• process improvement 

Do a global optimisation of the department by defining core, support and discretionary 
activities. Core activities, the reason for the existence of the department, use group skills 
to add value to the business. Support activities clearly support core ones. Discretionary 
activities are all of the running around necessary to get things done. Use time cards to 
determine where effort is spent. Analyse the results to determine how to reduce time on 
discretionary activities. Typically efficiency improvements of 10-20% are possible (Bellis- 
Jones and Hand 1989). 

Identify the external customers of the department. In the case of a CBT department, 
these are likely to be other departments. Find out exactly what these customers want or 
expect. Then work backwards within the department, identifying internal customers. Each 
worker is a link in a chain of customers that ultimately leads to the external one. Each 
link is at once a customer to one other link and a service provider to another. Focus on 
the interface between links. Find out what internal customers want by asking them 
questions like, "What do you need from me. How do you use my output, How can I 
improve my service to you. Am I providing things you don't use" (Lee 1991). 

Finally, develop a plan to implement a program of continuous process improvement 
(CPI), perhaps using some of the methods discussed in this paper. 

Training Needs 

Assess training needs. Train supervisors and workers in how to measure quality and 
implement quality programs. Focus on creating an environment where change and 
innovation come naturally. Encourage adoption of the most current proven tools (Vesey 
1991). 

Implementation 

To implement the CPI plan, set up a departmental steering committee. The purpose of the 
committee is to identify quality projects or problems. It uses various analytical techniques 



194 



QUALITY IN TEACHING AND TRAINING 



to decide which project or problems it should undertake. The committee establishes 
quality teams to analyse and eliminate the problem (Nandyal and Welch 1991). The 
findings of teams are binding on the committee; otherwise the exercise is pointless. 
Teams fc~us their discussions in the context of tools like quality function deployment 
(QFD) and fishbone diagrams, thus avoiding encounters that turn into gripe sessions or 
are otherwise counterproductive. 

COST REDUCTION 

The procedures \^Iow are presented in a rough order of implementation. Having workers 
do run diagrams, for example, while interesting will yield few benefits if there is no 
overall quality framework in the department. On the other hand, error review can be 
implemented readily without a framework. Similarly, maintenance, safety and security, 
job and version control are basic organisational needs, presented here to reinforce the idea 
that every aspect of an organisation has cost and quality implications. 

Some companies, for example, even stipulate the maximum number of allowable rings 
before someone answers the telephone. Again, it must be stressed that these are ideas for 
consideration and may or may not suit your organisation. 

Maintenance 

Develop procedures for engineered maintenance cycles for facilities and equipment. Do 
equipment maintenance in off-hours. Keep equipment in perfect condition. 

Safety and Security 

Develop procedures for safety and security. Include virus protection and backup and 
archiving procedures for computer systems. 

Job Control 

Develop procedures for assigning docket numbers as the primary job control. Use a 
mnemonic plus numerics. Use a checksum with numerics. Design information systems 
to check mnemonic against customer lookup table, and to check numerics using 
checksum. 

Version Control 

Develop policy and procedures for version control. In most cases a manual or sequential 
change model is sufficient. Where multilingual products are developed, a selectable 
change model may be required. The sequential model manages files, filenames, 
file-version names and the sequence of deltas that make up a file version (Cronk 1992). 

Suppliers 

Develop procedures to purchase products like diskettes from companies that certify 
quality at the source. If not, develop procedure to inspect goods immediately. Report 



2<C 



MANAGING CBT DEVELOPMENT 



195 



defects as soon as possible. Develop procedures to use economical order quantities based 
on available discounts and inventory costs. 

Undercapacity Scheduling 

Use undercapacity instead of capacity scheduling. As its name suggests, it schedules 
somewhat less than a maximum daily workload. For example, if there are 7.5 hours in 
the workday, you might schedule just enough work for 6.75 hours. 

The basis of undercapacity scheduling is to give workers enough time to do the job the 
right way the first time. This avoids costly re-runs due to errors. Also aim to do the same 
amount of work each day. 

Undercapacity scheduling also allocates time for training and problem solving, and is 
thus key to quality management. 

Adopting undercapacity scheduling may be difficult psychologically for supervisors and 
managers who feel they must run a tight ship. They keep the pressure on, and have 
difficulty understanding that undercapacity scheduling actually increases annual 
throughput. A production group producing a consistent, predictable amount each day will 
outperform one using capacity scheduling. 

To achieve undercapacity scheduling, use a Master Production Schedule having three 
divisions: master schedule, actual demand, and available-to-promise. Set up the master 
schedule based on forecasts. As orders are booked, enter hours for each job under actual 
demand and compute the difference between forecast and demand. The difference is 
available-to-promise to user departments. Knowing the available-to-promise obviously 
helps the scheduler. It avoids false promises and the inevitable user dissatisfaction they 
entail. 

Lot Size 

With undercapacity scheduling in place, gradually reduce the lot size. Instead of giving 
a worker 100 frames to complete by a given deadline, give him/her 20 frames and a 
shorter time frame. This increases the required supervision but benefits include less 
pressure on workers, hence higher quality, faster feedback, and increased flexibility. If 
one station frees up earlier than expected, work can be redirected. 

Cross-Training 

To enhance flexibility and productivity, cross-train workers in each other's job. For 
example, with training, graphic artists instead of the instructional designer can specify 
graphic requirements. Cross- training of this type may be increasingly necessary with 
multi-media, where Renaissance-type instructional designers may be rare (see Training 
and the Development of ISD Expertise, below). 

Set-Up Time 

Investigate and improve setup time. This is the time necessary to setup for a job or 
particular kind of work. The ideal is one-touch setup - one button to push. Reducing 
setup time cuts costs through increased productivity and increases flexibility. 



204 



196 QUALITY IN TEACHING AND TRAINING 

Just in Time 



As lot sizes are reduced gradually, move to "just in time" (JIT). JIT exposes problems, 
and enforces problem solving. Normally managers build extra inventory and put slack 
time into schedules to compensate for known problems. Slack time is a common practice 
in project management for CBT. Eliminating these buffers reduces overhead and 
associated costs. It exposes problems, forcing you to find remedies. This improves 
quality, reduces waste and re- runs and lowers work-in-progress inventory carrying costs 
(Alonso and Frasier 1991). 

For example, when schedules are not being met, the tendency is to release jobs early, 
in the hope that somehow deadlines will be met. In practice, the customer will use the 
extra time to issue change orders, increasing the uncertainty in the schedule, causing the 
system to become chaotic. 

JIT principles can also be applied to project management, reducing project life cycles. 
Any management practice that shortens the planning horizon improves project profits 
(Alonso and Frasier 1991). Planning frequently (i.e., monthly) increases the accuracy of 
forecasting and scheduling. In one case, "simultaneously improving all of the delays and 
goals under management control [benefited] the client's net cash by over 13% of sales" 
(Alonso and Frasier 1991). 

Scheduling 

There are various approaches to scheduling, but the bottom-line objective of JIT is a zero 
deviation from schedule. 

Use backward scheduling for periodical work or work by appointment. It starts with the 
distribution date then works backwards to arrive at the date when files must be delivered 
to the production department. 

Use forward scheduling for services on demand, i.e., rush jobs. Calculate forward 
schedules from delivery of raw data or the time at which personnel will be available to 
work on the job, to arrive at the delivery date/time. Rush jobs may not affect mainstream 
CBT activities but may be a factor in related service departments. 

Synchronised scheduling is a variation of backward scheduling. The final assembly date 
establishes the schedule for subprocesses, using standard offsets for lead time. Base the 
off -sets on production benchmarks. When calculating lead-time, consider these elements 
in descending order of significance (queue time often accounts for 90%) (Schonberger 
1985): 

• queue time 

• run time 

• setup time 

• wait (transportation) time 

• inspection time 

• move time 

• other 

Whichever technique you use, the resulting schedule must be regular. If it is, you can 
then aim to reduce the required time project by project. The scheduler must keep a close 
eye on work in progress (WIP), to avoid unnecessary levels or capacity scheduling. This 
improves service by reducing queue time, responding more quickly to users, reducing 
uncertainty in schedules, and giving users more reliable status information. As lead times 



20Z 



MANAGING CBT DEVELOPMENT 



197 



shorten, forecast accuracy improves. There is less need for shop-floor ccmrol by 
schedulers and supervisors. 

Obviously there are practical limits to reductions in WIP. There should be enough jobs 
to keep work centres busy. Loads at work stations will vary. There is pressure to overload 
some to avoid underloading others. However, increasing the workload will only increase 
queue times, making schedules less realistic, with al! the chaos and inefficiency that 
implies. Another difficulty is that it is often hard to predict queue times for any particular 
job. It may queue at several workstations before completing its routing. 

Like forecasting, scheduling is necessarily inexact. For example, vacation periods are 
usually unknown. Personnel book off sick. Customers miss deadlines. Nonetheless, 
scheduling is necessary and the key to strategic planning, eg: 

• changing procedures 

• modifying production methods 

• introducing new technologies 

• assessing resource requirements 

• forecasting budgets 

Good practices make it easier to meet schedules. First identify the key success variable 
for your operation (eg. on-time delivery) and design a system that produces the required 
results. Re- assess this key variable from time to time. 

Calculate the daily capacity of your production centre in hours rather than production 
units such as frames. Determine the effective capacity using a formula such as: 

Effective daily capacity = ((H-L)/H)(HxSxE) where: 

H = normal hours in a day excluding lunch, breaks, etc. 
S = number of shifts in a day 
E = number of employees 

L = loading factor for undercapacity scheduling (eg. 0.5 hours/shift) 
<(H-L)/H) = undercapacity factor 

When drawing up schedules, be aware of pre-established commitments for regular jobs. 
Make allowances for rush jobs, using estimates based on historical data. Schedule rush 
jobs separately from regular ones. Consider doing rush jobs, altogether, every day at a 
certain time. Use simpler control methods for rush jobs and charge them out at cost-plus. 
Use a quicker invoicing procedure for rush jobs. 

Also estimate expected sickness and absentee levels from historical data and maintain 
a list of part-time employees. Encourage employees to post their vacation periods as early 
in the year as possible. Schedule regular preventive maintenance, preferably in off-hours. 
Schedule to avoid known bottlenecks, levelling peaks. 

Understand and exploit the paradoxes. When schedules are irregular, there is a tendency 
to rush jobs into production as soon as possible, in the hope that they might be completed 
on time. Conversely, make it a rule to release regular jobs into production as late as 
possible, ie. just in time. This reduces the chance of late changes. Before releasing a job, 
be sure all information is complete and all components are available. If the job is 
incomplete, set deadlines for the supply of missing information. Charge for last-minute 
changes to jobs based on accurate records of time and materials. 

Update your schedule using information from time sheets and observation. Use flexible 
production, changing from single to partial or full second shifts as necessary. Determine 
economical breakpoints between the various options. For example, use temporary labour 



20G 



198 



QUALITY IN TEACHING AND TRAINING 



for the first 5% of extra work, use overtime for the next 5%, then reduce customer 
service by rescheduling if you can't justify a full second shift. 

Estimating 

Develop a policy to base customer charges on estimates instead of actuals (excluding 
change orders). This enforces problem solving if jobs cannot be completed within 
estimates. Develop a procedure to base estimates on benchmarks and the depth of design. 
Develop procedures to place estimates on dockets (Ziegfeld et al 1985). This contracts 
workers to the production budget and also encourages faster feedback, allowing them to 
renegotiate the contract if they believe the estimates are unrealistic. 

Benchmarks 

Develop production standards or rates for all of the various parts of a job. Use work 
measurement techniques such as statistical sampling, stop-watch studies, and tables of 
industry norms. Develop procedures for devising and monitoring internal benchmarks for 
activities like writing, graphics* programming. Use meaningful time frames for 
benchmarks. For example, it may be unrealistic to expect a writer to do 1500 words/day 
but realistic to expect 7500 words/week. Don't base benchmarks on completed 
components such as frames/day, pages/day, etc. Review benchmarks monthly. 

Depth of Design 

An important consideration is the depth of design (good-enough factor) the customer 
requires, eg. a Chevrolet or a BMW. Many CBT projects are budgeted simply by working 
backward from the customer's budget. If he/she wants three hours of courseware for 
$30,000, and the instructional company charges $60/h, then the depth of design is 200:1 
hours. This is fine if the customer truly understands what 200:1 means, or if 200:1 is 
good enough. Use QFD to determine customer needs and depth of design (see IMPROVE 
QUALITY); use simpler measures such as reverse calculation only for reality checks. 

Cycle Time 

Reducing cycle time (also called time compression management, time-based 
competitiveness and time-based innovation) for products or projects, a quality method in 
place in Japan since 1981, delivers major financial benefits, while allowing rapid response 
to changing market conditions (Musselwhite 1990). It is closely allied with concurrent 
engineering and JIT. 

Shorter cycle time, a response to quickly changing markets, gives quicker market entry, 
allows organisations to charge a premium price for new products, reduces costs and 
improves quality (Musselwhite 1990). It creates opportunities to increase market share, 
market leadership and profits (Vesey 1991). An additional benefit is that turning over 
projects more rapidly increases contact with the market and increases the rate of learning 
within the organisation. 

Cycle-time reduction cannot be implemented incrementally. Incremental change is 
efficient in building tools, like supporting computer systems, but will not significantly 
affect cycle time. 




MANAGING CBT DEVELOPMENT 



199 



Major gains are only possible through process elimination and innovation, especially at 
boundaries between departments. At this level there are two primary objectives: speeding 
new products to market and quickening response to customers in production and delivery 
(Sheridan 1991). Set targets like reducing cycle time by 50%. One very interesting 
approach is to concentrate on the time to break-even, rather than the entire product or 
project cycle (IM Management Roundtabie 1990), Break-even time encompasses all cash 
flow from a project and incorporates the entire development process from concept to sales 
(Inglesby 1991). 

Start by speeding up existing processes as much as possible (eg. use E-mail). This gives 
faster feedback, too, so problems are dealt with promptly and customer satisfaction is 
increased. Next, re-design the existing process, through organisation and procedures. 
Supporting business and computer systems must evolve simultaneously. Scalable solutions 
based on open systems are a prerequisite. They must be built with fourth-generation 
languages not aligned with any platform, operating system or database. The underlying 
data model must permit long-term growth of the department's database (Inglesby 1991). 

IMPROVE QUALITY 
House Style Guide 

Develop policy and procedures for creating and maintaining a house style guide. Develop 
procedures for customising the guide to meet individual customer/project needs. 
Incorporate the guide into style sheets and templates linked to authoring tools. 

Glossaries 

Develop procedures to build and maintain glossaries of nomenclature peculiar to the fields 
your organisation services. 

Control Points 

Make every workstation a quality control point, reducing the bad product passed forward. 
Develop check lists for each station. 

Immediate Evaluation 

Evaluate every module right after it is completed. Develop automated shells that 
completely compile CBT programs overnight, so c valuators have the latest version each 
day. 

Make everyone responsible for quality: give each worker responsibility to flag defects 
even if not in their area of responsibility. 

Defect Review 

Develop procedures to trace defects to their source immediately upon detection. The 
purpose is to devise a system improvement - not to blame the individual. Where defects 
are not found out until a later process, or until the customer uses the product, provide fast 
feedback right to the originators. 



20£ 



200 



QUALITY IN TEACHING AND TRAINING 



Defect Correction 

Make every workgroup responsible for correcting its own defects. Sometimes imagination 
is required. At a car factory in Canada, the purchasing department bought the wrong size 
of tires. They were mounted on 50 cars before the defect was caught. To correct the 
defect, the entire purchasing department was mustered in the parking lot to change the 
tires, using the standard tools found in the trunk. 

Internal Customers 

Develop a policy making each individual/group a customer/service provider to other 
individuals/groups. 

Internal customer service items: develop a procedure for defining internal customer 
requirements and a related benchmark and/or checklists. 

Pre-Design Planning 

Invest in pre-design screening, evaluating and planning. The larger the design project, the 
more important the planning phase (Ziegfeld et al 1985). As a rough guide, aim for 
distribution of effort like planning (40%), CBT design (25%), process design (15%), 
production, test, delivery (20%). Obviously distributions like this cannot be achieved with 
traditional ID and CBT methodologies. 

Quality Function Deployment 

QFD is a set of planning and communication routines that facilitate concurrent 
engineering and give focus to meetings of quality teams. It assumes that marketing, 
design and production people will work closely together from the first conception of a 
product. It allows organisations to learn from customer experience and to reconcile what 
customers want with what instructional designers can reasonably build (Hauser and 
Clausing 1988). The procedures in QFD are simple: 

1. Develop customer attributes: find out what customers want, expressed in their own 
words. These items, typically 30-100 of them, are customer attributes (CAh They may 
also include demands of regulators (eg. ISO 9000, MIL specs). Some attributes may 
be gathered into groups called bundles. 

2. Weight customer attributes: weight each attribute in percent, and place the data in a 
table. 

3. Analyse competitive advantage: on the right-hand side of the table, place a perceptual 
map giving ihe customer's perception of your capability of meeting an attribute (a), 
compared to the competition (b). Rate attributes on a scale of 1-5 (5 = best). This 
may relate closely to the type of authoring tools used or the expertise of your 
instructional-design team. One impediment here is that many design organisations are 
largely tool driven, since few existing tools are modular or extensible in any 
significant way. Competitive advantage data is usually collected using marketing 
methods. Identify areas of advantage (plan to maintain them) and opportunities for 
improvement. 



2VD 



MANAGING CBT DEVELOPMENT 



201 



4. Develop design characteristics: along the top of the table, develop design 
characteristics (DC) affecting each customer attribute. A DC may affect more than 
one CA. Express design characteristics in quantifiable terms. Analyse carefully, 
brainstorm, avoid vagueness. 

5. Assess design characteristics vis a vis attributes: determine what is the strength and 
direction of the relation between each DC and its associated CAs. Fill in the body of 
the table with this information, linking the DCs and CAS. Check for consistency 
between customer evaluations of CAs and measures of the related DCs. 

6. Ascertain what are the interdependences among DCs: set up a matrix depicting these 
relationships. 

7. Assess design characteristics vis a vis competitive benchmarks: furnish objective 
measures for current DCs. 

8. Set design targets: determine what are ideal values for the DCs. 

9. Assess design targets vis a vis desired cost and quality levels: determine whether 
design targets are within the quality level required by the customer and the 
appropriate cost bracket. Refer to the matrix depicting interrelations among DCs for 
assistance in making decisions regarding trade-offs to meet these constraints. 

10. Design development and production processes: establish development methodology 
and plan production based on target design. 

Design for Production 

Design CBT applications so a) they readily fit your existing authoring shells, and b) 
production can be automated as much as possible. If customers want features not readily 
accommodated by the shell, don't shoehorn the application. Extend the shell, or develop 
a new one. Use tools like HyTime, the SGML-based hypertext and multimedia structuring 
language. 

Robust Design 

Robust design (Genichi Taguchi methods) builds in tolerances for variables that are 
known to be unavoidable. Identify the controllable and uncontrollable factors in your 
production process. Optimise the controllable factors within a range that minimises any 
effect of the uncontrollable ones. Any deviation in quality, no matter how small, increases 
the product's ultimate cost, including warranty liability and lost customer goodwill. 

Concurrent Engineering 

Concurrent engineering is one of the prime tools for cycle- time compression. It uses 
cross-functional or multi -disciplinary teams to provide more effective product designs, 
design products and production processes simultaneously, reduce time to market and to 
link producible designs to highly productive processes (St. Charles 1990). 
Cross-functional teams work concurrently on all aspects of the project. 

Traditionally CBT uses a linear process: research, product design, process design, 
production, sales and distribution. One department does its thing, then tosses the job 
"over the wall" to the next (Shina 1991). 

This linear process has many review stages and is costly. Product is recycled often to 
correct errors or incorporate changes. Manufacturing studies have shown that 70-80% and 
75-80% of production costs are locked in during the design stage (Creese and Moore 
1990; Musselwhite 1990: Sutton 1991). 



210 



202 



QUALITY IN TEACHING AND TRAINING 



Concurrent engineering is similar to rapid prototyping except ihat it emphasises 
management as well as design skills, focussing on customer needs (Creese and Moore 
1990). One of its benefits is that typically most design changes occur before production 
starts (Turino 1991). A useful aspect of rapid prototyping is to use automated shells to 
load all completed modules every night, so e valuators can work each day with the current 
version. 

Concurrent engineering involves all departments simultaneously in : parallel process. 
Practice has shown there should be a minimum of five people on a team, to prevent one 
person from dominating. Such cross- functional teams have reduced new-product 
introduction time by 50-75% (Musselwhite 1990). According to Hauser, "...the use of 
interfunctional teams benefits design.... [but] what should these people talk about? How 
could they get their meeting off the ground? This is where the house of quality (QFD) 
comes in," (Hauser and Clausing 1988) or the committee approach advocated by Nandyal 
(Nandyal and Welch 1991) [see IMPLEMENTATION] . 

Run Diagrams 

Develop procedures for individuals to monitor their own performance using run diagrams. 
Identify and investigate system problems. 

Technical Planning 

A major strategic issue facing CBT organisations is responding to technological change. 
Research has shown that organisations having a high-level technology plan are more 
successful than others (ITAC 1990). A technology plan has many aspects. 

Products can be brought to market very quickly by having three design teams 
simultaneously develop solutions, but deliver them to market on different time lines. In 
some large-scale CBT projects, this may be a means of version planning. An original 
product is delivered quickly to the customer; significant change orders are incorporated 
into V.2.n; meanwhile more sophisticated versions are in development. 

The same model may be used in planning for CBT technology, in the sense of pacing 
the technology. An organisation producing text and graphic frames could plan to migrate 
to multi -media. At the moment, muhi -media is expensive and suited only to certain 
training situations. However, as multi-media functions are incorporated into operating 
systems, the cost will decrease while user expectations increase. 

One problem is that we get locked into the paradigm of existing operating systems. 
DOS users, and even Windows users, don't understand the seamless environment of 
Macintosh, and thus cannot envision the type of products it enables. Similarly, those of 
us who have not fooled around with desktop video cannot truly envision the applications 
it enables. The technology plan should therefore include a small budget for experimental 
technology. 

PROCESS IMPROVEMENT 

The previous sections of this paper have addressed techniques for reducing cost, 
improving quality and assessing quality. Many of the techniques and mechanisms (e.g., 
JIT) presented are designed to expose problems in the processes underlying design, 
development and production functions, as much as to reduce cycle time. The following 
sections focus in more detail on issues relating to the conception and execution of these 



211 



MANAGING CBT DEVELOPMENT 



203 



processes, with a view to making suggestions about how the cycle time for creation of 
CBT products might be compressed. 

Time compression is a major topic in the current literature on TQM and industrial 
engineering (cf Inglesby 1991; Musselwhite 1990; Sheridan 1991; Vesey 1991). The 
advantages of quicker product delivery in a manufacturing context are obvious: there are 
potential cost reductions and the possibility of greater market share through touching the 
market first, or more often in a given time span. Shorter cycle time also allows for a 
more incremental approach to product development and innovation and this brings lower 
risks. The analogy between manufacturing and CBT development is not perfect, since in 
most cases CBT products are custom tailored solutions rather than mass market products. 
However, the benefits of time compression are still clear: instructional development is one 
area in which, paradoxically, advances in technology have had the effect of significantly 
increasing the time and costs associated with design, development and production phases. 

There are two kinds of decisions which can be taken be taken with respect to time 
compression: total and incremental (Vesey 1991). Incremental decisions modify existing 
procedures to produce minor or modest gains in efficiency and time compression. Total 
decisions concern the radical redesign (or elimination) of processes and yield major 
improvements. Vesey reports a number of striking cases to illustrate the kind of impact 
total decisions can have. For example, the Ballistic Systems Division of Boeing 
Aerospace Corporation reduced design analysis from two weeks to 38 minutes. 

Both kinds of improvements - radical and incremental - are discussed below. Arguably, 
CBT development has a great deal to gain from incremental decisions. Even in 
sophisticated manufacturing which involves intensive design engineering efforts, product 
design typically accounts for 20% or 25% (Sutton 1991) of the overall cycle. In CBT, 
design typically represents up to 75% (Merrill 1988) of the overall cycle. Putting aside 
claims concerning the complexity of instructional design and the ill-defined nature of 
instructional design problems, these comparative figures suggest that existing procedures 
have not been optimized. 

Incremental Improvements 

Areas where improvements can be made include: the management of information 
generated in design and development activities, the design and implementation of 
error-proof processes with automatic checking devices in all phases of ISD and 
production, the introduction of new approaches in the knowledge acquisition phase of 
ISD, the elimination of prototyping, the automation of portions of the ID and production 
processes, and explicit strategies to develop expertise among instructional designers. 

Information Management in ISD 

There are many kinds of information generated during CBT development relating to 
initial conceptualization, feasibility, planning, development, production, delivery and 
evaluation. These include: the results of task and content analyses, objectives and 
sequences, media selection and instructional strategies analyses; test item banks; syllabi 
or outlines; scripts, course and lesson design specifications; graphics and text components, 
libraries of computer code. This information is highly interrelated. In fact many categories 
are the direct result of a transformation process that takes another category as input: tasks 
are converted to objectives, objectives are translated into test items, objectives are 
sequenced into lessons, and so forth. 




204 



QUALITY IN TEACHING AND TRAINING 



In small scale CBT development projects, with experienced teams, it is often possible 
to adopt a relatively informal approach to documenting the results of design and 
development activities and managing this information. In large scale projects, a systematic 
approach t information management founded on the construction and maintenance of 
linked databases associated with each stage in the ID process can provide efficiencies and 
reduce the risk of major, costly blunders. 

1. In the first place, information retrieval capabilities associated with databases can be 
used to support (and in some cases, to automate) analysis and design. For example, 
a database which can provide information concerning the relative proportion of 
different kinds of objectives can facilitate the design of a generic lesson structure. 
Another example: information concerning redundancy of tasks across different jobs 
could result in a more rational approach to training which consolidated these 
redundant tasks in one package. From a "performance engineering" standpoint the 
same information might be used to redesign jobs and reduce the need for training. 

2. Reports generated from databases can also assist in estimating production and 
development costs more accurately. For instance, a complete breakdown of objectives 
according to the media and instructional strategies selected for them will facilitate a 
more accurate projection of production time and costs. If it should prove necessary 
to compromise media or strategies selections, owing to constraints, then the same 
information will enable instructional designers to make the necessary adjustments 
more rapidly, consistently and rationally. 

3. Linked relational databases ensure that design and development decisions are 
consistent across the ISD process. The database system can be designed so that a 
change in an element associated with one phase in the ISD process can result in flags 
being set against related elements that are products of other phases. A change in an 
objective, for example, would result in flags being set against related artifacts such 
as test items, graphics selection, strategy selection, sequencing information and so 
forth. Automating the regulation of consistency in the ISD process obviously has 
even greater benefits when the content is not entirely fixed or frozen at the time 
design activities are undertaken. This is often the case in large scale projects in 
technical fields, where development of systems operation and maintenance training 
may have to begin before all systems are fully engineered and in production. The 
implication for training development based on initial design engineering specifications 
is clear: there may be many revisions. Under this condition it is essential to have tools 
which can register these changes and maintain the integrity of the ISD process by 
identifying inconsistencies as they are introduced and by facilitating the process of 
rippling required changes through the various subsystems of the evolving instructional 
design. This use of instructional design databases to ensure consistency is also an 
example o* tlie utilization of an error-proof process with automatic checking built in. 

4. Finally, ID databases with automatic report generation that will track the degree of 
task completion for various phases can obviously be used for project management 
purposes ?tid to assist in producing intermediate deliverables in the form of 
documentation of the various ID phases. 

Barriers to Implementing Information Management Systems in 
ISD 

Despite the obvious applicability of database approaches to managing development-related 
information in ISD, only a small proportion of commercial developers exploit the 



213 



MANAGING CBT DEVELOPMENT 



205 



possibilities. An informal survey of five Canadian CBT development operations indicated 
that not one employed database management systems (DBMSs) in the ISD process in a 
significant way (ie. in a way that links ISD phases and which exploits the potential of 
DBMSs for design decision support). Even in large scale projects, the operations surveyed 
routinely used DBMSs only for one phase (task analysis), and in some instances for 
project management purposes. 

One reason for the limited use of DBMSs in ISD is the lack of commercially available, 
iSD-specific, applications. Commercially available authoring systems often include as an 
option a DBMS that can be used to implement computer management of course delivery 
("computer-managed instruction") but they do not offer DBMS tools for ISD. 

A barrier to creating custom DBMS systems in-house is the issue of the different 
requirements which would have to be addressed. Different phases of the ISD process 
require different kinds of interactions with information and correspondingly different 
forms of representation and different interfaces (Gibbons and O'Neal 1989). 

In the analysis phase, for example, it is crucial that the developer be able to manipulate 
structural representations (course structures, lesson structures, unit structures) as well as 
the data associated with individual elements in a structure. In media selection, individual 
objectives are associated with ranked choices from among available media. With the 
kinds of analyses associated with this phase, wherein scenarios associated with alternative 
mixes of media are compared on the bases of instructional value and overall costs, a 
spreadsheet representation is more appropriate than a structural representation. 

A fully integrated tool for ISD would thus have to combine flexible outlining functions 
with relational database capabilities, and ideally would also handle graphics elements as 
data. In addition, there would be requirements for handling multiple versions and possibly 
collaborative work (multiple developers). ' A tool with these features and suitable 
interfaces is feasible (at the boundaries of the current envelope of technology) , but would 
require a significant research and development effort. 

Knowledge Acquisition 

Standard ID practice is based on a separation of subject matter expertise and pedagogical 
or instructional design expertise. Instructional designers work with subject matter experts 
(SMEs) to extract content, but apply their own expertise based on experience, 
instructional theory and instructional design models to make decisions regarding how to 
best structure and present that content. Knowledge acquisition is very often a major 
bottleneck in ID where complex knowledge or skills are concerned. 

Traditionally, the central tools of knowledge acquisition (and knowledge representation) 
in ID have been procedural and hierarchical task analyses. One problem associated with 
these tools is that where cognitive tasks are concerned experts' knowledge is often highly 
proceduralized and telescoped, meaning that several explicit procedures have been 
collapsed into a single step. As part of the knowledge acquisition job an instructional 
designer working with a highly skilled SME must facilitate the explicit recall of steps or 
procedures that have become tacit or highly automated. 

In a typical scenario, the instructional designer will observe the SME performing a 
series of tasks, or listen to an account of the performance of those tasks, and will prompt 
the SME for further details or explanations where these appear to be missing. This 
activity is time consuming and can be highly problematic, depending on such factors as 
the quality of communication between the SME and the instructional designer, the level 
of knowledge of the SME, and the fluency the instructional designer is able to acquire 
in the language of the SME's domain. 



214 



206 



QUALITY IN TEACHING AND TRAINING 



More meticulous and detailed approaches to knowledge acquisition and knowledge 
representation have emerged in the field of knowledge engineering (KE) which is 
associated with the development of expert systems, decision support systems and 
intelligent tutoring systems. Both KE and traditional ID use interviews and protocol 
analysis as their basic methodologies. The basic difference between the two concerns the 
emphasis in KE on how the expert structures his knowledge. Task analysis focuses on the 
tasks performed by an expert; KE also emphasizes the expert's knowledge base (which 
provides an explanation for why things are done in a particular way). 

KE is even more labour intensive than conventional ID knowledge acquisition and 
representation and is warranted only when the instructional goals necessitate the 
additional effort (ie. when quality is defined at that level). However, having said that, it 
is worth considering whether any innovations in methodology that have arisen in KE 
could be transplanted to conventional ID practice to improve efficiency. KE is so time 
consuming that the impetus driving process improvement has been higher than in ISD. 
Since the basic methods are the same, many advances in KE would potentially be 
transportable to ID practice. 

KE has taken two directions that are relevant. Both involve the attempt to transfer more 
of the effort in knowledge acquisition on to the expert. This makes considerable sense on 
an intuitive level, since the element contributing most to the bottleneck in knowledge 
acquisition is the instructional designer's ignorance of the domain. 

The first approach attempts to eliminate the instructional designer's role, replacing her 
with a set of computer-based, possibly interactive, knowledge acquisition tools that the 
SME works with alone to build at least a first-cut explicit representation of her 
knowledge. In the area of knowledge-based systems development there has been some 
success in the strategy of providing shells to SMEs for producing rule-based systems. 
SMEs are then able to develop and test the rule base themselves, without the requirement 
of learning a complex language and programming environment. 

Another example of this approach deals with declarative rather than procedural 
knowledge. Shaw and Gaines (1987) have developed a methodology which they call 
Personal Construct Technology (PCD. In the PCT approach, an SME begins by 
identifying all the major elements in the domain. This domain is then decomposed into 
all possible triads. The SME then is required to identify, for all triads, how two elements 
are similar and how they differ from the third. At the end of this process an exhaustive 
taxonomy of the domain has been produced and the categories or features that underlie 
the taxonomy have been made explicit. Shaw and Gaines have developed computer-based 
incarnations of PCT that will guide the SME through this process in an interactive mode. 

PCT is very much a bottom-up approach, and real efficiencies in KE rather will likely 
come from top-down approaches which present a framework that can guide and focus KE 
activities in a given type of domain. Some work, such as Breuker's concerning generic 
models of diagnostic tasks, is being carried out in this direction (Breuker and Wielcnga 
1987). While results to date have not indicated radical breakthroughs, this would seem 
to be an area worth watching. 

An alternative to approaches such as PCT which seek to replace instructional designers 
or knowledge engineers is one which utilizes SMEs to a greater extent in a number of 
variations on the standard appnjach to SME interviews and collection of protocols. In one 
such variant the knowledge engineer observes two SMEs who, in turn, pose representative 
problems and provide solutions for those set by their colleague, while the protocols are 
recorded. The technique is based on the premise that the task of defining a representative 
and comprehensive set of problems or scenarios usually can be carried out much more 
efficiently by two experts than by a SME working alone with an instructional designer. 



2 IZ 



MANAGING CBT DEVELOPMENT 



207 



Training and the Development of ISD Expertise 

The knowledge acquisition bottleneck is one area where appropriate training can often 
play a role in increasing efficiency (Wielenga and Breuker 1986). Even those who have 
had formal schooling in ID have typically never received any specific training regarding 
knowledge acquisition from SMEs. There is reason to believe that training in a structured 
methodology for knowledge acquisition would improve the process. 

Lack of general ISD training and related design experience is also frequently a problem 
in instructional development operations. Many instructional designers lack formal training 
in their field; those with formal training who prove to be very competent often move 
quickly from their roles as instructional developers to assume supervisory or management 
positions. In the area of CBT in particular, there are apparently only a relatively small 
number of designers and developers with significant work experience. In cutting edge 
are?^ such as multimedia systems and intelligent computer- assisted instruction the 
situation is even more serious: there is little cumulative knowledge and experience to 
draw on in : [tempting to formulate design theory and development methodologies. Such 
advanced areas also seem to entail redefinitions of the traditional roles of instructional 
designers. 

Several responses to the problem posed by insufficient numbers of expert instructional 
designers have beer, advanced in the field of CBT development. On the one hand, there 
have been attempts to engineer systems with built-in expertise that might eliminate or at 
least reduce the need for expert designers. This is the thrust of the SOCRATES project 
(Doucet and Ranker 1990) and also to some extent of Merrills *s IDexpert project (Merrill 
and Li 1989). IDexpert. the better known of the two, provides instructional design 
decision support and aims at a level of automation of CBT development through the 
creation of a series of "transaction shells," each of which is intended to handle a specific 
type of instruction by adapting reusable course component structures. 

Conversely, some commentators have argued that the key to improved performance and 
quality in development must lie in developing appropriate skills in instructional designers. 
Duchastel insists that the instructional designers must be in "creative control" (Duchastel 
1988). If you accept this position, the question becomes how one can develop that 
expertise as quickly as possible. In addition to conventional team- building strategies, 
technology can perhaps play a role here, also. Duchastel has argued that designers need 
to have access to their own design experience in some expedient fashion, as well as to 
the cumulative experience of design teams. Misselt (Misselt 1989) has similarly argued 
the need for a courseware design library and has provided some analysis of the 
characteristics required of such a library. Other, more explict strategies have also been 
explored. RRI (Research Reference Interface) is a component of IDexpert that provides 
"robust" explanations justifying the design prescriptions provided by IDexperfs 
instructional design engine (Li et al 1991). IDE is a hypertext-based tool that allows 
developers to record reasons and explanations for design decisions along with course 
content. It provides for the management (compilation and retrieval) of this information 
as well as the means to represent it in different ways. These capabilities facilitate group 
decision making, make developers more reflective through requiring an explicit statement 
of design decisions and their underlying rationale, and provide a database of design 
decisions and explanations from which to learn. 

Cross training of CBT development teams is another area where gains can be made. 
Communication of requirements across functions will be facilitated if team members have 
a better understanding of the different tasks. Recent developments such as hypermedia 
and CD-I and DV-I applications require greater cooperation and collaboration among 





208 



QUALITY IN TEACHING AND TRAINING 



team components, a larger, more central, role for graphics design, and a different, more 
creative (narrative), approach to course design. With this increased interaction among 
different functions, cross training may be especially beneficial. In some forms of 
development, cross-training has been shown to cut new product introduction time by as 
much as 50% (Mussel white 1990). 

What Do Instructional Designers Really Do? 

ID theory is not an area of knowledge that is highly developed. While there exists a large 
number of procedural models, which are very similar to one another, for doing 
instructional design there is little research and hard data concerning what instructional 
designers actually do. The standard procedural models provide a general methodology for 
developing instructional artifacts; they do not address the specific strategies and tactics 
and representations that an instructional designer utilizes when performing ID tasks. 
Better understanding of ID as a design activity per se is needed if major breakthroughs 
are to be made. To date only a very little research has been carried out in this area (eg. 
Pirolli and Greeno 1988; Pirolli and Greeno 1990; Jonassen and Wilson 1990). 

Eliminating Prototyping 

ID is generally justified on the grounds of effectiveness, not efficiency, and it is well 
known that under normal "real world" constraints all steps of the conventional ID models 
cannot be carried out in the degree prescribed. Recent improvements in methodology have 
been made in CBT development by combining elements of rapid prototyping with some 
components of traditional ISD. Major gains in time compression can be achieved if the 
prototyping stage, which is itself highly intensive, can be eliminated. 

Prototypes play several roles in CBT development: feasibility (a prototype can provide 
a very good sense of the total development effort required to produce the complete 
product, where new tools are being used and/or new designs are being implemented); 
verification of the effectiveness of the design (prototype evaluation with end users); 
communication of standards to production teams and agreement on standards with the 
customer. Ideally, prototypes should not play a role as a medium for extensive design 
experimentation, unless the context is R & D, rather than a CBT production operation. 
Properly exploited, prototyping eliminates any possibility oi the need for expensive 
revisions arising during the production phase. 

Elimination of prototyping as a major stage will only be possible w'ien: (a) production 
tools allow for more modular construction of CBT, to facilitate revisions; (b) clients have 
greater familiarity with CBT and hence are more fluent in de&criblng the quality level 
(instructional design features, aesthetic and functional characteristics) they require and 
better able to understand what they are being offered, and (c) our knowledge progresses 
to the point where cumulative experience, new procedural models, and detailed 
prescriptive theories can replace the role of prototypes in ensuring effectiveness. 

QUALITY INDICATORS 

A number of examples of quality indicators are given below. But don't monitor 
everything. Select a few indicators (or devise your own) that are meaningful and focus 
on them. Don't build a bean-counting bureaucracy. 



MANAGING CBT DEVELOPMENT 



209 



Overhead 

Develop procedures to monitor overhead as a percentage of processing cost 
(activity-based accounting). Take total costs for direct labour, materials, processing and 
overhead. Subtract labour and materials, and divide the balance by processing costs. Now 
for each project, multiply its processing costs by the remainder to get a contribution to 
overhead. Add project costs for labour, materials and processing to get total project costs. 

Customer Turnover 

Monitor the percent customer turn-over. Long-term customers are easier to sell on 
services, reducing marketing costs. A high customer turnover indicates dissatisfaction 
with quality in some sense. 

Development Cost 

Monitor the ratio of development hours to course hours. A decreasing ratio shows an 
improvement in quality, probably in the design process. 

Unit Cost 

Monitor unit cost. A declining cost indicates improvement. 
Unit Effort 

Monitor unit effort. A declining rate indicates improvement in cycle time. Monitor both 
effort and cost because in some cases effort may decline while cost does not (wage 
settlements may increase costs). 

Utilization Rate 

Monitor the job queue using the utilization rate, U = A/u, or percent of time busy. 
Utilisation rate is the ratio of the mean arrival rate of jobs (units/h) to the mean service 
rate. For example, an arrival rate of 2.1 units/h and a service rate of 1.9 yields a 
utilisation rate of 1 .10. Anything over 1.0 means »^hs cannot be completed during normal 
hours and overtime is necessary. A rate of 0.85, representing undercapacity scheduling, 
is suggested as an experimental target. 

Cost Index 

Monitor the cost index for a CBT project. It is the ratio of actuals/estimates. A value 
greater than 1.00 represents over expenditure and the need for corrective action. 



210 



QUALITY IN TEACHING AND TRAINING 



Achievement Index 

Monitor the achievement index. It is the ratio of actual percent complete on a project to 
the planned percent complete at any given time. A value less than 1.00 indicates 
underachievement, ie. the project is slipping. 

Status Index 

Monitor the status index. It is the ratio of the achievement index to the cost index. As a 
rule of thumb, 0.9-1.1 is a normal range, 1.1-1.3 and 0.7-0.9 requires management 
attention/action, >1.3 and <0.7 requires immediate management action (Gore and Stubbe 
1988). Plot the cost, achievement and status indices on the same chart. 

Change Orders 

Monitor the number of change orders. A declining number shows an increase in quality 
in the design process. 

Benchmark Trendlines 

Monitor the trendline in benchmarks. A decreasing trend shows an improvement in 
quality. Note that by the time problems show up in trendlines. they may be fairly serious. 

Break-even lime 

Monitor the number of days to break-even on a given class of project. 

CONCLUSIONS 

We have presented a basic model for considering TQM. This should be regarded as a 
framework of interrelated considerations. The reader may select, at her own discretion, 
various elements or configurations of elements from among the various approaches and 
techniques that were outlined, to develop an approach that suits her particular 
organization. 

However, in the last analysis any approach to TQM must be systematic and systemic. 
Several of the techniques which are discussed above will yield incremental benefits if 
introduced in isolation. These gains not only may be incremental but may also be 
transitory. 

Those contemplating a move to TQM clearly do need to think carefully about the 
culture of their organizations with regard to innovation. The fullest benefits of TQM can 
only accrue when there is an organizational strategic commitment to the paradigm. 
However, much can be accomplished at the departmental level. 

It takes time to successfuly implement TQM and entrench a philosophy of continuous 
improvement. The need to improve competitiveness, and initiatives such as the 
widespread adaptation of ISO 9000, will eventually force many organizations to make 
rapid changes that may be very disruptive and counterproductive; it is better to plan 
implementations now, to ensure success. 



21C 



MANAGING CBT DEVELOPMENT 



211 



The concepts underlying the approaches encountered in TQM are comfortable ones to 
those in the field of instructional technology who have adopted a performance technology 
orientation - which deals with performance related factors in general, including job 
design, job support, technology, organization, procedures, as well as explicit training 
solutions. Training departments and organizations are thus perhaps in a position to lead 
the way in the introduction of TQM philosophy, principles and methods. 

References 

Albregtse R J, Hejka J A and McNeley P K (1991) Quality Management: How do you 
do it? Automation August pp 30-32 

Alonso R L and Frasier C W (1991) JIT Hits Home: a case study in rescuing 
management delays Sloan Management Review Summer pp 59-67 

Bellis-Jones R and Hand M (1989) Are Total Quality Management Programmes a Fact 
or a Management Fad? Management Accounting May pp 36-37 

Breuker J A and Wielenga B J (1987) Use of Models in Interpreting Verbal Data In A 
Kidd (Ed.) Knowledge Acquisition for Expert Systems: a practical handbook NY 
Plenum pp 17-44 

Brown D H (1991) Product Information Management: the great orchestrator Computer- 
Aided Engineering May pp 61-61, 66 

Collins A (1990) Quality Control as a Model for Education: it would improve our output 
Engineering Education May/June pp 470-471 

Creese R C and Moore L T (1990) Cost Modelling for Concurrent Engineering Cost 
Engineering June pp 23-27 

Cronk R D (1992) Tributaries and Deltas Byte January pp 172 

Doucet R M and Ranker R A (1990) SOCRATES, an Instructional Design Advisor 
Program Computers in Human Behaviour 6 pp 201-206 

Duchastel P C (1988) Models for AI in Education and Training Artificial Intelligence 
Tools in Education pp 17-28 Amsterdam North-Holland 

Duchastel P C (1990) Cognitive Design for Instructional Design Instructional Science 
19 pp 437-444 

Edosomwan J A and Savage-Moore W (1991) Assess Your Organization's TQM Posture 
and Readiness to Successfully Compete for the Malcolm Baldridge Award Industrial 
Engineering February pp 20-24 

Gibbons A S and O'Neal A F (1989) New Work Surfaces for Instructional Systems 
Design J of Interactive Instructional Development Spring pp 16-20 

Gore M and Stubbe J W (1988) Elements of Systems Analysis Wm C Brown Publishers 
College Division 



220 



212 



QUALITY IN TEACHING AND TRAINING 



Hauser J R and Clausing D (1988) The House of Quality Harvard Business Review 
May-June pp 65-73 

Higgens R and Messer G H (1990) Improving Instruction Using Statistical Process 
Control Engineering Education May/June pp 456-470 

IM Management Roundiable (1990) Concurrent Engineering, Global Competitiveness, 
and Staying Alive: an industrial management roundtable IM July/August pp 6-10 

Inglebsy T (1991) Time Waits for No Manufacturer) HP Manufacturing Systems June 
pp 4,6,9,12,14,17,19,20 

ITAC (1990) Putting the Pieces Together Information Technology Association of 
Canada Toronto 

Jonassen D H and Wilson B G (1990) Analyzing Automated Instructional Systems: 
metaphors from related design professions In M R Simonson and C Hargrave (Eds.) 
Proceedings of Selected Research Paper Presentations at the 1990 Annual Convention 
of the Association for Educational Communications and Technology (pp 309-328) Ames 
Iowa Instructional Resources Center College of Education Iowa State University 

Keys D E (1991) Five Critical Barriers to Successful Implementation of JIT and Total 
Quality Control Industrial Engineering January pp 22-24, 61 

Lee C (1991) The Customer Within Training July pp 21-26 

Li Z, O'Neal H F and Baker E L (1991) Developing a Research Reference Interface for 
Knowledge-Based Instructional Design Tools Educational Technology 31(8) pp 7-16 

Li Z and Merrill M D (1990) Transaction Shells: a new approach to courseware 
authoring J of Research on Computing in Education 23(1) pp 72-86 

Merrill M D (1988) Implementation of an Expert System for Instructional Design: 
Phase 2 - design document and technical report (Contract No. OPM-87-9041) 
Alexandria VA: Army Research Institute for the Behavioral and Social Sciences 

Merrill M D and Li Z (1989) An Instructional Design Expert Journal of Computer- 
Based Instruction 16(3) pp 95-101 

Misselt A L (1989) Needed: a courseware design library CBT Directions 2 pp 16-22 

Musselwhite W C (1990) Time-Based Innovation: the new competitive advantage 
Training & Development Journal pp 53-56 

Nandyal R K and Welch D (1991) Goals-Orientation, SPC and Incentives Quality June 
pp 12-14 

Pirolli P L and Grceno J G (1988) The Problem Space of Instructional Design In J 
Psotka, L Massey and S Mutter (Eds.) Intelligent Tutoring Systems: lessons learned 
(pp 181-201) Hillsdale NJ Laurence Erlbaum. 



MANAGING CBT DEVELOPMENT 213 

Pirolli P L and Greeno J G (1990) The Instructional Design Environment: technology 
to support design problem solving Instructional Science 19 pp 121-144 

Ranker R A and Doucet R M (1990) SOCRATES: A Computer-Based Lesson 
Development Advisor Educational Technology 30(3) pp 46-50 

Russell D M (1988) IDE: The Interpreter In J Psotka, L Massey and S Mutter (Eds.) 
Intelligent Tutoring Systems; lessons learned (pp 323-349) Hillsdale NJ Laurence 
Erlbaum 

Russell D, Moran T and Jordan D (1988) The Instructional Design Environment In J 
Psotka, L Massey and S Mutter (Eds.) Intelligent Tutoring Systems: lessons learned 
(pp 203-228) Hillsdale NJ Laurence Erlbaum 

Schonberger R J (1985) Operations Management - Productivity and Quality Business 
Publications Inc Piano Texas 

Shaw M L G and Gaines B R (1987) An Interactive Knowl'^e-Elicitation Technique 
Using Personal Construct Technology In A Kidd (EdO Knowledge Acquisition for 
Expert Systems: a practical handbook NY Plenum pp 109-136 

Sheridan J K (1991) Racing Against Time Industry Week June 17 pp 22-28 

Shina S G (1991) New Rules for World-Class Companies IEEE Spectrum July pp 23-26 

St. Charles D (1990) Don't Toss it Over - Break Down the Walls Automation June 
pp 68-69 

Sutton J R (1991) New Cost Management Tools Offer Competitive Advantage 
Industrial Engineering September pp 20-23 

Tag.chi G and Clausing D (1990) Robust Quality Hansard Business Review Jan-Feb 
pp 65-75 

Tripp S D and Bicheimeyer B (1990) Rapid Prototyping: an alternative instructional 
design strategy Educational Technology Research & Development 38(1) pp 31-44 

Turino J (1991) Making It Work Calls for Input from Everyone IEEE Spectrum July 
pp 30-32 

Vesey J T (1991) The New Competitors: they think in terms of "speed-to-market u 
Academy of Management Executive * 5(2) pp 23-33 

Wielenga B J and Breuker J A (1986) Training of Knowledge Engineers Using a 
Structured Methodology In T Bernold (Ed.) Expert Systems and Knowledge Engineering 
Elsevier Science Publishers 

Ziegfeld R, Hawkins R, Judd W and Mahany R (1985) Preparing for a Successful 
Large-Scale Courseware Development Project In R Barrett (Ed.) Text, Context and 
Hypertext The MIT Press Cambridge MA 



222 



31 Towards quality management in 
training design 



Meurig Williams and Graham Carr, Training Development Group, Lloyds Bank 
pic, Hindhead, GU26 6QZ, UK. 

INTRODUCTION 

An integrated quality management policy is applied throughout the training development 
cycle, using a consistent approach across all training media. This strategy is based on 
all-round commitment to quality and on sensitively defined standards. Quality 
management techniques and independent quality reviews and audits successfully 
complement established validation techniques. The paper outlines the development of the 
strategy, the skills and conditions necessary for its success, its benefits and its relationship 
to international standards, and includes a case study of the use of quality management 
tools in CBT courseware production. 

Lloyds Bank Training Development Group (TDG) designs training for 35,000 staff 
throughout the United Kingdom, using both traditional and technology-based delivery 
methods. Quality management techniques, originally introduced in the development of 
computer-based training (CBT) and owing something to software engineering, have been 
successfully extended to other media. 

The forerunner of TDG, Lloyds Bank Distance Learning Centre (Slade 1989) was 
rapidly expanding the production of CBT packages and courses to meet the needs of the 
Bank's Branch Information Technology (BIT) project, a major internal customer. Both 
design and programming of many courses were being subcontracted to external suppliers. 

Project teams were following the conventional systematic training design process. A 
training requirements study led in turn to an outline design and detailed design. Then the 
programmer received a script specifying screen layout, content, program branching, 
course navigation, user interface and any other responses to student interaction. Staff 
external to the project approved the requirements study, and tested the completed course 
before its release, but there was no other independent review. 

In pre-release testing, reviewers recorded observations on course design, content and 
operation. An unacceptable proportion of defects (defined as negative observations 
requiring re-programming) was occurring. These ranged from typographical and 
grammatical errors to serious software bugs, and included factual errors and perceived 
poor training design. They contributed substantially to project costs and delays in 
completion. 

ATTACKING THE PROBLEM 

A member of staff (the first author) was appointed, and given the necessary independence 
and authority, to tackle the quality problem. Technical problems that could only have 
been introduced at the programming stage were the most urgent and important, so a 
technical production guide (Lloyds Bank 1990) was developed to ensure that all 
concerned were aware of relevant screen design conventions and equipment limitations, 
and to facilitate communication between designers and programmers. Both internal staff 
and external subcontt actors received copies. 

In the meantime observations of defects were being systematically recorded, classified 
and graded according to criticality, later using Pareto analysis. This allowed informed 
decisions to be taken about which areas it was possible to standardise. Concurrently, 



QUALITY IN TRAINING DESIGN 



215 



independent reviews were introduced on completion of critical intermediate stages in 
courseware development, such as the outline design and the programming script. These 
were intended to trap potential difficulties early enough to prevent their becoming 
expensive problems. 

Thus the classic quality management techniques of process observation, event recording 
and design review were introduced. 

DEVELOPING THE SOLUTION 

As a means of specifying technical limitations the technical production guide was a 
success, but in non -technical matters it did not reflect a consensus of design views and 
so was not widely accepted. A parallel investigation of the difficulties of communicating 
design information (England 1990) showed that in a fast-changing field like training 
technology, individual projects might have to adopt their own conventions. It became 
clear that a standards-based approach to quality was valid but would succeed only if 
environmental influences were taken into account. 

Time and cost, of development, observance and enforcement, compared with benefit, 
help to determine whether or not a standard is issued. So does demand, whether from 
designers, quality assurance (QA) or production staff. What any one standard covers is 
determined by delivery medium, demand and enforceability, while the principal factors 
affecting effectiveness are usability and acceptability. 

As described above, the first effective standard, the technical production guide, was 
developed empirically. A pre-release review of a number of CBT courses revealed 
defects. Classification enabled attention to be given to critical areas and an ideal situation 
to be determined. Hence a standard for the critical areas could be defined and guidelines 
issued. The next step was assessment of subsequent projects against the standard. 

Currently, when a new standard is required, an example of the item (process or 
product) concerned undergoes an initial assessment, by means of inspection, to determine 
what constitutes a defect. All observations are recorded. The resulting Pareto analysis 
leads first to revealing what needs standardising, and then to discussions with interested 
parties and the definition of a standard. Guidelines for new designers are then produced, 
to help them understand how the standard is intended to be interpreted and in what 
circumstances exceptions may be advisable. This cycle is repeated as many times as 
necessary. 

The actions described - inspection, recording, analysis, addressing standards, producing 
guidelines, and assessment, re-inspection or validation - resemble the traditional 
courseware development cycle and also the classic quality control loop (Figure 31.1), with 
the assessor or tester being the sensor and the reference being the standard. 

The review at each stage of course development is a logical part of the training 
validation plan which is built in lo each project. It often uses familiar techniques like 
course piloting with designers* peers or with the target population. Assessment against 
a standard may be eased by using checklists and, sometimes, testing plans. It is 
nevertheless a task requiring a variety of well-developed skills and judgment in the 
assessor, depending on the medium used for the training. In most areas of training design 
there will be borderline cases, where technical infringements do not affect training 
outcomes or would cost a great deal to correct. However, each assessment should produce 
a record of findings, a correction agenda and a possible improvement action programme. 



*• *w -4. 

9 



216 



QUALITY IN TEACHING AND TRAINING 



Actuator 



Sensor 
(tester) 



input 



Process 



Reference 
(standard) 



4 
I 
I 
I 

JL 

Comparator 



Measurement j 



Figure 31.1 Vie classic control loop 
THE ROAD TO SUCCESS 

A combination of independent review and acceptably defined standards and 
well-presented guidelines thus form the foundation of a workable quality management 
system. Its success requires acceptability, independent assessment and a means of 
completing the improvement action loop. 

Acceptability is assisted by involving project teams closely in the review process and 
in the definition of standards, and by ensuring that the reviewers are appropriately skilled. 
The standards themselves must be easy to understand and apply. 

Independence is provided through the existence of a quality assurance team whose 
members, although Lloyds Bank TDG staff, are not members of project teams. 

The improvement action loop is addressed by feedback meetings open to all design and 
production staff. These address specific topics and plan subsequent action, such as a 
project for writing guidelines. 



QUALITY IN ALL TRAINING MEDIA 



In 1991 Training Development Group extended its quality effort to media other than 
CBT. The techniques described above are now applied to all training design. Guidelines 
for writers and editors have been produced, and in early 1992 observation data was being 
gathered from classroom sessions in order to analyse the delivery of face-to-face training. 

In the process of extending the system to other media QA staff acquired new skills, 
and began to be consultants to the four design teams, rather than inspectors. They need 
skills in writing, editing, proof reading, software testing and interface design, and 
observation, as well as in quality management. Coaching and giving feedback is important 
in the review process and good inter-personal skills are vital, as is a full knowledge of 
the training design and delivery process. 



QUALITY IN TRAINING DESIGN 



217 



BENEFITS 



Directly quantifiable benefits are discussed in the case study. Among the indirect benefits, 
the approach is extensible as required, uses many existing skills, can be incorporated into 
initial project costings and so can be closely monitored, is widely applicable and is 
sensitive to designers' feelings. External suppliers can easily be involved, without 
conflicting with their own quality systems. Quality audits can be incorporated, and vast 
amounts of paperwork are not generated. The visibility of the system is an encouragement 
to TDG's suppliers and customers alike. 

Unlike the approach adopted in BS5750 (BSI 1987) TDG's approach is not wholly 
inspection-based and can be implemented fairly gradually in particular areas. Unlike the 
British Standard also, it is results-based rather than purely activity-based (Schaffer and 
Thomson 1992) as its primary focus is on improvement action, not fidelity to procedures 
alone. 



CASE STUDY: USING QUALITY MANAGEMENT TOOLS IN 
CBT COURSEWARE PRODUCTION 



Extensive data from the production and testing phase of CBT courseware development 
has been collated and analysed. The major source of quality problems have been 
identified by using simple statistical techniques. Designers and programmers have thus 
been enabled to improve quality by taking particular care in critical areas. The case study 
illustrates the benefits and problems arising from the previously largely untried use of 
quality management tools in developing CBT courseware. 

The quality improvement process adopted during CBT courseware development consists 
of four distinct stages: identification, analysis, improvement via communication, and 
monitoring. 

Identification 

The identification stage can be the part most organisations find difficult to accept. The 
fact tha. the identification process is considered necessary indicates there are problems 
which need addressing, and there is thus room for improvement. As a corollary, this 
suggests that both processes and people will need to change which can be cause for 
significant resistance. 

Accepting the sensitive position the quality assurance (QA) team are in, it is realised 
that it is not always "what you say" but "how you say it" that counts, and much effort 
is expended in this type of approach. For instance specific training on coaching and 
feedback techniques considerably assist this role. In TDG, designers and programmers 
want to produce better training material and are therefore willing to consider and 
implement improvement ideas. It is for these reasons QA can feel at ease in offering 
genuine remedies to common problems and remain confident in the fact they will be 
welcomed and implemented. 

Many inconsistencies (observations) occur throughout CBT production, and the basic 
method of recording these is on a QA report form. The columns show the observation 
number (sequential), where in the course the observation occurred, what it was, and 
recommendations (if any). The "importance" column is described in detail later. The 
remaining four columns show the process of correction until final recheck by QA. The 
forms are not for the sole use of QA members as the designer or other interested parties 




218 



QUALITY IN TEACHING AND TRAINING 



are actively encouraged to use them. 

Each course will generate a number of these forms and therefore a significant number 
of observations. Following a 1t brainstorming H session the lengthy "classification of 
observation" list shown below was produced. 



A Key Functionality (D) 

B Key Functionality (P) 

C Help (D) 

D Help(P) 

E Interactions ie questions (D) 

F Interactions ie questions (P) 

G Graphics (D) 

H Graphics (P) 

I Application Simulation (D) 

J Application Simulation (P) 

K Spelling and Punctuation 

L Technical Accuracy 

M Tutorial Text (D) 

N Tutorial Text (P) 

O Mouse 

P Observation Readvised 



These classifications are intended to cater for all possible inconsistencies likely to be 
encountered during procl iction of a CBT course and, for clarity, have been assigned 
individual letters. The (D) and (P) refer to designer and programmer respectively. 

Analysis 

The analysis stage uses numerous quality management techniques to ascertain the extent 
(ie frequency) of the observations being made throughout course production, and identify 
major areas of concern. 

A simple "tally chart" was devised to collate and classify the observations from three 
CBT courses, one internally designed and internally programmed (course one, released 
in 1989), one externally designed and internally programmed (course two, 1991), and one 
internally designed and externally programmed (course three, almost ready for release in 
April 1992). This revealed remarkable consistency in the distribution of different types 
of deficiency. 

From the sample of observations from course one, the classification H M" ie Tutorial 
Text-Design provided the most frequent comments. This is where the programmer ha3 
correctly followed the designer's script but for example, a sentence or paragraph does not 
read correctly, and the observation is consequently highlighted and attributed to the text 
design. The next highest was the "K" classification (Spelling and Punctuation) with "B" 
and "F" classifications (Key Functionality - Programming and Interaction (Questions) - 
Programming respectively) also featuring highly. The latter two classifications refer to 
the usability of the keys lor navigation throughout the course, and although the design 
script can sometimes provide incorrect instructions for the programmer, it is more often 
found that the observation relates to the programming related deficiency. 

The second course included a "simulation" of the application as a method of testing the 
learning. This shows similar priority to course one in terms of Tutorial Text - Design but 
Key Functionality - Programming came second pushing Spelling and Punctuation into 



QUALITY IN TRAINING DESIGN 



219 



third position. Understandably the design and programming of the "Application 
Simulation" also feature highly. Another interesting point is the appearance of Tutorial 
Text - Programming ("N") in fourth spot, as this indicates inability to reproduce the text 
on screen from the designer's script. 

Course three, designed and produced most recently, again shows similarities to the other 
two courses. The spelling and punctuation has in fact reduced in significance but the 
appearance of the programming of graphics ("H") is now a feature. Being the first 
"Mouse" controlled course to be monitored the appearance of comments under 
classification "O" was, to a certain extent, anticipated The latest classification added to 
the list (ie "P" Observation Readvised) was a direct result of analysing the QA report 
forms and noticing the recurrence of previous comments. 

Understandably an abundance of information has been produced, which renders 
concentrating on many areas at once, totally impracticable. We therefore prioritised the 
observations in terms of frequency, which allows comparisons across individual courses. 
This was achieved with the aid of Pareto analysis, 

The overall results from course three confirm the findings in terms of priorities 
mentioned above (ie M,B,N,H,0,K). We were also able to focus on the observations by 
importance as touched on previously in the "QA report form". Each observation is 
categorised in terms of importance as follows: 



Category 1 Critical Defect 



Category 2 Major Defect 



Category 3 Minor Defect 



rendering release of course highly unlikely ie execution 
error (software failure) where the course has "crashed" 
and needs reprogramming, or where some text has been 
incorrectly keyed which could seriously inhibit the 
transfer of learning. 

making release unlikely but not impossible, ie key 
malfunction on one screen but fully functional 
elsewhere, or where a graphic does not hinder the 
learning but could be improved on, or difficulty in 
understanding how to proceed, 
where release is still likely if insufficient time to 
correct, ie spelling or punctuation error. 



Each level of observation is likely to be corrected before release, but this level of 
importance prioritises the work involved. 

The analysis of observations by category assist the QA team in identifying where to 
focus attention. By analysing Category 1 observations only, we found the top three 
classifications all focus on programming inaccuracies. 

Many more of these charts have been produced which gives us clear indications where 
to concentrate our initial efforts in order to make the most impact. Another technique 
being investigated is that of the "cusum" chart. This refers to the cumulative sum (hence 
"cusum") of observations found in a sample number of screens. In essence a reference 
or "target value" is subtracted from each successive sample observation and the results 
cumulated. Values of this cumulative sum are plotted and "trend lines" are drawn on the 
resulting graph. If this is approximately horizontal, the value of the variable is about the 
same as the expected value. An overall slope downwards shows a value less than 
expected, and if the slope is upwards it is greater. 

Another technique is that of the "cause and effect diagram" (Ishikawa) as shown in Fig. 
31.2, which provides an overall view of the processes that could lead towards a 
nonconformity report. 



220 



QUALITY IN TEACHING AND TRAINING 



Failure to 
follow 
design 




Text design 
Outline \ Spelling 
design \ / 



Inadequate- 
pilot 



Inadequate 
detailed — 
design 



Design 




USABILITY /SOFTWARE 



Programming ^ of c,arit V 



SIMULATION \ LEARNING 

ERROR \ PROBLEM 



Testing Programming 




NONCONFORMANCE 
REPORT 



Failure 
to 

observe 
standard 



poor oversight 



Figure 31.2 Cause and effect diagram (Ishikawa diagram) 



Improvement Via Communication 

As mentioned, it is not possible to improve all the problems at once, so we have tended 
to concentrate effort on a selection of the more important and frequent observations. The 
improvements currently being worked on or already achieved include: 

Tutorial Text - Design ("M") 

Improved reviews of designer's script. Many of the design observations should be 
picked up at the design review stage, as should Technical Accuracy C'L"). This would 
dramatically reduce the classification of observation list so that (ideally) only pure 
programming deficiencies remained. 

Many reviewers find difficulty in visualising the paper version of text on a screen 
resulting in more comments when the material is later reviewed on the screen. This 
is being investigated by producing the pure script on screen but without the usual key 
functions so as to restrict comments to content, screen design and layout only. 

Key Functionality - Programming CB") 

If the above were successful, much more could be achieved in this area by further 
developing tools such as "Fault Tree Analysis" (FTA*s) which would help to pinpoint 
exactly why a programmer was allowing inaccuracies. It could be inexperience, lack 
of time, insufficient testing, etc. 




QUALITY IN TRAINING DESIGN 



221 



The creation of a "shell" for navigation purposes (ie "Exit" key "Help" key "Next" 
key etc) will standardise the process, and significantly reduce this classification 
without stifling design. 

Many courses are now becoming "Windows" based, and the use of the recorder 
function in "Windows" allows selected key presses to be automatically remembered 
and reproduced on demo; A . This encourages a more efficient testing process to be 
maintained, thereby increai 'ng the speed of course release. 

Spelling and Punctuation ("K") 

This has been improved in recent courses by using a programming language that is 
able to import the designer's script thereby cutting the keying-in process by half (and 
consequently virtually eliminating Tutorial Text - Programming ("N") classification 
in the process). 

These improvements are only likely through effective communication and a willingness 
to improve. It is up to QA to identify, analyse, encourage, support and coach so that both 
the training provided and the method of producing that training are, and continue to be, 
of the highest quality. 

Monitoring 

We need to continue the improvement process by constant monitoring. As technology 
advances so do the problems. The difference in complexity of, for example, course one 
and course three is enormous. This is proven by comparing the type and frequency of 
classification of each course. It is true to say that as soon as you have agreed a standard 
(or preferably - a guideline) so the parameters change. It is for these reasons a monitoring 
process can, in itself, present difficulties. However we are able to compare the number 
of observations per sample screens, and also indicate whether we have been successful 
in shifting the percentage of observations from importance (category) one to two and 
from two to three, to show improvement. This information is regularly communicated to 
other TDG members whereupon the Identification stage will restart. 

References 

BSI (1987) British Standard 5750: Quality systems BSI London 

England E (1990) Streamlining Computer Based Training Production: improving 
communication between commissioners and contractors Aspects of Educational and 
7 raining Technology XXIV 

Lloyds Bank (1990) Technical Production Guide Distance Learning Centre (internal 
document) Lloyds Bank Hindhead 

Schaffer R H and Thomson H A (1992) Successful Change Programs Begin with Results 
Harvard Business Review Jan/Feb pp 80-89 

Slade A (1989) How Lloyds Bank Utilises Distance Learning Training Technology 
July/August 



23Q 

9 



32 Achieving quality in networked 
interactive video 



Peter Willis and John Eary, Scottish Police College, Kincardine-on-Forth, FK8 
4BE and National Computing Centre, Manchester, Ml 7ED, UK 

INTRODUCTION 

The undoubted growth of computer based training and interactive video in the past ten 
years has been almost exclusively concerned with individualised learning. By contrast, 
classroom based training methods during this period have increasingly used group centred 
and team approaches to promote more active participation. The National Computing 
Centre (NCC) has developed an interactive group training system (VISTRAIN) with the 
Scottish Police College to help train police super/isory officers in crowd control at major 
spectator events, such as football matches. 

VISTRAIN is a multimedia, multi-user network training system using video based 
scenarios under the control of a facilitator. The system is designed to test decision 
making and communications skills in time-critical situations, A multimedia approach was 
chosen because of the difficulty of mounting realistic training exercises. The trainees will 
be able to act out their roles in a multimedia simulation. Video material shot at major 
football matches has been extensively used to give an authentic feel to the decisions 
taken. 

It is believed to be the first known application of networked interactive video, within 
a co-ordinated group learning environment. This approach involves technology enhanced 
training in a group learning environment, and is expected to have a significant impact on 
management training methods since it will enable new levels of sophistication and realism 
to be incorporated into exercises to practice decision making and communication skills 
under pressure. 

BACKGROUND 

The control of crowds at major spectator evenX such as football events involving around 
40,000 spectators, requires a high level of decision making and communication skills. 
The incidence of violent disturbances at football matches in many European cities is 
unfortunately a common occurrence. Police forces are being faced with increasing 
problems in planning and policing major football matches and it is recognised that senior 
officers need to be trained to high standards. 

Effective training in crowd control and similar situations requires the intelligent 
application of procedures in realistic circumstances. Football match commanders, 
responsible for the safety of many thousands of people, are required to make many rapid 
decisions and communicate clear instructions in an often "highly charged" atmosphere 
where noise levels are extremely high, making good communication very difficult. 

Little formal training in crowd control has previously existed because of the difficulty 
of mounting realistic practical exercises which would simulate conditions experienced at 
a real football match. 

One approach that has been tried is paper based exercises where teams of officers are 
given notes describing a situation and asked how they would respond. These exercises 
have been observed by the authors. They lack realism and do not put sufficient pressure 
on the participants to prepare them for actual and often dangerous events. 



NETWORKED INTERACTIVE VIDEO 



223 



THE PROJECT 



The Scottish Police College, in seeking a means of developing decision making and 
communication skills which, in addition to testing these skills, would also simulate the 
atmosphere and conditions experienced at a football match consulted the National 
Computing Centre (NCC). The consultation with NCC resulted in the joint submission 
of a competitive bid for funding to the UK Government's Employment Department, 
which was successfully accepted as part of the Technology in Learning programme. An 
award in the region of £100,000 was made, and the project to plan and develop the 
exercise commenced on 1 December 1989. 

The Scottish Police College is the central police training establishment, serving all eight 
police forces in Scotland, representing some 14,500 officers, and all levels of training are 
provided. The college has considerable experience in the use of computer based training 
and interactive video, particularly when this is applied to learning in groups. 

The Scottish Police College role in the project ha'j been to provide the subject matter 
expertise and the supply and production of video material. The college has also 
conducted formal evaluation trials of the system as it is used as part of senior training 
courses at the college. 

The National Computing Centre is a membership organisation run on a commercial 
basis. Profits from products and services are invested in the future of effective 
information technology for users. NCC Training has developed innovative CBT and 
interactive video courseware for over seven years. 

NCC Training has provided the project management, instructional design, software 
development and integration of the system. The two organisations have worked closely 
together sharing expertise. 



The system is used in a specially prepared exercise which forms part of senior police 
training courses. Sixteen senior officers participate. The officers attend a briefing at 
which the layout of the stadium and match details are given. They are split into four 
groups of four officers taking one of four senior police officer roles: match commander; 
officer in-charge inside the ground; officer in-charge outside the ground; divisional 
control. 

On the screens of their team workstations they see video scenes that relate to their role. 
They have only a partial view of the situation, eg the group taking on the role of officer 
in-charge inside the ground can only see views inside the ground and cannot see what is 
happening outside the ground. The network is used to synchronise the video scenes. 

The teams are required to respond to various "cues", ic potentially serious incidents to 
which they should respond. These incidents include moving video, radio and telephone 
messages. The teams are required to discuss the actions they should take in response to 
these incidents and relay these decisions via a personal radio to the appropriate teams, 
whose "view" changes accordingly. Other teams in turn must respond to the messages 
that are received and decide upon their actions and so on. 

A member of the college staff acts as facilitator to keep overall control and provide 
advice. The facilitator is able to monitor the teams* progress by listening in on the 
personal radio channels. He or she can also observe what the teams arc viewing on their 
video display on a bank of four duplicate monitors. 

The facilitator is required to z^sess the teams* performance during the exercise. The 
facilitator is provided with an on-screen storyboard indicating the appropriate actions 



THE SYSTEM IN USE 





224 



QUALITY IN TEACHING AND TRAINING 



that each team should take in response to the events. By the use of a mouse and 
especially designed puii down windows, the facilitator is able to quickly record the 
actions taken by the teams. 



MATCH 
COMMANDER 



OFFICER-IN-CHARGE 
OUTSIDE THE GROUND 




OFFICE R-IN-CHARGE 
INSIDE THE GROUND 



^0 



l ^i =3 



Figure 32.1 The network 

Technology is also employed to assist the participating teams. The teams* workstations 
make use of a WIMP environment to offer support to the teams. Using a mouse the team 



NETWORKED INTERACTIVE VIDEO 



225 



can select context sensitive help. There is also a browse facility so that the team can 
recall text and pictorial information from the briefing to assist them in their decision 
making. 

The exercise is divided into four scenarios, before kick off, first half of the match, 
second half and dispersal after the match. There are some eight or nine events in each 
scenario which can be selected by the facilitator from the workstation. Examples of these 
events include the selling of forged tickets, illegal entry and the explosion of a tear gas 
canister. 

After the first scenario has been played a debrief session is held. The system assists 
the facilitator to select those events where wrong decisions have been taken or have been 
communicated incorrectly. 

During the debrief session, which involves all sixteen officers, the selected events are 
constructed by combining the video cues together with the teams' radio messages. These 
have been digitally recorded onto a personal computer incorporated into the system. 
During the debrief participants are able to review the actions taken and identify 
alternative actions and their likely result. 

The remaining scenarios are played in a similar way, but the teams change roles. Thus 
by the end of the exercise they have had the opportunity to experience all four roles. 

A formal evaluation of the system was carried out over a 9 month period at the College. 
The resulting report has demonstrated that the delivery of a high quality learning 
experience has been achieved. Those who have had previous experience of policing 
football matches have indicated that the degree of realism achieved was outstanding. 
Those without such experience have developed a real insight into the many pressures and 
problems facing a match commander and feel much better equipped to cope with such 
a responsibility. All indicated that there was no comparison between this exercise and a 
"paper feed" exercise and all found the technology easy to use. 



The system that has been developed for the project is called VISTRAIN (Video-based 
Integrated System for Training Applications). VISTRAIN is a multimedia, multi-user 
network training system using video based scenarios under the control of a facilitator. 
An essential feature of VISTRAIN is that it allows users to participate in training 
exercises where each has only a partial view of the total situation upon which decisions 
must be taken. 
The VISTRAIN system has three main components: 

• a network of interactive video workstations linked to a facilitator workstation 

• a two channel dedicated intercom network 

• a software "shell" which, in addition to synchronising the video displayed, 
enables the facilitator to control the triggering of various training incidents; mark 
and record the performance of the participants; and reconstruct the events and the 
participants responses during the debrief session. 

The equipment used for the system comprises four Commodore Amiga 2000 computers, 
fitted with Commodore genlocks and attached to Sony 3300 laservision videodisk players. 
The facilitator's workstation is based on an Amiga 3000. These workstations are linked 
in a star network using ASAP's Amiox network boards. A PC compatible Commodore 
Bridgeboard computer is fitted to one of the Amiga 2000 computers to digitally record 
the personal radio messages and replay these at the debrief. 



EQUIPMENT USED 




226 



QUALITY IN TEACHING AND TRAINING 



QUALITY APPROACH IN THE DESIGN OF THE SYSTEM 



The design of the system followed NCC Training's standard methodology to ensure high 
quality was achieved in the design of technology based training courseware: 

• agree training objectives 

• determine training "events" 

• produce an outline design 

• identify suitable video and textual material 

• conduct a technical feasibility study 

• produce a functional specification of the system. 



An early meeting was held at the Scottish Police College to formulate training objectives. 
The meeting was attended by a number of college staff, including the Commandant and 
Deputy Commandant. Two senior Scottish police officers responsible for crowd control 
at major football matches also contributed. South Yorkshire Police were represented by 
an officer with considerable experience of crowd control. 

Some thirty detailed objectives were identified. They related to three major objectives - 
that during the exercise participants will demonstrate the ability to: 

• communicate adequately and properly with all parties involved 

• take appropriate action within areas of responsibility 

• apply appropriately their knowledge of relevant legislation and regulations. 

The more detailed enabling objectives were further refined and incorporated in the outline 
design documentation. 



During the discussion of the training objectives a number of typical incidents were 
identified which would provide suitable training opportunities. The incidents were 
collated to form a series of training events. It was decided to categorise these into four 
main scenarios: 

• pre-match 

• first half of the match 

• second half 

• post match. 

The initial plan was to allow fifteen minutes for the playing of each scenario. This was 
found insufficient given the need for the large number of interactions between the 
participating teams. 

The general background was established against which these events would take place, 
eg an all-ticket football match, capacity crowd, known history of animosity between 
supporters etc. 

An important decision taken at this stage was that the ground would be fictitious 
composed of elements of actual grounds. It was felt to base the scenarios on a specific 
ground would risk alienating officers from other forces who may not feel the training 
events were relevant to their own situation. 



TRAINING OBJECTIVES 



TRAINING EVENTS 




9 

ERIC 



NETWORKED INTERACTIVE VIDEO 227 



OUTLINE DESIGN 

The training objectives and training events were incorporated in an outline design 
document. This document also included the agreed target audiences, a primary audience 
of police officers, and secondary audience drawn from the other emergency services and 
staff from the football clubs and authorities. The roles of the participant were identified. 
To accommodate the sixteen attendees of a typical course, it was decided to allocate four 
officers to each of four roles: 

♦ the match commander, the officer in overall command in the control room 

♦ the officer in charge inside the stadium 

♦ the officer in charge outside the ground 

♦ divisional control (responsible for officers beyond the immediate vicinity of the 
ground). 

Each team was to be shown video sequences and receive radio messages. The teams 
were given a limited time in which to discuss a situation and determine the best course 
of action. Teams were allowed to have the ability to call up information presented at the 
briefing for the match and pictorial information including street maps and layout of the 
stadium. Context sensitive help facilities would also be available to assist the teams in 
their decision making. 

The role of the facilitator was also defined. The facilitator would control the exercise 
by triggering the training events and could monitor the exercise by listening in to the 
radio messages sent by the teams and keep track of the resources deployed. 

The facilitator needed to log the actions taken and mark the performance of the teams. 
The main categories identified were: 

♦ correct action taken and decision clearly communicated 

♦ correct action taken but inadequately communicated 

♦ incorrect action taken 

♦ no action taken where action was required. 

FUNCTIONAL SPECIFICATION 

A functional specification was prepared, based on the outline design document and 
consideration of how the system would operate. The specification covered the following 
aspects: 

♦ video source 

♦ team workstations 

♦ central facilitator's workstation 

♦ verbal communication 

♦ data networks 

♦ video and audio distribution 

♦ help facilities. 

This document was to form the basis for an investigation of the technical opions and 
therefore focused on the essential aspects of the system. 

The functions of the team workstations were described and it was decided that the 
remaining decision time would be displayed to add pressure on the decision making. 



23 



|ERjC 



228 



QUALITY IN TEACHING AND TRAINING 



Alternative ways for the teams to communicate were considered. Typing was regarded 
as an unnatural way for formulating verbal commands and the participants could not be 
assumed to have keyboard skills. A displayed menu of possible actions would limit the 
number of options available to the teams. It was therefore decided to simulate a personal 
radio system. NCC investigated the use of a proprietary intercom system which utilised 
the 240 volt main as the carrier signal. However this could not simulate a two channel 
radio network. 

The various tasks required of the workstation to support the facilitator were identified 
for both the playing of scenarios and in the debrief. It was decided that a time log would 
be used for recording the radio messages so that they could be played back in sequence 
with the video sequences interspersed. 

The initial design was based on a storyboard showing the preferred actions by each 
team in response to a particular event. The project team was fortunate to receive 
excellent co-operation f rom two experienced officers responsible for policing major crowd 
control events. The first scenario was developed as a prototype in which various 
conventions were developed. These included: 

♦ an event number and time; 

♦ the cues that would trigger decisions; 

♦ the actual text of pre-recorded personal radio messages or descriptions of video 
cue; 

♦ the expected actions of the four teams. 

A significant decision was to determine the "model answers" rather than all possible 
options as this was likely to be confusing to the facilitator. A specially designed 
storyboard was devised to show the interactions of the four teams. 

MANAGEMENT TRAINING APPLICATIONS 

VISTRAIN addresses a training requirement common to all police forces. It has already 
attracted attention from police forces throughout the UK, Europe and the USA. The 
system has now been successfully used on twelve courses and has won three awards 
including the European Technology Training Award. 

Although designed to help train police officers in crowd control at major spectator 
events, the approach used to produce VISTRAIN has significant potential applications to 
other areas. VISTRAIN is a unique software framework onto which any scenario could 
be scripted. 

The approach is applicable where training is required in decision making and 
communications commonly required in management training. In particular, where one 
or more of the following training factors apply: 

♦ situations where decision makers have only a partial view of the total situation; 

♦ role playing is required; 

♦ training takes place in groups; 

♦ facilitation is preferred to instruction; 

♦ communication skills need to be practiced; 

♦ training needs to have a high level of realism. 

The significance of the VISTRAIN approach is that it takes the sophisticated technology 
of multimedia from being training for the individual to being applied to the equally 
important development in participative teamwork training. 



237 



Section 6: Quality in Technology 

This final section is concerned with quality as it impinges on the technology that we use 
in education. We begin with David Hawkridge's fundamental and controversial appraisal 
of the state and place of educational technology as we approach the end of the twentieth 
century - are we still locked into the behaviourism that originated much earlier in this 
century? How can we assure "Quality in modernising educational technology"? 

Perhaps some evidence for development (or lack of it) might be found by a careful 
inspection of the study of current trends undertaken and reported by Donald Ely in 
"Trends and issues in educational technology". 

In "Training quality and new technology", Antonio Bartolome describes current 
initiatives being taken in Spain to discover the levels of utilisation of media in the 
classroom and the training initiatives being implemented to improve the quality of use. 
An application of CAI methods and technology to the laboratory teaching of electronics 
in Japan is addressed by Yoshikazu Araki in his consideration of "Connecting lectures 
and laboratory learning with CAI". 

With "Graphical routes to quality courseware" we move into the realms of multimedia 
and hypermedia, where Charles Lamont and Philip Barker illustrate techniques that can 
help us to produce quality courseware for CBT applications both more efficiently and 
more effectively. Finally, in "Enhancing the quality of student- centred mathematics", 
David Bowers and Rod Burrell look at applications specifically in the area of 
mathematics workshops and describe their use of software to set up meaningful dialogue 
between computer and student. 



23S 



33 Quality in modernising educational 
technology 



David Hawkridge, The Open University, Milton Keynes, MK7 6AA, UK 
INTRODUCTION 

Educational technology purports to be modem, even futuristic, yet uses 1960s 
psychological theory as its basis. Is the perceived quality of educational technologists ' 
performance and products being adversely affected by this cracked and obsolete 
foundation? How can educational technology courses be successfully modernised in the 
1990s? Can we borrow from cognitive science? Are we integrating information 
technology? Can we rise to the political and moral challenges? 

My thesis is that it is time to modernise educational technology. This is not an easy 
task. I shall first examine the 1960s theory, drawn mainly from behavioural psychology, 
now discredited, and the thinking of the audio- visual movement. Familiar tools such as 
behavioural objectives are blunt, but continue to be used. Many UK educational 
technology courses, usually at the postgraduate level, follow the example of similar 
courses in the US by teaching behavioural psychology and the systems approach. I shall 
speculate on the extent to which we can borrow ideas from cognitive science instead, 
bearing in mind problems being experienced in that discipline. I shall look at how well 
we are integrating information technology into our theory. Finally, I shall identify 
political and moral challenges being flung at educational technologists in general. 

A CRACKED AND OBSOLETE FOUNDATION 

The term "educational technology" came into general use in the late 1960s, after the 
heyday of programmed learning. Ely (1968) had two definitions: technology (hardware) 
use in education, and the application of behavioural science to education. If we include 
the period in which programmed learning was being developed, it seems as though the 
dominant paradigm has been the same for about forty years. To put it another way, it 
is now a cracked and obsolete foundation. 

It is time to modernise educational technology. This is not an easy task. We probably 
all agree with Gagne (1987) that educational technology involves the systematic 
application of knowledge derived from scientific research. It is a fact t however t that the 
majority of educational technologists have clung to behavioural psychology and continue 
to advocate the use of familiar tools such as behavioural objectives. Many UK 
educational technology courses teach behavioural psychology and the systems approach. 
The same psychology underlies much of the thinking about performance-based models 
of education and the definition of competencies. Behind the jargon of this conference are 
hidden assumptions about the nature of learning and what constitutes success. My own 
opinion is that we are in danger of saying to the world of education "never mind the 
quality of educational technology, just feel the width". 

Consider the books for 1980s courses about educational technology in British 
universities, polytechnics and colleges. Romiszowski (1981) built entirely on ideas of the 
1970s, derived from North American behaviourism and the systems approach. As Eraut 
(1989) commented, these ideas do not constitute established knowledge, though they may 
be useful to us, nor do they take account of constructivist views of learning. Percival and 
Ellington (1984, revised in 1988) took a similar line, presenting a handbook long on 



232 



QUALITY IN TECHNOLOGY 



techniques, short on theory. Rowntree (1974, revised in 1982), widely used in teacher 
education, was anything but theoretical perhaps because he preferred to appeal to readers' 
commonsense, not their knowledge of philosophy or psychology. Spencer's (1988) book 
on the psychological roots of educational technology depended greatly on behaviourism. 
He showed how Thomdike and Pavlov laid the foundations for a science of learning, but 
gave Watson and Skinner, both eminent behavioural psychologists, most of the credit. 
He also drew particularly on Bloom, Briggs, Gagne, Keller, Mager, Popham and Tyler, 
all Americans in the behaviourist mould, and he was right to do so, even if it underlined 
how much British educational technology had been influenced by them. Spencer did go 
further than this to consider the work of Piaget, Bruner and Vygotsky, all of them 
interested in how children internalise what they learn as cognitive structures or models. 
Finally, he referred to Papert and Salomon's research, which may yet link educational 
technology to cognitive science. 

A couple of years ago, Romiszowski (1990) summarised the position. The educational 
technology movement had been criticised, he said, for insisting on too specific, 
product-based, uniform objectives; for placing too much emphasis on prior design and 
development of materials, followed by dissemination of the standard messages to all 
learners indiscriminately: for emphasising behaviours mastered rather than ideas 
processed and correction of errors rather than reflection on the implications of viewpoints; 
and for h Jng shallow and superficial rather than encouraging the processing of complex 
multi -faceted content. 

Our neglect of cognitive science may be due to its rhetorical flavour. Behaviourists 
stick closely to the analysis of behaviour, and do their best, even in the case of humans, 
to develop their discipline along strictly scientific, positivist lines. Cognitive scientists 
have much less empirical data available on which to base their theories. Like certain 
nineteenth century psychologists, they are attempting to analyse what happens inside 
human beings' brains, not outside. They derive their ideas from linguistics, artificial 
intelligence and psychology, but as yet cognitive scientists can tell educational 
technologists very little about how to guide learning according to the principles of their 
science. 

The search is on for a science of learning, a set of normative rules that, if followed, will 
lead to successful teaching. After four decades of effort by cognitive scientists, however, 
it is clear that these ideas have been slow to reach children and their teachers, even with 
the advent of microcomputers. There is a large gap between the research laboratories and 
classrooms. For example artificial intelligence specialists admit that some of their work 
has been misguided, and that they have explored blind alleys. Their field data are hard 
to interpret for practical purposes. 

Intelligent tutoring systems, which act like a good tutor, represent the extreme example 
of cognitive science applied to education. Analysis of possible errors lies at the heart of 
the development of intelligent tutoring systems, yet, as the behaviourists found, human 
learning is extraordinarily complex and predictions are very difficult. Tom (1984) does 
suggest that even if there are no breakthroughs, cognitive scientists may have an impact 
on teaching by encouraging teachers to regard their pupils more as thinkers and less as 
bch avers. 



It is clear from many of the papers at this conference that educational technologists are 
making great use of information technology. That is not surprising. As technophiles 
rather than technophobes, we welcome the technology. In doing so, we are continuing 



INTEGRATING INFORMATION TECHNOLOGY 




MODERNISING EDUCATIONAL TECHNOLOGY 



233 



the line of thought we inherited from the audio visual movement that started in the 1920s. 
The challenge to us, however, is how will the field of educational technology be changed, 
to incorporate information technology where desirable, and to exploit its special 
characteristics for the sake of education? At the moment, we are assailed by 
technology-led solutions to educational problems. That is not what educational 
technologists stand for: we believe in student-led solutions in which information 
technology plays an appropriate role. It is a pity that in the United States and to some 
extent in this country too, the introduction of information technology into education has 
simply enabled educational technologists to apply the "failed" behavioural psychology all 
over again. Perhaps this was inevitable, because many technological innovations are used 
at first to do faster what was done before, rather than to change the nature of what is 
done. I need only mention the drill-and-practice programs that flooded the US education 
market in the late 1980s and you will see what I mean. 

Of course, British schools are now using computers for many other purposes besides 
drill-and-practice. Some educational technologists now advocate the use of Logo, which 
is based on construct vist psychology. Many others expect children to learn how to use 
databases, word processing programs and spreadsheets, if not graphics and other software. 
In doing so, however, these latter educational technologists appear to be atheoretical, 
advocating practical courses of action without theoretical justification. We should also 
consider at least a few of the critics of information technology's ideology in education. 
For example, Chandler (1989, 1990) argued that computers project an ideology that 
conflicts with the purposes of the educators who put them into schools. The nature of 
information technology is such that we cannot avoid its influence on teaching and 
learning. He gave three examples: data-handling, simulation and word- processing. Let me 
take the first of these, in particular. 

Data-bases are introduced into schools to teach children data-handling: how to record, 
sort, search for and display data. As Chandler said, these students learn 
"information-processing", using information technology, despite the fact that computers 
only contain data, not information. Data are changed into information by humans, not by 
computers. Computers deny the human origin of information. Humans negotiate meanings 
through discourse. Chandler said this is not merely a semantic argument: the language of 
computers threatens to redefine the world in its own terms. 

Chandler also pointed out that because humans give the computer power and authority, 
data stored elsewhere appear to lose some of their significance. Data-handling systems 
also distort information, diminishing the value of things they cannot record, including 
much everyday knowledge. Creators of databases have to be willing to tailor their 
intentions. Searchers have to limit the questions they ask. 

Chandler was worried that using computerised databases may lead students to believe 
that thinking is data processing. He asserted that thought and memory require building 
and rebuilding of models of the world, not through passive data capture but through 
"interpretation and elaboration of information according to changing hypotheses ... We 
create ideas: computers can't". Databases disregard meaning. 

Computers promote the notion of their own objectivity, said Chandler. Their users often 
ascribe more authority to databases than to the printed word. Because databases contain 
standardised data, without details of sources, they seem misleadingly objective: each item 
carries equal weight, equal certainty. The creators' biases are invisible, the items are 
stripped of their origins and context. Data-processing requires fragmentation of the whole, 
with loss of meaning, declared Chandler. It prohibits a holistic perspective. 

Saettler (1990) suggested that there was a hidden agenda in incorporating information 
technology into education. He said that improved managerial efficiency was the goal: 
more students taught in less time by fewer teachers using less space and at lower cost. 



24: 



234 



QUALITY IN TECHNOLOGY 



This agenda is no longer hidden. Politicians and educators in the UK and US, if nowhere 
else, have turned to information technology as a potential means of increased efficiency, 
despite the fact that the evidence from studies of costs is not reassuring. The technology 
seems always to be an add-on cost. So educational technologists must ask themselves 
whether they are bringing information technology into their work in response to such an 
agenda, or because the information technology actually serves the theory they hold. 

POLITICAL AND MORAL CHALLENGES 

If educational technology is to be truly modem, its practitioners must also respond to 
political and moral challenges, which happen to come mainly from the Left. 

The Frankfurt School of critical theory was formed in 1923. Its founders, including 
Marcuse, developed studies of authoritarianism. In the 1930s they began a critique of 
new mass media, particularly radio, and of how they were being used by the fascists. 
After World War II, they mounted a critique of mass culture, which they saw as 
affirming the rule, or hegemony, of bourgeois values because it offered escapism and 
gratification, not insight. Habermas and others who came after Marcuse used critical 
theory to attack capitalism more directly. In particular, some critical theorists argued that 
capitalism was a system in which objects and the processes that produced them were 
valued by marketplace exchange, that is, they were turned into commodities or 
commodified. The commodification of the whole of society was proceeding apace, said 
these critics, recalling that the function of marketed objects was to profit the capitalist. 

But what does critical theory have to do with educational technology? Burrell and 
Morgan (1979) state that proponents of critical theory "seek to reveal society for what it 
is, to unmask its essence and mode of operation and to lay the foundations for human 
emancipation through deep-seated social change" (p 284). Translated for us, this might 
read "seek to reveal educational technology for what it is, to unmask its essence and 
mode of operation and lay the foundation for learners* emancipation through rooting it 
out". Harris (1987) thinks of educational technology as a species of positivism, and says 
that all the major authors in the field of critical theory consider positivism to be a 
particularly acute and modem form of domination. Bowers (1988) asserted that 
educational technology, far from being neutral, is a powerful force that will alienate 
individuals, destroy community values and, through promoting a falsely objective view 
of the world, serve right-wing political and economic forces. 

Left-wing critics like these find intellectual support in the work of Lyotard (1984), 
whose message about the "post-modern condition of knowledge", in present day western 
industrialised society, is that capitalists are turning knowledge into a commodity, to be 
bought and sold in a marketplace which they dominate. Fox (1989), who aims at open 
and distance learning, rather than directly at educational technology, sees these as a 
branch of the "knowledge industry". He suggested ihat "knowledge and the opportunity 
to learn are benefits to be bought by private individuals rather than fundamental rights 
belonging to every member of a democracy" (p 270). In the circumstances, open and 
distance learning become a "dream instrument" for the control of the ruled. This may be 
a surprise to those of you like me who are engaged in designing this learning. Burt 
(1991) suggested that "educational technologists cannot escape the epistemological crisis 
which confronts all the other social disciplines. The new critical theory aims to address 
the crisis by making previously suppressed voices heard" (p 229). Burt saw the 
educational and training technology literature as tending to express the ideologies of 
technological optimism and national economic competition, close to the dominant 
ideology of British and US society. 



24C 



MODERNISING EDUCATIONAL TECHNOLOGY 



235 



This sort of criticism is hard to take for educational technologists who feel that they 
have frequently participated in benevolent, well -intended educational reform. They are 
forced to admit that educational technology can be used for the wrong ends, but not that 
educational technology is always used thus. Even in general it does not have such a 
malign influence, we hear them say. Critical theorists do not deny the good intentions 
of educational technologists: they simply say that the results of such endeavours enhance 
the promotion of "falsehood and negative value". 

The best defence against the critical theorists is to point out that they propose 
substituting their own hegemony for the capitalist one, and that educational technologists 
desire to be slaves to neither. Critical theorists are essentially nihilistic: they offer no 
constructive ideas to those they accuse. Nevertheless, modern educational technologists 
would do well to develop their own political consciousness, just as they should modernise 
their theoretical foundations, and examine why they are integrating information 
technology. 

References 

Bowers C A (1988) The Cultural Dimensions of Educational Computing: understanding 
the non-neutrality of technology Teachers College Press New York 

Burrell G and Morgan G (1979) Sociological Paradigms and Organisational Analysis 
Heinemann London 

Burt Gordon (1991) Opinion: culture and ideology in the training literature Educational 
and Training Technology International 28(3) pp 229-237 

Chandler D (1989) The Purpose of the Computer in the Classroom In Beynon J et al 
(eds.) Computers into Classrooms: a critical appraisal Falmer Press Lewes 

Chander D (1990) The Educational Ideology of the Computer British Journal of 
Educational Technology 21(3) pp 165-174 

Ely D P (1968) Educational Technology as Instructional Design Educational Technology 
8 4-6 (referred to by Saettler 1990) 

Eraut M (1989) Conceptual Frameworks and Historical Development In Eraut M (ed.) 
The International Encyclopedia of Educational Technology Pergamon Oxford 

Fox S (1989) The Production and Distribution of Knowledge Through Open and 
Distance Learning Educational and Training Technology International 26(3) pp 



Gagne R (ed.) (1987) Instructional Technology Foundations Lawrence Erlbaum 
Hillsdale New Jersey 

Harris D (1987) Openness and Closure in Distance Education Falmer Press Lewes 

Lyotard J-F (1984) The Postmodern Condition: a report on knowledge University of 
Minnesota Press Minneapolis 



269-280 





236 QUALITY IN TECHNOLOGY 

Percivai F and Ellington H (1984) A Handbook of Educational Technology Kogan Page 
London 

Romiszowski A (1981) Designing Instructional Systems Kogan Page London 

Romiszowski A (1990) Shifting Paradigms in Education and Training: what is the 
connection with telecommunications? Educational and Training Technology International 
27(3) pp 233-237 

Rowntree D (1974 revised 1982) Educational Technology in Curriculum Development 
Harper and Row (1974 edition) Chapman (1982 edition) London 

Saettler P (1990) Tiie Evolution of American Educational Technology Libraries 
Unlimited Englewood Colorado 

Spencer K (1988) The Psychology of Educational Technology and Instructional Media 
Croom Helm Beckenham UK 

Tom A (1984) Teaching as a Moral Craft Longman New York 



34 Trends and issues in educational 
technology 



Donald P Ely, ERIC Clearinghouse on Information Resources, Syracuse University, 



There are many ways in which trends couid be identified: expert opinion, panels of 
specialists, or informed observation. This study chose content analysis as the primary 
vehicle for determining trends based on earlier work of Naisbitt (1982) and his model 
(Jar.owitz 1976). The basic premise of these works is that current trends can best be 
determined by what people are saying publicly, through newspapers and magazines. This 
study, and those that preceded it, used the same basic procedure: identification of 
emerging topics in key publications over a period of one year. The literature of the field 
has been chosen as the best comprehensive coverage of current thinking and events in the 
field. A selected body of literature has been reviewed using a team of educational 
technology specialists. 



To maintain consistency from year to year, the same sources of information were used 
as in two earlier studies, Ely (1989, 1990), with a few exceptions. To aid in the selection 
of sources, the Moore and Braden (1988) report was used. This source reported the 
people, publications, and institutions of "high Prestige" that were identified by a survey 
of personnel in the field. The highest ranking journals and the dissertations produced by 
American universities that ranked highest served as two major sources of literature. 

Additional sources of data were the papers given at major national and international 
conferences and the input to the ERIC database in the field of educational technology. 
The ERIC system solicits unpublished materials such as reports, evaluations, studies and 
papers for review and, following evaluative criteria, selects the best for inclusion in the 
database. The Clearinghouse on Information Resources is responsible for the field of 
educational technology and therefore documents selected from that source are likely to 
represent current developments in the field. 
The complete set of sources used was: 

Journals (October 1990 - September 1991) 

British Journal of Educational Technology (United Kingdom) 

Education and Training Technology International (United Kingdom) 

Educational Technology 

Educational Technology Research and Development 
TechTrends 

Dissertation Sources (1991) 

Arizona State University Syracuse University 

Florida State University University of Southern California 

Indiana University 

Conferences (1991) 

Association for Educational Communications and Technology 
Educational Technology International Conference (United Kingdom) 
National Society for Performance and Instruction 
ERIC Input (October 1990 - September 1991) 



Syracuse, NY 13214, USA 



INTRODUCTION 



LITERATURE SOURCES 




238 



QUALITY IN TECHNOLOGY 



From the reviews of four coders, discussing more than 1300 items, came a list of "topics" 
that were most frequently presented in the literature. The recording units offer a first 
indicator of trends. Further anaiysis of each category and subcategory reveals sharper 
distinctions. At this point the key literature is added to the mix. Key literature consists 
of policy papers, reports and statistical data for each topic area that was published during 
the time period of the study. This information, together with the content analysis, was 
studied by the author who then drafted the trends and sent them to external reviewers. 
Changes were made based on their comments. 

CONTEXT 

This study answers the question, "Where is educational technology headed?" Technology 
does not move alone apart from the society in which it exists. Information and 
communication technology is being used in the home and workplace at all levels - local, 
state, regional, national, and international. To separate it from the context is to highlight 
its products rather that its uses and impact. Therefore, much of the discussion in this 
study involves the total fabric of technology in society rather than technology as an entity 
in itself. 

TRENDS 

Using a content analysis of the 1991 educational technology literature, the following 
trends have emerged: 

1 , The creation of technology-based teaching/learning products is based largely upon 
instructional design and development principles. 

There appears to be more evidence that materials developed for the purpose of teaching 
and learning use design principles that have their roots in cognitive psychology and 
instructional science. More than 15% of the items reviewed for this year's study were 
devoted to design and development. Major subheadings included message design, product 
development and course development. 

I'. Evaluation ha: taken on greater importance as the concept of performance technology 
has been further c* \ sloped 

More than 1 09c \>i the 1991 literature was concerned with some aspect of evaluation: 
proc< ;ss evaluation, product evaluation, cost/effectiveness assessments and formative 
evaluation. 

3. The number of educational technology case studies is growing and provides general 
guidance for potential users. 

More than 1 1% of the literature reported on specific use of media and technology in 
teaching/learning settings. Almost all the case studies were "successful " and many could 
serve as models for potential users. Very few reported "failure" or negative outcomes. 
About one-half of all the case studies related to computer use in teaching and learning. 
There were almost no cases of traditional media use or instructional procedures that have 
been "proven" in the past. 

4. Distance education is evident at almost every educational level in almost every sector. 
Distance education has become a major instructional force in education. A recent 

estimate is that 25%-50% of U.S. students are reached by distance learning technology 



243 



TRENDS IN EDUCATIONAL TECHNOLOGY 



239 



(School Technology News, 1991). There is probably no other single trend that 
encompasses the theory and practice of educational technology better than distance 
education. Its frequency in the literature confirms this observation. 

5. The field of educational technology has more and better information about itself than 
ever before. 

Eighty surveys about various aspects of the field were reported during the time frame 
of this study. For example, studies of the most frequently published.textbook authors, 
a list of current dissertations, the extent of microcomputer penetration in the schools and 
other such reports help to paint a quantitative picture of the profession. 

6. Computers are pervasive in the schools. Virtually every school in the U.S. has 
microcomputers. 

Computer applications permeated the literature of educational technology in 1991. 
Purposely omitted from the analysis of trends were 14 journals associated with computer 
assisted instruction (CAI) and conferences that focused on the computer as an 
instructional medium. Sometimes the items were directly focused on the use of 
computers in the classroom for direct subject-matter instruction but most referred to 
learning about the computer as a tool. Many items discussed the resistance or 
"roadblocks" to the use of computers in schools. 

7. Telecommunications is the link that is connecting education to the world. 
Telecommunications is an overarching term that describes electronic point-to-point 

connections between individuals and groups. Translated into electronic delivery terms, 
telecommunications technology includes connections that utilize existing telephone lines, 
dedicated lines, cable and satellite transmission. Some messages are intended to be 
interactive, such as electronic mail (e-mail), computer conferences and two-way audio and 
video conferences. Some are intended to be one-way, e.g., television directed to 
classrooms through cable and satellite systems. The dominant trend within 
telecommunications is networking - the electronic connection of individuals who have 
common interests. 

8. The teacher's role in the teaching and learning process is changing as new 
technologies are introduced into the classroom. 

Positive attitudes toward technology are emerging from teachers' unions and from 
teachers themselves. Introduction of such new approaches as Integrated Learning 
Systems (ILS) bring about new roles for the teacher. 

9. There is increasing pressure for the schools to consider the adoption of technology 
while, at the same time, concern is expressed for the impact of technology on children 
in society at large. 

This trend is a two-edged sword. Picas for the use of technology in the schools are 
increasing in frequency. Simultaneously, there are cries of concern over the impact of 
technology, especially television, on children and youth. 

10. Professional education of educational technologists has stabilized in size and scope. 
There may not be much uniformity in the titles of academic programs that prepare 

individuals to serve in the field of educational technology and the academic "homes" are 
not consistent from university to university but, in general, the field is holding its own. 
Programs tend to include similar content, are primarily offered at the graduate level and 
prepare people for similar positions. 




240 

References 



QUALITY IN TECHNOLOGY 



Ely D P (1989) Trends and Issues in Educational Technology 1988 Syracuse NY 
ERIC Clearinghouse on Information Resources 

Ely D P (1990) Trends and Issues in Educational Technology 1989 Syracuse NY 
ERIC Clearinghouse on Information Resources 

Janowitz R M (1976) Content Analysis and the Study of Sociopolitical Change Journal 
of Communication 26(4) pp 20-21 

Moore D M and Braden R A (1988) Prestige and Influence in the Field of Educational 
Technology Performance and Instruction 21(2) pp 15-23 

Naisbitt J (1982) Megatrends Warner Books Inc New York 

School Technology News (1991) "Wade Right In," Manager tells Educators (8)9 p 1 



24 



o 



35 Training quality and new technology 



Antonio R Bartolome, Dpt. de Diddctica i Organitzacio Escolar, Universitat de 
Barcelona, SPAIN 

INTRODUCTION 

The aim here is to describe an interactive course that attempts to improve the competence 
of teachers in the use of videos and computers and to discuss the outcomes. The material 
is being used in research concerning interactivity levels, path control, communication 
channel and cognitive styles. The paper includes results from research work undertaken 
in Catalonia, and the design of the interactive video course. 

Nine years ago, Chadwick (1983) announced the "Law of Hammer": if you give a small 
child a hammer, he will immediately conclude that all around needs a good hammering. 
In the educational field, give a trainer a new resource and he will immediately conclude 
that ... 

Perhaps it is the "Law of Hammer" that turns trainers upside down in computer assisted 
instruction (CAI), as they take the big leap to artificial intelligence, flying over hypertexts 
on their way towards new authoring languages. In my opinion, it is at this moment that 
they should emerge into the street from the multimedia sewer in order to get a glimpse 
of the unreal virtual reality world. Today we find ourselves in such fields as these, with 
more projects than realities. 

At Barcelona University, the Department of Curriculum and School Management has 
been training the teachers of the future in the use of educational technology for years, 
both with undergraduate and postgraduate courses. Several methodological tools have 
been developed for these courses. Among them, we have the "New Technology for 
Teaching", hypermedia with interactive videodisc support. 

NEEDS ANALYSIS 

From 1987 to 1992 several different studies were carried out in Catalonia about the use 
of technological resources in schools. In fact, this line of study has been followed by the 
introduction of the New Technology Work Group (GTNT). The research was financed 
by the Service for Popular Culture Foundation. This study investigated the way in which 
teachers used video in their teaching practice and was based on field work that was done 
by 6 observers in 50 schools chosen as a good cross-section of all the different schools 
in Catalonia (Bartolome and Ferres. 1992). 

Amongst the general conclusions were: video equipment was a resource that teachers 
handled a lot, but to which students had much less access (see figure 35.1). Even where 
pupils had access to video equipment, this was largely as viewers - 92.3% had viewed 
video, whereas only 18.5% had used a camera, out of a total of 4000 pupils. 

Here we must point out that, in relation to computing, the situation is different: the 
utilisation has a much smaller base, but the relative involvement of the students is much 
higher. 

And so, what is video used for? Out of 15 possible Video uses, the most accepted was 
"to transmit information to the classroom group". So we should not be very surprised 
when we realize that 78.8% of the 150 teachers who were asked had never even 
considered the video as a resource for consulting or studying on an individual basis. 

Thus, pupils get little access to the sets of equipment nor can they use the playing or 
recordings systems, nor can they get to a viewing system in order to do individualized 



24* 



242 



QUALITY IN TECHNOLOGY 



or group work. 

We obtained other conclusions from this work and these have provided the basis for the 
design of the course "New Technology for Teaching". 




H- Yes 
| No 



Schools in wliich teachers Schools in which pupils 

use video use video 

Figure 35.1 School utilisation of video 



CONCLUSIONS ABOUT VIDEO USE 

We can conclude from the research that the teaching method is not modified by the 
simple introduction of video technology in the teaching-learning process. Really, it is the 
complete opposite. There is a tendency to "domesticate" the media, forcing it 10 act as 
a second teacher. 

But this study suggests that there is room for more innovative uses, both creative and 
participative, where teachers actually integrate the technology in this way. 

Generally speaking, we cannot say that teachers have reached the stage at which the 
video has been completely integrated in the classroom. Most pupils see video as an 
isolated experience, and they do not look upon it as an integral part of every day school 
activities. 

Despite the relatively small amount of time for which the video is used in schools, 
pupils value its incorporation in a very positive way, as an aid to learning. In this sense, 
it is a pity that it is not used in a better way, both from a quantitative and a qualitative 
point of view. 

In general, the study shows that it is the most passive and the most one-way video uses 
that are incorporated in schools: the video player is used more often than the camera, and 
it is handled more by the teacher, than by the pupils; programme- viewing is more typical 
than other activities; more video lessons are watched than motivating programmes. 

This tendency towards a passive use of video can be explained by three factors: firstly 
the lack of teacher-training about more imaginative and participative uses, secondly the 
lack of an adequate technological infrastructure in schools, and thirdly because there is 
no time available for creating new "types of use. 

A further factor that must also be taken into account is that the manufacturers of 
educational video-programmes practically always produce the same video- programme 
model, without offering any other alternatives. 



250 



ERjC 



TRAINING QUALITY AND NEW TECHNOLOGY 



243 



Among the positive values, we ought to point out the fact that the use of the video 
workshop is on the increase in schools. Despite the lack of means for this activity and 
the fact that it only can be used by a small number of pupils, this is a very creative and 
participative formula. 

This study has shown that there is a significant deficiency in schools regarding 
audiovisual literacy and critical reading of media. Very few schools offer education in 
this field, and this means there is a very big reparation between school and society in use 
of media. 

Incorporating video in teaching practice do;s require an extra effort from the teacher, 
and therefore only those teachers who are particularly interested, or who are very 
enthusiastic, are likely to put this means into practice. 

A PROJECT TO IMPROVE TEACHING QUALITY 

From this and other research, some action plans were drawn up. One of these was to 
develop material about new technology use that laid emphases on the methodological 
aspects of this use. The final products can be defined as follows: 

♦ An optical videodisc entitled "New Technology for Education" incorporating 
motion pictures and still images about the introduction of video and computers in 
both primary and secondary education. 

♦ A control software that allows access to the different sequences, helps the user to 
get to specific information, gives access to a dictionary of technical terms, and that 
also includes written textual information about certain aspects of the subject. 

♦ Some versions include individualized and group learning programmes. 

Three aspects were taken into account in the course design: 

♦ The technical aspects of equipment 

♦ The communicative aspects of the media: media language 

♦ The educational aspects relating to how to use these media: resources 
methodology. 

COMMUNICATIVE ASPECTS 

The programme has two types of sequence: 
Base-sequences 

Present key aspects of computing, video and telecommunications in an intuitive and 
general way. They do not offer detailed information, but try to give clues tha'. will lead 
to further discussion, to suggestions for group-work, to encourage individualized study. 
They are video-sequences with moving images and sound effects, and their lengths vary 
between 20 and 120 seconds. 

Complementary-sequences 

Present certain aspects in an accurate and detailed fashion. In order to do this, they use 
a written text in the computer, which is illustrated and is based on still images shown 



244 



QUALITY IN TECHNOLOGY 



from the videodisc. 

Using any of these sequences, users can access the Dictionary and other buttons are 
available as illustrated in figure 35.2. 



Macintosh HD:CIDE/Soft:CIDE/Soft BflSICQ:NNTT.SoftUDJENG 



Computers in Education 



^panish 



Computers achieve different results in education 

IQI?* It is an object for studying : pupils learn how to get to know its technical characteristics; 

they also learn how to program it. 
U^^* It is also another means for learning - , it transmits information, it develops skills, it 

changes attitudes. 

iTj3> As a didactic means it is a tool in the hinds of teachers and pupils. 

Finally it is a tool for administration; economic or academic, in the same classroom. 



Contents of this chapter 



m 














Computers in Education 
Setting Up the Equipment 
Computers Room 
The computer the classroom 
Acquiriring programmes 

; T>arJw-Tr*-imnn 



Figure 35.2 Sample video screen display 



DIDACTIC PLANNING 

This programme has been designed to meet different needs: 

• As a tool to support teaching explanation in specific subjects - the teacher can 
offer support to his demonstration by using still images or by employing 
whichever video-sequences he considers appropriate. 

• As a means to generate small-sized or average-sized group dynamics the - 
video-sequences lead to group discussion or other role-playing type activities. 

• As support to group learning - the pupils use written guides in which different 
themes are presented. Each individual uses the programme to collect information 
about these themes, as well as using other resources such as books. Later, together 
with the teacher, the results they have achieved in their work are discussed. 

• A support to individualized learning - a computer programme directs the learning 
by asking a series of questions in which the pupils must use the videodisc, with 
different levels of interactivity, to answer these questions. 



25° 



TRAINING QUALITY AND NEW TECHNOLOGY 



245 



SHOULD EDUCATIONAL TECHNOLOGY BE USED IN 
SCHOOLS? 

The last aspect that we need to consider is to ask if it is really worthwhile to use these 
resources. 

We should like to be able to say that pupils, with the help of technology "learn more, 
in a better way and faster ...\ Unfortunately, according to Clark (1983), five decades of 
research have shown that there are no benefits to be gained in learning from using 
different teaching media. With regard to reducing the amount of necessary learning time, 
Clark points out that the new media hold a slight advantage over more conventional ones. 
Kulik et al (1980) carried out a meta-analysis between traditional courses and courses that 
used computer-based training, with results showing an effectiveness in favour of the latter 
type. This greater effectiveness was considerably reduced when the same teacher prepared 
and taught both the experimental and control group. In the long-term results, this 
effectiveness practically disappeared. 

But these considerations make a basic mistake: they try to make a comparison to show 
what happens when we teach old material with new resources. MacLuhan was probably 
right when he related contents and media: new media teach new contents, they address 
different aims than traditional media. 

The use of new technology is related in a strange manner to studies about the hidden 
curriculum and with the school that we present to our pupils: a shool which is separated 
and different from the world outside in which they live (Treffel, 1981). A school without 
computers teaches something to students: "this school is not related to your world, to 
your needs, to your future; we are not interested in your society". It does not matter that 
teachers do not think so: this is the content of the hidden curriculum. 

References 

Bartolome A Y and Ferres J (1992) L'Us Del Video a les Escoles de Catalunya Serveis 
de Cultura Popular Barcelona 

Chadwick C B (1983) Los actuales desafcos para la Tecnolog^a Educativa Revista de 
Tecnologga Educativa 8(2) pp 99-109 

Clark R E (1983) Reconsidering Research on Learning from Media Review of 
Educational Research 53(4) pp 445-459 

Kulik C et al (1980) Effects of Computer Based Teaching on Secondary School Students 
Journal of Educational Psychology 75 pp 19-26 

Treffel Jacques (1981) Presents et Fuiurs de l' Audiovisual en Education La 
Documentation Franchise Paris 



36 Connecting lectures and laboratory 
learning with CAI 



Yoshikazu Araki, Saitama Institute of Technology, 1690 Fusaiji Okabe Saitama, 
Japan 369-02 

INTRODUCTION 

As the number of college and university students rapidly increases, their educational 
needs, motivations, levels and abilities become more diversified. Also their attitudes 
towards their studies and their interests in college life are quickly becoming more varied. 
To make matters worse, the number of students who are unable to fully understand the 
lectures, and are therefore less likely to be willing to study, is gradually increasing. It 
may be because the students do not understand the materials being presented to them or 
that teachers are not making an effort to examine their students* progress. Perhaps the 
fault is with the teaching methods. The use of textbooks and handouts does not suffice 
to ensure the students' comprehension of a lecture. A professor {particularly of 
Engineering) cannot expect the students to easily grasp theories and schematic 
explanations without the use of visual aids. In Engineering, if we can explain theories 
with data derived not only from simulation but also from the actual device or circuit, 
lectures will be more persuasive and practical. Using personal computers, the author is 
introducing a technique to show real experimental data that verify the theories presented 
in the lectures. 

PROBLEMS WITH LABORATORY 

It has been usual in Japanese universities and colleges for the lectures and seminars to 
be given independently of the laboratory courses. The same teacher rarely does both. 
In order to provide the students with more practical knowledge, the laboratory courses 
are particularly important in the university curriculum for engineering. 

In order to maximise effectiveness, it is highly desirable that the relationship between 
the lectures and relevant laboratory learning be significantly strengthened. 

In an average Japanese laboratory course, a student is obliged to work with several 
other students because of the shortage of measuring equipment and space. Thus , some 
students are unable to participate in all the projects of the laboratory. Unhappily, the 
result is that those students tend to rely on their classmates for the material in their 
reports, and do not develop their own ability to research the subject thoroughly. 

APPROACH TO SOLVE THE PROBLEMS 

In order to solve these problems and to improve this situation. Imamura et al 1989 have 
suggested that teachers should make much more effort to improve their teaching 
techniques, by utilizing various new media. The author is developing such a new 
teaching system. It consists of three parts: first, the Pre-Laboratory-CAI Test (Pre-Test); 
second, the laboratory; and third, the Post-Laboratory-CAI Test (Post-Test). We call this 
system RECALL (Real Experience Computer Assisted Laboratory Learning). Figure 36.1 
shows the fundamental concept of RECALL system. 

The Pre-Test is done to check basic knowledge of experimental theories, equations and 
methods which are necessary for understanding and finishing the laboratory successfully 



2^/ 



CONNECTING LECTURES AND LABORATORY LEARNING 247 



within a limited time. If the result of the Pre-Test is less than 70%, the student should 
study the subjects using tutorial type CAI. Then he should take the Pre-Test again. If 
the result is over 70%, he is permitted to begin the laboratory. He is permitted to take 
the Pre-Test three times within 45 minutes. 



Pre- 


Test — d 


Quiz-type CAI 
Tutorial type 


Test 
CAI 






i 






Laboratory 


Experiments | 








i 




Post 


-Test — d 


Quiz-type CAI 
Tutorial, type 


Test 
CAI 



i 



[ Making Reports 1 
Figure 36.1 Fundamental concept of RECALL system 



COMPUTER ASSISTED MEASUREMENTS 

We have been developing a system, using a computer, to measure voltage-current 
characteristics of a bipolar junction transistor (BJT), a junction type field effect transistor 
(JFET) and a metal-oxide-semiconductor type field effect transistor (MOS FET). 

The hardware configuration of the system consists of common hardware, a curve tracer 
unit and signal generators. The common hardware consists of a personal computer, an 
I/O interface card and an analogue interface unit. The personal computer is PC-9801 
(NEC) with 8086 family CPU. The analogue interface unit converts external analog 
signals into digital data and vice versa with 12 bits A/D, D/A converters. The curve 
tracer unit is used literally as a curve tracer for the measurement of voltage-current 
characteristics of BJT and FET. The unit contains V-I converters and voltage amplifiers. 
The converter converts voltage supplied from the D/A converter into current (e.g. I B of 
a BJT). The amplifier is used for providing voltage (e.g. V CE of a BJT). 

Common software consists of some controlling programs and the main program. The 
former control data flow between analog interface unit and personal computer for each 
application. The latter integrates application softwares. 

Before starting the laboratory, students should know the theories behind these devices. 
In the laboratory, students not familiar with measurement equipment often acquire 
insufficient or incorrect data due to errors in operation, thus wasting time. This system 
reduces measurement time allowing students more time to spend on consideration of the 
main theme of the laboratory. 

In the laboratory, input, output and mutual transfer static characteristics of the devices 
are measured. All measurements are done automatically by following messages displayed 
on the CRT of the personal computer. Table 36.1 shows an example of measuring 



248 



QUALITY IN TECHNOLOGY 



procedures. Measuring steps for V DS -I DS output static characteristics of FET are as 
follows; 

1 Inputting student's number in order to store learning record. 

2 Inputting measuring conditions such as weather, temperature and humidity. 

3 Displaying the explanation of a measuring operation. 

4 Inputting a name of a device. 

5 Preparation for measurement. 
Setting i) output voltage V DS 

ii) maximum input voltage V GSo)tx 

iii) maximum output current 1^^ 

iv) step increase in output current I DS 

6 Measuring Vx> s -los output static characteristics. 

Here V DS and V GS are applied from the D/A convener. Values of I DS are taken 
into the A/D converter. 

7 Outputting V DS -I DS output static characteristics on a CRT and in a printed form. 

8 Measuring other characteristics. 

Alter taking the measurements students should learn the characteristics of the 
device by using their own data to answer questions such as the calculation of h- 
parameters, input resistance and output resistance. For example, from the result 
of Grounded Emitter output static characteristics, a student should calculate hpp 
at a certain point on the output characteristic curve according to a message 
displayed on a CRT. After calculating the h^ the student inputs the value and the 
computer checks it. These processes are repeated until the right answer is 
obtained. Thus students should come to understand the actual characteristics by 
the use of real data. 



After all the processes mentioned above are finished, the Post- laboratory- CAI Test (Post- 
Test) is taken to check understanding of the completed section of the laboratory. If the 
result of the Post-Test is less than 80%, the student should study the subjects using 
tutorial type CAI. Then he should take the Post-Test again. If the result is over 80%, he 
is permitted to finish the laboratory. The time allotted for the Post-Test is limited to 45 
minutes. At the end of the laboratory, students submit reports including data, problems, 
results of the Pre -Test and other considerations. 



Heasuring Flow 




( 1 ) 1 npu t t i ng 




(2) Preparation for neasurement 




(3) Measuring 




(4) Outputt ing 


Laboratory Experiment Condition 


Measurement f 


or Voi Ids output static characteristics of FET 




(D output voltage V os 




© aaxiaun input voltage Vgv»«* 




(3) aaxiaua output current Ios. t . 




© step increase in output current A li% 


Measurement 


for V t Ic output static characteristics of BJT 




OD output voltage Vt a 




(?) aaxi&um input voltage Vun. 




(3) aaxiausi input current It... 




0) step increase in input current Ah 



Tabic 36.1 Measuring procedures 



CONNECTING LECTURES AND LABORATORY LEARNING 249 



CONCLUSION 

The author has developed a new learning system where students measure characteristics 
of real semiconductor devices such as V DS -I DS or h parameters using personal computers, 
then confirm the theoretical characteristics by CAI. This system combines practical data 
and phenomena with theory which is taught in a regular lecture course. 

Through the RECALL System (the Pre-laboratory-CAI Test, the laboratory and the 
Post-laboratory-CAI Test) and CMI system, students* understanding of laboratory is 
accurately and easily evaluated. 

ACKNOWLEDGEMENTS 

The author wishes to thank Professor A Nakajima, Mr H Imamura, Mr Y Kobori and Mr 
T Ono for their advice. Also he would like to thank Ms E Chatillon for correcting the 
English. 

References 

Araki A (1991) CAI and CMI Considering Student's Achievement and Consciousness 
in Winterburn R (Ed) Aspects of Educational Technology Realizing Human Potential 
Kogan Page London 

Imamura H, Kobori Y, Ono T and Araki Y (1989) A New Teaching Method Using 
Computer Processed Real Data 1989 Frontiers in Education Conference Proceedings 
pp 105-110 



257 



37 Graphical routes to quality courseware 



Philip Barker and Charles Lamont, University of Teesside, Cleveland, UK 
INTRODUCTION 

Computer-based training (CBT) is concerned with the automatic delivery of training and 
learning material using a computer system. In the past, screen-based text has been the 
main method of communication. However, it has been shown that augmenting basic 
textual material with appropriately designed static graphic imagery produces a 
communication mechanism of significant pedagogic effectiveness. The use of moving 
pictures (in the form of TV quality video images and animated graphics) has similar 
potential for improving the quality of multimedia instructional delivery within CBT 
systems. Further improvements in the quality of CBT courseware can be achieved 
through the incorporation of hypermedia techniques (Richards and Barker 1992). 

Unfortunately, the development of multimedia and hypermedia courseware can be highly 
problematic unltss suitable development tools are used. For this reason a considerable 
amount of research has been devoted to the development of tools to automate the 
production of courseware (Barker 1990: Barker and Lamont 1992). The motivation for 
using these tools primarily arises for four basic reasons: first, the need to decrease the 
number of steps and/or phases involved in the overall courseware development process; 
second, the need to reduce the level of expertise needed to develop effective and efficient 
courseware; third, the growing necessity to cut down courseware development costs; and 
fourth, the need to reduce the amount of time devoted to producing instructional software. 

In this paper we describe an approach to courseware automation which we have found 
to be extremely effective. It is based upon the use of "custom editors" that are designed 
to augment and enhance the facilities provided by the graphical user interface to the 
PRO PI courseware development system (ASYS Computer 'Services 1988). PROPI is 
essentially a courseware generator that produces PC/PILOT code (Barker 1987; Conlon 
1984). In the remainder of this paper we provide a short description of the PROPI 
graphical user interface and then illustrate how custom editors (written in C) can be used 
to provide high-level, easy-to-use authoring facilities to ease the production of 
high-quality multimedia and hypermedia courseware. 

GRAPHICAL USER INTERFACES 

People interact with computers by means of appropriately designed human-computer 
interfaces (Barker. 1989). Screen-based interfaces fall into two basic categories: character 
orientated (based upon the use of a keyboard); and graphical user interfaces. A graphical 
user interface (GUI) relies upon the use of icons, images and pictures to support pointing 
operations based upon the use of a mouse. There is considerable evidence to support the 
claim that graphical user interfaces are easier to use and that they can lead to substantial 
increases in end-user productivity. 

A number of CBT authoring systems are now supplied with graphical user interfaces 
which act as 'front-ends' to an existing development tool. TenCORE and PC/PILOT are 
two examples in which this approach is used. In many other systems (such as 
HyperCard, ToolBook and Authorware Professional) the GUI is the only interface that 
is available to courseware authors. Because of their ease of use, systems of this sort are 
becoming extremely popular - mainly because users need not have any technical 
knowledge of computers or do not need to know how to write computer programs. There 



GRAPHICAL ROUTES TO QUALITY COURSEWARE 



251 



is another reason why these GUI-based authoring systems are growing in popularity: they 
provide a relatively easy way of developing high-quality courseware. 

The ease of use of a GUI-based authoring system stems from the fact that each generic 
type of courseware authoring function is represented on the author's computer screen by 
means of a reactive icon. PROPI, for example, has seven basic classes of icon: input, 
output, decision, mark, question, group and external. A fundamental property of these 
reactive icon. c , is that they can be selected (using a mouse) and cloned. A cloned icon can 
then be positioned on the CRT screen so as to form part of some more complicated 
graphical structure. 

In order to produce a piece of instructional software, a courseware author uses the 
screen -based menu of icons in order to construct a "lesson map". This is simply a 
pictorial representation of the logic flow and functionality that is to be embedded within 
the courseware. The lesson map is essentially a graph structure which is very similar in 
appearance to a computer flow-chart. Each type of node within a lesson map has 
associated with it a series of content "editor" templates. These arc used either to specify 
the pedagogic content of a node or to give control information that specifies how a node 
is to be delivered. Once a graphical lesson map has been produced it can be 
automatically converted into CBT code. 

CUSTOM EDITORS 

Authoring tools of the type described in the previous section provide a powerful 
mechanism for generating courseware provided, of course, that they are able to offer all 
the major functions that authors require. Invariably, due to the very nature of progress, 
there will always be some things that are missing. Some mechanism must therefore be 
provided which will allow users to "extend" an authoring environment in ways that are 
relevant to their own particular needs and requirements. This mechanism is provided in 
PROPI, for example, by means of "custom nodes" (ASYS Computer Services 1988). The 
purpose of custom nodes is to facilitate the automatic generation of PC/PILOT code for 
performing tasks which are not directly catered for by PROPI itself. 

A custom node (within a PROPI lesson map) is a generic node which, when invoked, 
causes the activation of an external (user written) "custom editor". The purpose of a 
custom editor is to generate CBT code from parameters entered by the courseware author 
while a CBT lesson is being developed. Obviously, there is no limit to the range of 
custom editors that a particular PROPI authoring environment can make available to 
courseware authors. 

The program code used to produce a custom editor can be written in any programming 
language (we use C for most of ours). During its execution a custom editor must 
generate a user-interface to facilitate the entry of the parameters that are needed in order 
to build the target code that is to be generated. Obviously the design of end -user 
interfaces to custom editors offers considerable scope for the application of GUI 
techniques and the provision of mechanisms that will minimise any errors that authors 
make with respect to the entry of code generation parameters. 

In the work that we have undertaken using PROPI, a wide range cf custom editors have 
been built for use with electronic books (Barker 1991) and for controlling various 
multimedia presentation devices. The following section presents a case study which 
describes one of the courseware automation projects in which we have made extensive 
use of PROPI custom editors. 



252 QUALITY IN TECHNOLOGY 

A CASE STUDY - MULTIMEDIA AND HYPERMEDIA CBT 



In this case study we describe some of the ways in which we have been using custom 
editors (within the context of courseware automation) in order to: (a) provide "user 
friendly" interfaces to "high technology" information display devices; (b) make available 
a standard and consistent graphical user-interface (GUI) to a range of different peripheral 
devices; and (c) improve the quality of CBT courseware through the incorporation of 
interactive multimedia and hypermedia techniques. The essential features of this 
courseware automation project are illustrated schematically in figure 37.1. 



Philips vP-705 Custom Editors 



video D.se Ed.tors 



Slngtl Iirt40l Dliptay 






VTdae Clip DltpUy 




Ravarsa Play 




Fcst Play 




SlSir Play (FhJ) 




Autto t ConUal 




Audio U Control 




Vldao MuU OrJOU 




ate 





Cu«or*. Ed *0' utti'y 



Hitachi CDR-1503S Editor* 



Stauc Plitur* D:tP:ay 



CDROJ.t Editors 



Functional Love! 



Video 
Disc 




Corrajct 


Disc 




Grsprrcs 
Boards 


Otr-cr 
Devices 



PROPI 
Courseware 
Automation 

System 



Generic Level 



System Level 



Figure 37,1 Multimedia custom editors for use in PROPI 

In order to "insulate" courseware developers from the complexity of programming these 
devices the GUIs employed in our custom editors are standard ones for all devices (or 
technologies - such as CD-I and DVI) within a given generic class. That is, no matter 
which actual device is employed for information delivery, the GUI allows a consistent 
style of use, has a standard appearance and provides access to a standard repertoire of 
cont' 1 options. Courseware authors can access the particular editor of their choice either 
by a menu selection strategy or directly by entering its name. 

To date we have implemented a full set of custom editors for the various types of video 
disc equipment that we use. Custom editors arc also available for basic compact disc 
equipment, selected graphics boards and one or two special devices such as a touch 
screen and a concept keyboard. Descriptions of the design, implementation and evaluation 
(from the courseware authors* point of view) of these custom editors are presented in 
detail elsewhere (Lamont 1991). Our current work with custom editors is now directed 
at extending the range of editors that we have available for use with digital optical 
storage technologies. Because tiiese technologies are likely to have a significant impact 
on courseware development for CBT, we are now extending our range of eduors to 
include facilities for handling digital video interactive (DVI), compact disc - interactive 
(CD-I) and compact disc extended architecture (CDROM-XA). Further details of this 
work are described in Lamont and Barker (1992). One very important aspect of this 
particular courseware automation project from both the courseware authors* and the 



2ii0 



GRAPHICAL ROUTES TO QUALITY COURSEWARE 



253 



students* point of view is "image reactivity". That is, the ability to embed and use 
reactive areas (or hotspots) within images and text so that when one of these areas is 
selected (using a pointing device such as a mouse) other related processes are initiated. 
Image and text reactivity is an essential requirement for the creation of hypermedia 
structures and their subsequent incorporation into CBT courseware. To this end we have 
designed (and are currently in the process of implementing) a range of custom editors to 
facilitate: (1) the insertion of hotspots into images; and (2) the linking together of image* 
text and sound within a hypermedia knowledge corpus that is held on digital optical 
storage media (primarily, CD-ROM). We believe that these custom editors will 
substantially improve the productivity of our authors and the quality of the courseware 
that they produce. 



Multimedia and hypermedia techniques are playing an increasingly important role in the 
development of quality courseware. However, the incorporation of these techniques into 
instructional software requires significant technical (and pedagogic) skills if they are to 
be used in an effective way. While it is difficult to automate the pedagogic aspects of 
authoring, the technical requirements of authors can be substantially reduced if 
appropriate courseware automation techniques are made available. We have found that 
graphical user interfaces (used in conjunction with custom editors) can play a significant 
role in easing the burden of courseware authoring for multimedia and hypermedia CBT. 



The authors are indebted to both the Science and Engineering Research Council (SERC) 
and to A.P. Chesters and Associates, 17 Newhall Court, George Street, Birmingham, B3 
1QA, for the financial support to enable this work to be undertaken. Financial 
contributions from University of Teesside, Eyetech Ltd and the Training, Enterprise and 
Education Directorate of the Department of Employment are also gratefully 
acknowledged. 

References 

ASYS Computer Services (1988) PROPI User's Manual ASYS Computer Services 
2601 North Shore Road Bellingham WA 98226 USA 

Barker P G (1987) Author Languages for CAL Macmillan London UK 

Barker P G (1989) Basic Principles of Human-Computer Interface Design 
Century- Hutchinson London UK 

Barker PG (1990) Automating the Production of Courseware in Farmer B, Eastcott D 
and Lanz B (eds) Aspects of Educational Technology- Volume XXIII: Making Learning 
Systems Work Kogan Page London UK 

Barker P G (cd) (1991) Electronic Books Special Edition of Education and Training 
Technology International 28(4) 



CONCLUSION 



ACKNOWLEDGEMENTS 





254 



QUALITY IN TECHNOLOGY 



Barker P G and Lamont C W (1992) Courseware Automation Through Custom Editors 
in The CTISS File Special Edition on Courseware Authoring Systems 

Conlon T (1984) PILOT - the language and how to use it Prentice Hall Englewood 
Cliffs New Jersey USA 

Lamont C W (1991) Human-Computer Interfaces to Reactive Graphical Images Draft 
PhD Thesis University of Teesside Cleveland UK 

Lamont C W and Barker P G (1992) The Design of Custom Editors for Digital Optical 
Storage Media Working Paper University of Teesside Cleveland UK 

Richards S and Barker S (1992) Improving the Quality of CBT Through Hypermedia 
paper submitted to AETT '92 International Conference University of York 6th-9th April 
1992 



262 



38 Enhancing the quality of student-centred 
mathematics 

David Bowers and Rod Burrell, Suffolk College, Ipswich, UK 
INTRODUCTION 

This workshop addresses how aspects of information technology can influence the quality 
of the learning experience and in particular how IT can promote a flexible response to 
individual student need. This will be related primarily to delivery within the area of 
mathematics. 

The need to meet customer demands in delivering a quality flexible response has led 
Suffolk College to investigate the use of authoring software which allows a one to one 
relationship between the student and the lecturer through the use of such software. 

The need to identify the diagnostic element of the student lecturer exchange can be 
satisfied by the logic built into the authoring solution. More sophisticated communication 
is non-linear, expressing at least a two way conversation: the use of a quality IT solution 
delivers this. 

The aim of the workshop was to encourage participants to identify quality delivery 
within the mathematics area, specifically related to information technology solutions; and 
to allow hands-on experience of a number of examples of related software. 

The objectives were: 

i) to set the scene within die wider educational arena; 

ii) to explain how Suffolk College has harnessed a number of organisational and 
technical solutions to deliver mathematics to a diverse student population; 

tii) to explore the use of authoring software as a link between the flexible response 
and the individual student need. 

SETTING THE SCENE 
Suffolk College 

Suffolk College is a classic example of a mixed economy institution. It plays an essential 
role in the delivery of Suffolk's Post- 16 education and training. This includes 
school-linked A level and vocational provision, non- advanced further education, adult and 
community education, and advanced further education. 

The Academic planning figures for the next three years project an increase in student 
numbers of 31 percent and all departments within the College arc addressing ways of 
enhancing the quality of student provision. 

Mathematics 

Recent developments in the qualitative nature of mathematics provision were discussed, 
as well as advances in computer technology. Together, these provide motivation for 
investigating the opportunities for computer based learning in mathematics, and workshop 
participants were invited to contribute their own experiences. In particular, the following 
have had an impact on mathematics provision: 



263 

9 



256 



QUALITY IN TECHNOLOGY 



♦ Common Skills, Core Competences, and Curriculum Entitlement - this new 
awareness in course design requires a formal response from maths education 

♦ Syllabus changes - these result in potential inconsistencies at course "interfaces" 
(eg GCSE. BTEC OND) 

♦ Mathematics Workshops - these necessitate a re-appraisal of delivery methods (eg 
resource-based learning) 

♦ Mechanical calculating devices - yesterday hand calculators, today 
microcomputers, tomorrow ...? 

Information Technology 
Hardware Developments 

Hardware performance has increased dramatically whilst, at the same time, costs have 
fallen. This has led to the availability of hardware to perform complex processing tasks 
at affordable prices. 

Software Developments 

The release of Microsoft's Windows 3.1 together with its multi -media extensions kit, will 
allow the ease of use of such media as CD-Roms, still and moving video, and sound, in 
many application areas. 

Delivery of the Curriculum 

The use of Computer Based Learning has already allowed the curriculum to be delivered 
in non- traditional ways, the coming of improved hardware and software products will 
allow the curriculum to be delivered in an even greater variety of ways. The College has 
to be able to facilitate this delivery. 

SUFFOLK COLLEGE CASE STUDY 

This case study served as a basis upon which participants could relate their own work. 
It also provided examples for critical evaluation and as a springboard for new 
developments. 

A recent HMI report noted that: "Almost half the colleges in England have... 
Mathematics Workshops, where students may study on their own or under guidance with 
learning materials which suit their individual needs ... a balance between structured 
sessions and workshops is a particularly good approach to teaching mathematics at this 
(FHE) level". 

However, it continued: "The use of IT in mathematics teaching is still not widespread- 
there is scope for using the microcomputer more extensively and imaginatively in 
mathematics ... IT equipment must be made more accessible to students". 

The facilities of the Mathematics Workshop at Suffolk College were outlined, where an 
open plan area contained a 24 station NOVELL pc network as well as a BBC 
microcomputer ECONET network. 

Two examples of currently available commercially produced mathematics 
leaching/learning software packages were available for demonstration on BBC 
microcomputer. 



264 



STUDENT-CENTRED MATHEMATICS 



257 



These packages ("Revise Maths" by A VP Computing, and "Mathspad" by Longman 
MicroSoftware) had proved particularly successful at Suffolk College. Participants in the 
workshop could evaluate this software, and in doing so become aware of; examples of 
good practise; areas with the potential for further development. 

SHARING OF PARTICIPANTS* EXPERIENCE 
Group Activity 1 

The participating groups were asked to share examples of good and useful software that 
they have used.Why did they find it so? Does it have any drawbacks or limitations? 

Group Activity 2 

To establish the criteria for good software packages to aid the teaching/learning of 
mathematics, the Group were invited to answer the question "effective teaching and 
learning software should ensure that 
The following criteria were established by the group; 

• use of the software should be transparent to the user 

• the user should attain a sense of achievement by using the software 

• the software should make full use of visual display capability (colour, 
graphics etc) 

• the software should be "tried and tested" 

• users should be encouraged by the software to consider options and make choices 
at regular stages 

• the user should be allowed to respond freely (ie not have to make a choice from 
pre-defined responses) 

• the software should offer different "difficulty levels" at the outset 

• the software should include a "help" option ("panic button") to give the user 
clarification and advice 

• the objectives of the software solution should be pre-determined 

• the software should not cause too many irritating bleeping noises 

• the software should allow precise definitions of the "task" by the teacher 

• the user should be given "diagnostic" response by the software 

• the user should be given "recap" facilities (return to start; tc other pages) 

• the software should give summary and detailed feedback to the user 

• the user should be given "mapping facilities" by the software 

• the software should comment on what the hardware is doing. 

INTRODUCTION TO AUTHORING SOFTWARE 

There is a need for effective methods of developing material for computer based training 
systems. 

In particular, there is an increasing need for authoring to; 

• promote Open Learning, which is widely acknowledged as a method of delivery 
which provides flexibility and improved utilisation of staff and student time 



Aw ^ I J 



258 



QUALITY IN TECHNOLOGY 



♦ improve delivery methodology by incorporating technology and multi-media to 
enhance the presentation 

♦ allow different areas to be customised to specific needs, so allowing an 
organisation or individual to fill in the gaps that more traditional methods cannot 
provide. 

Authoring is now viable because of: 

♦ the technological developments in PC technology 

♦ the development of multi-media technology - sound, picture, video 

♦ the reduction in cost of hardware, so making authoring economically viable 

♦ and the need for improvements in the quality of teaching and training. 

What makes for good authoring software? 

♦ ease of development of the teaching material - the interface of the authoring 
system must be highly interactive, and require non -programming skills 

♦ a high level of interaction between user and system including monitoring and 
assessment of the learner 

♦ interface to multi-media technology - the ability of the systems to use interactive 
laser disk, CD-Rom, sound and scanned pictures if required. 

DEMONSTRATION OF THE SOFTWARE SOLUTIONS 

The participants were able to try a number of simple authoring packages. In particular, 
the overhead displays used throughout the session were generated using IBM's 
LINKWAY, an authoring solution which is particularly appropriate for presentation 
purposes. 

A number of illustrations of mathematics solutions using LINKWAY were available, 
which demonstrated the ease of use and effectiveness of simple features such as "pop-up 
screens" and "go-buttons". More advanced features were discussed after the 
demonstrations and brought the session to a successful close. 



oqan Page 
120 PentonviiieRoad 
London N1 9JN 



Published m the USA by 
Nichols/GP Publishing Company 
PO Box 6036, East Brunswick, New Jersey 08816 



ISBN (UK) 07494 08979 
ISBN (US)Q-89397-385-8 




25'