Full text of "ERIC ED358830: Quality in Education and Training. Aspects of Educational and Training Technology. Volume XXVI."

See other formats

DOCUMENT RESUME -
ED 358 830 IR 016 111

AUTHOR
TITLE

INSTITUTION

REPORT NO
PUB DATE
NOTE

AVAILABLE FROM

PUB TYPE

Shaw, Malcolm, Ed. ; Roper, Eric, Ed.
Quality in Education and Training* Aspects of
Educational and Training Technology. Volume XXVI.
Association for Educational and Training Technology,
London (England) ♦

ISBN-0-7494-0897-9; ISBN-0-89397-385-8
92

268p.; Proceedings of the Association for Educational
and Training Technology's International Conference
(1992) .

Kogan Page Limited, 120 Pentonville Road, London Nl
9JN, England; Nichols Publishing, P.O. Box 6036, East
Brunswick, NJ 08816.

Books (010) — Collected Works - Conference
Proceedings (021) — Reports - Evaluative/Feasibility
(142)

EDRS PRICE
DESCRIPTORS

IDENTIFIERS

MF01/PC11 Plus Postage.

Computer Software Development; Curriculum
Devel opment ; Educat i onal Assessment ; *Educat i onal
Quality; *Educat i onal Technology; Elementary
Secondary Education; Instructional Design;
Interactive Video; Organizational Change ;
Postsecondary Education; ^Quality Control; Teaching
Methods ; ^Technological Advancement ; ^Training
Association for Educational and Training Tech;
Stakeholder Evaluation; Total Quality Management

ABSTRACT

The 38 conference papers in this volume were chosen
to exemplify different definitions of, and approaches to, quality, as
they are applied in a wide range of educational and training
contexts. The papers are: "Designing Organisations That Learn" (D. J.
Dicks); "Quality Assurance in a European Context" (D. Alexander, J.
Morgan); " f What's in It for Me?'" (C. Hart, M. Shoolbred); "Quality
and the Academic Administrator" (C. D. Payne); "Total Quality
Management in an Education and Training Context" (B. Ellis);
"Performance Management in a Competence Framework" (E. Sauve) ;
"Quality in Further Education: An Unchanging Agenda" (D.
Shepherds on) ; "Defining Quality in Higher Education" (A. Burrows, L.
Harvey); "Total Quality Implementation — Cultural Issues and Training"
(B. Hurley); "Developing the European Quality Model" (J. S. Oakland,
L. J. Porter); 'Strategic Quality Management" (R. J. Newton);
"Profits and Pitfalls in Establishing a Qualytechnic" (J. Heap, H.
Solomon); "Student Satisfaction and Perceptions of Quality" (P. M.
Mazelan and others); "Experiences in the Design and Conduct of
Enterprise Audits" (D. Edgar); "Monitoring the Quality of Quality
Control Systems" (H. I. Ellington, G. T. N. Ross); "Improving the
Quality of a National Curriculum" (I. Musallam, M. Brophy, M.
Schilling); "Developing Quality in Education" (R. J. D. Rutherford);
"Quality in Course Design" (E. Roper); "Quality Horses for Quality
Courses: Matching Students with Courses in Music" (L. Gibbs) ;
"Recognising Quality in Engineering Education" (D. C. Hughes, R. G.
S. Matthew); "Exploring Learners 1 Perceptions of Quality" (D. Miller,
P. Funnell); "Concept Mapping, Post-Questioning and Feedback in
Distance Education" (R. M. Bernard, S. Naidu, K. Lundgre-Cayrol^ ;
"Comparing Chalk and Cheese — Quality in Assessing Worx-Based

Learning" (C. Bucklow) ; "Using Learning Contracts To Enhance the
Quality of Work-Based Learning" (I. S. Marshall, M. L. Mill);
"Quality of Assessment" (P. Race); "BS5750 for Assessment" (P. Race);
"Eating Frogs and Bridging Gaps — Post-Warnock Conditions for Teaching
Quality" (C. Colling); "Teaching Quality" (D. Jones, J* Hanson);
"Upward Appraisal and Its Implications for Higher Education" (G.
McElwee and others); "A Total Quality Approach to Managing CBT
Development" (S. G. Shaw, D. R. Shaw); "Towards Quality Management in
Training Design" (M. Williams, G. Carr) ; "Achieving Quality in
Networking Interactive Video" (P. Willis, J. Early); "Quality in
Modernising Educational Technology" (D. Hawkridge) ; "Trends and
Issues in Educational Technology" (D. P, Ely); "Training Quality and
New Technology" (A. R. Bartolome) ; "Connecting Lectures and
Laboratory Learning with CAI" (Y. Araki) ; "Graphical Routes to
Quality Courseware" (P. Barker, C. Lamont) ; and "Enhancing the
Quality of Student-Centered Mathematics" (D. Bowers, R. Burrell) .
Most of the papers include references. (MES)

it it jfc it it it it it it it it it it it it it it it it it it it it it it it A A Vc it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it

i: Reproductions supplied by EDRS are the best that can be made *
* from the original document. *

it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it

Assets ®$ EdidS&ifonal and
Training Technology XXVi

^ rr^ .^i j' rro " r ^ ? r^? ^Ti
Li L vlh^J b U u L b u ii Li ^vj

Edited by

Malcolm Shaw and Eric flop or

U.S. DEPARTMENT Of EOUCATtON

Office ot Educational At March and Improvement

EDUCATIONAL RESOURCES INFORMATION
CENTER (ERIC)

C This document has been reproduced at
'«»ceived from the person or organization
originating it

C Minor changes have oeen made to improve

'eoroduction Quality

e Pomts ot view o< opinions stated <n this docu--
-^ent do not necessarily represent official

OERl position or policy

"PERMISSION TO REPRODUCE THIS
MATERIAL HAS BEEN GRANTED BY

N.R. Winterburn

ERJC

TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER {ERIC)."

Aspects of Educational and
Training Technology

VolumeXXW

Aspects of Educational and
Training Technology

VolumeXXW

Quality in Education and Training

Edited for the Association for Educational and
Training Technology by

Malcolm Shaw and Eric Roper

TNICHOLS/GP
[PUBLISHING'

Kogan Page, London

Nichols Publishing Company, New Jersey

First published in 1993

Apart from am fair dealing for the purposes of research or private study, or criticism or
review, as permitted under the Copyright, Designs and Patents Act. 1988, this
publication may onlv be reproduced /stored or transmitted, in any form or by any
means, with the prior permission in writing of the publishers, or in the case of
reprographic reproduction in accordance with the terms of licences issued by the
Copyright Licensing Agency. Enquiries concerning reproduction outside those terms
should be sent to the publishers at the undermentioned address :

Kogan Page Limited
120 Pentonville Road
London N19JN

t The Association for Educational and Training 'technology. 1993

British Library Cataloguing in Publication Data

A CI P record for this book is available from the British Library
ISBNO 7494 0807 9

Published in the United States of America by Nichols Publishing,
P.O. Boxf)Q36. East Brunswick, New Jersey 08816

A C I P record for this book is available from the Library of Congress

ISBN 0-89397-385-8

Printed and hound in Great Britain by
Biddies Ltd, Guildford and King's Lynn

Contents

List of contributors 1X
Editorial x
Section 1: Perspectives on Quality in Organisations 1

11 Strategic quality management: if I pull it up the flagpole will anyone
salute?

Robert James Newton

12 Profits and pitfalls in establishing a qualytechnic
John Heap and Hazel Solomon

3
10

1 Designing organisations that leam
Dennis J Dicks

2 Quality assurance in a European context
David Alexander and Jim Morgan

3 "What's in it for me?" Organisational culture, rewards and quality 16
Christopher Hart and Michael Shoolbred

4 Quality and the academic administrator
Catherine D Payne

5 Total quality management in an education and training context 29
Barbara Ellis

6 Performance management in a competence framework 35
Elaine Sauve

7 Quality in further education: an unchanging agenda 4 0
David Shepherdson

8 Defining quality in higher education - the stakeholder approach 44
Alison Burrows and Lee Harvey

Section 2: Implementing Quality in Organisations 51

9 Total quality implementation - cultural issues and training 53
Brian Hurley

10 Developing the European quality model 59
John S Oakland and Leslie J Porter

13 Student satisfaction and perceptions of quality 76
P M Mazelan, D M Green, C R Brannigan and P F Tormey

14 Experiences in the design and conduct of enterprise audits 82
Douglas Edgar

15 Monitoring the quality of quality control systems
Henry I Ellington and Gavin T N Ross

Section 3: Quality in Courses 93

16 Improving the quality of a national curriculum: the role of textbooks 95
/ Musallam, M Brophy and M Schilling

17 Developing quality in education: a way forward 102
RJD Rutherford

18 Quality in course design: empowering students through course processes 107
and structures in a professional development scheme

Eric Roper

19 Quality horses for quality courses: matching students with courses in music 1 14
Louise Gibbs

20 Recognising quality in engineering education 120
D C Hughes and RG S Matthew

Section 4: Quality in Learning and Assessment 123

21 Exploring learners' perceptions of quality 125
Dave Miiller and Peter Funnell

22 Concept mapping, post questioning and feedback in distance education 132
Robert M Bernard, Som Naidu and Karin Lundgren-Cayrol

23 Comparing chalk and cheese - quality in assessing work-based learning 140
Caroline Bucklow

24 Using learning contracts to enhance the quality of work-based learning 144
Iain S Marshall and Margaret L Mill

25 Quality of assessment 149
Phil Race

26 BS5750 for assessment 162
Phil Race

Section 5: Quality in Teaching and Training 173

27 Eating frogs and bridging gaps - post-Wamock conditions for teaching quality 175
Clive Colling

28 Teaching quality - "I know it when I see it!" 180
David Jones and Janet Hanson

29 Upward appraisal and its implications for higher education 186
Gerard McElwee, Tom Redman, David Edson and Adrian Evans

30 A total quality approach to managing CBT development 192
Steven G Shaw and David R Shaw

31 Towards quality management in training design 214
Meurig Williams and Graham Carr

32 Achieving quality in networked interactive video 222
Peter Willis and John Eary

Section 6: Quality in Technology 229

33 Quality in modernising educational technology 231
David Hawkridge

34 Trends and issues in educational technology 237
Donald P Ely

35 Training quality and new technology 241
Antonio R Bartolome

36 Connecting lectures and laboratory learning with CAI 246
Yoshikazu Araki

37 Graphical routes to quality courseware 250
Philip Barker and Charles Lament

38 Enhancing the quality of student-centred mathematics 255
David Bowers and Rod Burrell

Index of contributors

Alexander D 10

McElwee G 186

Araki Y 246

Marshall I S 144

Matthew R G S 120

Barker P 250

Maze lan P M 76

Bartolome A R 241

Mill M L 144

Bernard R M 132

Morgan J 10

Bowers D 255

Muller D 125

Brannigan C R 76

Musallam I 95

Brophy M 95

Bucklow C 140

Naidu S 132

Burrell R 255

Newton R J 68

Rtiirnw^ A 44

Oakland J S 59

Carr G 214

Colline C 175

Payne C D 22

Porter L J 59

Dicks D J 3

Race P 149, 162

Eary J 222

Redman T 186

Edgar D 82

Roper E 107

Edson D 186

Ross G T N 88

Ellington H I 88

Rutherford R J D 102

Ellis B 29

Ely D P 237

Sauve E 35

Evans A 186

O _t_ 'II' „ \ m AC

Schilling M vd

Shaw D R 192

Funnell P 125

Shaw S G 192

Shepherdson D 40

Gibbs L 114

Shoolbred M 16

Green D M 76

Solomon H 72

Hanson J 180

Tormey P F 76

Hart C 16

Harvey L 44

Williams M 214

Hawkridge D 231

Willis P 222

Heap J 72

Hughes D C 120

Hurley B 53

Jones D 180

Lamont C 250

Lundgren-Cayrol K 132

Editorial

This volume constitutes the Proceedings of the Association for Educational and Training
Technology's 1992 International Conference, which explored "Quality in education and
training". The theme is topical in view of the growing Government initiated climate of
accountability, performance indicators and monitoring of quality, and in the context of
the developing activities of sector-specific agencies such as the Higher Education Fu *ding
Council through its quality audit activities and its quality assessment initiatives.

One of the origins of the quality "movement" was the attempt to re-build Japanese
industry after World War II. Initially many of the ideas were American, but the Japanese
subsequently developed their own approaches to quality. Nowadays the movement is
worldwide, and the approaches increasingly differentiated and sophisticated. However,
the extent to which ideas and procedures developed in industrial and business applications
and in different cultures can be applied to specific education and training contexts is
problematic. To take some examples, is it possible and/or desirable in education and
training to design "error-free" processes, to get it "right first time", to isolate variables
and represent them statistically, to conform to requirements? Isn't education and training
a more humane process, in which both teacher and learner are reaching out into what is,
for each of them, the unknown? In doing so, aren't they extending the limits of their
capabilities and surprising themselves - a process in which error or failure, if not feared
or disguised (or eliminated!), can generate powerful insights and contribute to learning?

Whilst acknowledging its problematic nature, much is being done to develop ways of
enhancing and assuring quality in education and training. The papers in this volume have
been chosen to exemplify different definitions of and approaches to quality, as they are
applied in a wide range of educational and training contexts, and to explore some of the
issues raised. Although the total quality management perspective is strongly represented,
so too are others such as British Standard 5750, fitness for purpose, identifying and
meeting the needs of different stakeholders, and empowering the customer or client. Also
included are examples of applications of some of these perspectives outside education and
training - for example, in commercial CBT development (Chapter 30) - which may (or
may not) inspire us to adopt similar procedures in education and training, but which
exemplify the methodology. Reflecting the international nature of the conference, several
of the contributions are from outside the UK.

The eclecticism of the contributions is inevitable. They have been written by
practitioners wrestling with the problems of identifying and delivering quality in specific
contexts, sometimes drawing explicitly on theoretical perspectives or attempting to adapt
industrial applications of quality to educational settings, but often developing pragmatic
solutions in response to institutional and external demands for quality assurance or in
addressing day-to-day problems. The contributors demonstrate first-hand knowledge,
expertise, originality and familiarity with relevant literature. In presenting their work in
ways which vividly capture the richness of their experiences - the problems and struggles
as well as the solutions - they help to demystify the search for quality and allow others
to make connections with their own experiences and concerns.

The organisation of these Proceedings reflects that of the conference. The organisers and
editors struggled with alternative arrangements, but settled for these rather arbitrary
divisions. We are aware that many papers could have been included in alternative
sections. This is particularly the case with those illustrating applications of technology.
We apologise to those contributors who feel that their work has been misplaced! Each

of the sections has an introduction which provides an overview of the papers included.

The timeliness of the conference theme was demonstrated by the number of contributions
offered (over 70 were delivered as papers, seminars and workshops, together with poster
and exhibition sessions). Such was the response that the editors have been unable to
include in these Proceedings all of the conference contributions. They wish to
acknowledge here the part played by all conference contributors in stimulating and
engaging in what was a high quality, detailed and constructive debate. All conference
participants were able to find sessions from which they were able to take new knowledge
c : perspectives, together with practical ideas. The keynote addresses, delivered by Geoff
Stanton of the Further Education Unit, John Oakland, Brian Hurley and Dave Miiller and
Peter Funnell, all provided ideas and stimulated thinking which participants related to and
developed in other conference sessions. The editors* thanks are extended to these
keynote speakers for providing the "warp" in the "weft" of the conference!

Organising a major Conference, like AETT 1992, requires the help and commitment of
a considerable number of people. Whilst it would be absolutely impossible here to name
all those who contributed to AETT 1992. it is certainly an appropriate place to single out
the following friends and groups for a special mention in view of the extensive and vital
contribution that they made to the success of our venture ...

Members of the Planning Group: Graham Cheetham. Katherine Hayes, Bob Matthew,
David Pratt and Hazel Solomon.

Our sponsors: Cartwright Brice Office Supplies (Leeds), Kodak Photocopiers Division,
Nestle Rowntree Division (York), York City Council Economic Development Unit, York
Wines of Sheriff Hutton, Leeds Metropolitan University (Computing Services Unit,
Media Services Unit, Enterprise in HE Unit, Communications and External Relations).

AETT stalwarts, but in particular: Henry Ellington. Fred Percival and Roy Winterburn.

Our Conference Team: students (Philippa Allott, Sarah Howson, Caroline D* Attorre and
Melissa Labram) and technician - Bruce Knights.

Special thanks are also due to our colleague, Stuart Rawnsley, for his unflagging and
always enthusiastic achievements not least with regard to the social programme and
sponsorship.

We are sure that all would wish to join us, once more, in saying a re Ay big thank you
to the one person on our Team who, above all. seemed able to remain absolutely calm,
whilst the rest of us were quietly panicking, and who has done much to organise your
editors through the task of producing these Proceedings * Jane Thomas, the conference
secretary.

Finally our thanks go to all the contributors and delegates, without whose support we
could not have had such a memorable, enjoyable and QUALITY conference!

Eric Roper
Malcolm Shaw

Leeds Metropolitan University

Vjl

Section 1: Perspectives on Quality in
Organisations

Contributors here have focussed on a number of theoretical issues and matters of
principle that are suggested, in many cases, as necessary conditions for the application
of quality approaches in organisations.

The contexts vary widely, beginning with the global concerns outlined in "Designing
organisations that learn" by Dennis Dicks, who questions the validity of trying to transfer
essentially Japanese notions of quality into a British context and culture. This cultural
theme is echoed in "Quality assurance in a European context" by David Alexander and
Jim Morgan, where the context is Europe and the paradoxes and dilemmas inherent in
the transfer of some quality indicators across higher education institutions in its Member
States.

In "What's in it for me?", Chris Hart and Michael Shoolbred focus on the individual
organisation and the cultural features that might inhibit or nurture the growth of quality.
Similarly, the focus is firmly located in the organisation in Catherine Payne's
consideration of "Quality and the academic administrator", where factors influencing the
implementation of TQM in higher education are viewed from the perspective of the
educational administrator. Barbara Ellis also describes the necessary conditions for
application of TQM in higher education and uses a case study approach by means of
illustration.

The role of effective management training in the push for TQM is the theme explored by
Elaine Sauve in "Performance management in a competence framework" and this is
linked to some contemporary training initiatives. The importance of training is also
echoed by David Shepherdson, who goes on to describe a large scale training and
development initiative across a Local Education Authority.

Finally, ihe way in which the views on quality of the different stakeholders in education
are to be compared and contrasted is the subject of work described by Alison Burrows
and Lee Harvey in their multi-perspective approach - "Defining quality in higher
education: the stakeholder approach".

1 2

1 Designing organisations that learn

Dennis J Dicks, Concordia University, Montreal, Canada
INTRODUCTION

There is much debate in the U.K. and in North America about the way we organise work.
A central issue is the impact of Japanese practices: can they be transplanted? Will they
converge towards our methods or not?

This debate arises in the context of a broader argument about the nature of industrial
production. Some theorists suggest that the Japanese system heralds a break with the
"Fordist" model of mass production: work teams, job rotation and innovation replacing
centralised control, task fragmentation, and regimentation or small-firm networks
replacing large corporations (Piore and Sabel 1984). Others see it as a transitory
phenomenon (Gertler 1988) or even a refined form of Fordism (Dohse et al 1986).

In any case, the current industrial imperative demands flexible response to changes in
the business environment, innovation, short design to production cycles, tight scheduling,
lean operation and quality control. The Japanese have demonstrated that they can achieve
these things; and they have done so by creating organisations that learn, organisations in
which comparative advantage is based upon the development and deployment of workers'
skills and intelligence (see for example Dore et al 1989; Koike 1988).

How is this done? The evidence indicates that the Japanese approach is systemic, in
so far as it integrates values and practices developed in several institutions: the family,
school, university, industry and government.

DESIGNING A LEARNING ORGANISATION IN JAPAN
Human Capital: Investing in People

Much has been written about the superiority of Japanese management techniques.
However, one could argue that managing in Japan is easier than in North America
because human resources are belter suited to the tasks at hand. This is because skills
development more closely follows the idealized "human capital" model; in short, people
are given the information, incentives, and tools to invest in themselves with the promise
of future payoffs. These investments are exploited, for mutual benefit, in the ways these
skills are developed and deployed in the workplace. Japanese firms thus display what has
been called "organisational capital".

Ln the Japanese context, there arc five key elements contributing to the development of
human and organizational capital: formal education, recruitment, deployment,
remuneration, information transfer.

Formal Education

We have all heard about the rigours of the educational system in Japan. Students spend
more hours per day, more days per week, and more weeks per year at school. On
average, they are more dedicated than their North American peers to learning. One
reason for this is the extreme competitiveness of a large society crowded onto a small
island, with virtually no opportunity to move horizontally to other countries. Another
reason is the complexities of the language, which require enormous concentration and
effort to reach the compulsory level of mastery at the end of high school. After that

PERSPECTIVES ON QUALITY IN ORGANISATIONS

point, with entry to a good technical college or university, demands on the student in fact
become less rigorous. Students (and parents) are willing to invest their time and money
in this process because they know it generally pays off in the trappings of success -
money, status, the means to raise a family.

Recruitment

This effort would be lost if the skills produced by the educational system were not
recruited into productive work. Graduates are carefully matched to employment
opportunities in a highly organised recruitment system working on a national scale.
TTiose from technical colleges flow into the openings for skilled labour, those from
universities into the openings for office workers. As part of the effort to have employees
identify with the firm rather than with economic classes, distinctions are minimized, so
that the technical jobs which we would characterize as "blue collar" have relatively high
status in Japan.

Graduates are not recruited for specific positions, but for their potential to contribute
to the organization in the long term. Accordingly, the level of recruitment is not tightly
bound to economic conditions; and the criteria for selection focus more on attitude, basic
ability and willingness to learn rather than precocious performance.

Deployment

Deployment has three key facets: probation, job rotation/working group, job-related
training.

Graduates fresh from school enter a period of probation, lasting perhaps three years.
This entails a program of activities intended to develop skills and build identification with
the group: "boot camps", orientations, and rotating appointments to several relevant parts
of the organization. By the end of this period, the new employee and the firm will have
assessed the potential for a fruitful relationship and substantial portion of the entrants will
have left or been sent on their way by this point.

This pattern of developing skills continues throughout the employee's life in the
organization. Employees hired from other firms follow a similar pattern. Training may
be organised within the firm, or the employee may be strongly encouraged to pursue
courses outside working hours. Job rotation and working in groups not only develop a
broad range of operational skills but also instill a clearer understanding of the firm's
activities.

Remuneration

One of the glues holding these practices together is the system of remuneration, which
blends a number of factors into the payment schedule for each individual: seniority,
individual and group performance, and special needs such as family situation,
geographical location, commuting distance. Each firm has its own way of weighting
these factors, but the basic principles of the so-called "seniority-merit" system are the
same: salary profiles follow a roughly s-shaped function (the familiar learning curve) in
a sense underpaying the recruit during the probationary period, and overpaying during the
period when the demands of raising a family are greatest. The system thus rewards
loyalty as well as performance. It also conforms to "human capital theory" (Becker
1964) in so far as payments are lower when the employee is less productive and is in fact
benefitting from the skills being instilled at the organisation's expense. Japanese firms

DESIGNING ORGANISATIONS THAT LEARN

can thus invest in "general 11 skills because the prospect of higher payments later on serves
as the incentive for the employee to remain with the firm.

Information Transfer

If remuneration patterns are the glue that holds the system together, the lubricant that
keeps it going is information transfer. One of the most significant features of the
Japanese form of organisation is the decentralisation of control over information.
Essentially, responsibility for dissemination of job-related information is pushed down to
the lowest level possible. The work -team leader, and even team members, are expected
to share their knowledge with co-workers. In effect, the day-to-day training function
becomes the job of ordinary workers.

Sharing information is one of the most important ways of raising the status of people
performing technical jobs. In Japan, "blue collar" work takes on many of the decisional
and innovative aspects of "white collar" work.

Organisational Capital

Together, the measures taken in Japan to develop human capital also contribute to an
analogous form of investment, "organisational capital". In this case, investment is in
organisational form, such that the organisation develops practices which enable it to adapt
more easily to changing conditions (Prescott and Visscher 1980). Firms create
organisational capital in three ways: by optimising the match between people and jobs,
by optimising the match between co-workers, by optimising feedback so that the
organisation can learn from its own experience.

As outlined above, the Japanese system tends to optimise the match between people and
jobs through careful recruitment, job rotation, and training. Secondly, it tends to optimise
the match between co-workers by organising them in working groups, where peer
processes are more important than external control; flows of information are decentralised
to optimize this form of organisation. Thirdly, peer processes, the remuneration system
and other factors favouring a focus on long-term goals facilitate the development,
adoption and rapid diffusion of "best available practice" through the firm. All these
practices encourage flexibility and adaptiveness.

A knowledgeable reader would point out that the practices which I have attributed to
the Japanese model are those of the major firms, those capable of sustaining the ideal of
"lifetime employment". Different estimates would suggest th 4 the model might cover
from 30% to 70% of the Japanese workforce, almost exclusively males. However, the
point is that the practices I have described do in fact serve as the ideal which informs the
decisions of those who invest in human capital: students, parents, governments and
employers. In an earlier paper (Dicks 1986), I suggested that this provides a fourth way
of developing organisational capital: when most of the firms competing for human
resources optimise in the ways listed above, then organisational capital is created on a
societal or national scale.

Their overall goal is to minimise risk by sharing it over all employees (Nagatani 1991).
Information critical to the enterprise's success is readily developed, quickly diffused and
turned into comparative advantage. In this sense, Japanese organisations not only
facilitate the learning of individual workers, but can be said to leam collectively from
past experience.

Though the details vary from firm to firm, and the pattern becomes fainter in smaller
firms, the Japanese approach to building organisations that learn is rational and systemic:

PERSPECTIVES ON QUALITY IN ORGANISATIONS

all the parts - formal education, recruitment, deployment, remuneration and information
flows - fit together and are even mutually supportive. Given the interdependence of these
parts, the system cannot be transferred easily to different organisational contexts, such as
we find in the Anglo-Saxon parts of the world.

THE ANGLO-SAXON EMPLOYMENT MODEL

In contrast, "Fordist" methods of organising work seem to be increasingly counter-
productive. I lump them together as the Anglo-Saxon model, for they have arisen in and
dominated English-speaking cultures, at least those on the Atlantic rim. The model is
based on free- market concents such as McMillan's "Type A" (McMillan 1979). That is
to say, individuals are expected to develop their skills and sell them to the highest bidder.
Emphasis falls on quick profit (minimum wage for serving fast food) rather than long
term planning (finishing secondary school). Skills are seen to be the property of
individuals, rather than the attributes of a working group. Similarly, organisations tend
to be viewed from a structural rather than a dynamic perspective. Their overall goal is
profit-maximisation, and so they tend to cost-reduction by minimising the quantity and
quality of labour, thus pushing control cf information and profit-taking ever upward in
the hierarchy. The free-market model is generally antithetical to the human capital model
discussed above.

The free-market model probably suffices when production makes minimal demands for
skills (eg the factory system), when labour is plentiful, when other resource costs are low,
when there is little competition for markets. It served North America well during most
of this century, particularly after the Second World War. However, there is every
indication that it is outmoded: we have high unemployment and skill shortages at the
same time; state of the art has moved elsewhere; education and training are expensive but
undervalued. Further, the basic flaws in the free-market model are being exposed: skill
prices do not reflect uniformly the cost of their production, and do not move down as fast
they move up. Even more fundamentally, we are learning that individual expertise is not
necessarily the best unit for constructing a competitive organisation.

DESIGNING A LEARNING ORGANISATION IN THE ANGLO-
SAXON CONTEXT

Obviously, organisations in North America or the U.K. cannot create a new educational
infrastructure, nor change all the rules governing recruitment, deployment and
remuneration in an environment shared with competitors, labour unions and governments.
Consequently, their approach to creating organisations that learn must be very different
from the highly orchestrated one which has dominated Japan since 1945. For the most
part, they must work within the boundaries imposed by our current system of developing
and deploying human resources. Several different trends arc detectable.

One is to literally transplant Japanese-type operations. Most of the transplants seem to
have successfully transferred many Japanese practices eg. the prize-winning Toyota plant
in Cambridge (Romain 1991). But these cases have several features which suggest that
their success may not be easily generalised: they are largely restricted to assembly
operations, in the auto industry; these operations tend to be managed from Japan, by
Japanese, as branches of large corporations; they are put together by very careful
selection of workers, on the basis of education, attitude and union-affiliation. They have
succeeded because they have minimised the impact of our established practices,
effectively side-stepping the mainstream of our organisational culture. However, they

DESIGNING ORGANISATIONS THAT LEARN

may have a broader impact by increasing competition, and demonstrating that alternative
methods of organising work are not only possible, but sensible.

Another trend has been to change the nature of the organisation indirectly, by examining
all jobs, redefining their informational requirements, and then providing more or less
traditional training, albeit on a massive scale. During the 1980*5, Motorola changed from
an organisation that "...hired people to perform set tasks and didn't ask them to do a lot
of thinking" to one which emphasises basic skills, problem solving in teams, and a focus
on quality (Wiggenhorn 1990). This was accomplished by getting everyone involved in
training programs, from elementary to university levels, working with educators to
overcome shortcomings in the school environment (Wiggenhorn 1990). In short,
Motorola became a learning organisation by breaking the monopoly of managers on job-
related information and the authority to act on it. This approach was indirect in the sense
that it created a corporate culture conducive to learning on the part of individual workers.
Success is therefore measured in terms of their performance. Better performance of the
firm as a whole is of course desired, but might be said to be an emergent outcome.

On the other hand, a more direct approach is to change the organisation by renovating
specific structures and procedures. In this case, success is measured by the collective
learning of whole groups of people. One example of the direct approach is an outgrowth
of the vast literature on "learning" or "experience curves" in production management,
where it has long been known that manufacturing units improve the speed and quality of
their work over time. Only recently, however, has anyone attempted to explore the
factors determining whether and how learning actually occurs in these circumstances
(Adler 1990).

Adler conducted a case-study of the evolution of the design and manufacture of a high-
tech product in a multi-national firm - in effect, an application of "concurrent
engineering". He examined how changes in the flows of critical information among the
firm's functional units affected the quality of processes and product. Broadly speaking,
the firm had to replace some of its formal rules, which defined its structure in hierarchical
terms, with more informal guidelines prescribing a more timely flow of information. This
was accomplished by changing patterns of communication: by creating new sub-units for
liaison, by creating "flying squads" to carry vital information to branch plants, and by
job- rotation on a large scale.

In Adler's terms, these changes were based upon a clearer perception of the degree to
which these units performed differentiated functions, and the degree to which they were
interdependent. For example, the firm created "centres of competence" to recognise and
reinforce creativity at branch plants. These centres had a lot of autonomy, with rich
internal communications, but little contact with other branches. On the other hand, the
firm recognised the interdependence of the design and manufacture phases of production
by rotating engineers between these two functions.

Adler* s work is significant in two ways. First, it provides some empirical evidence of
"shared learning", of deliberately created organisational capital. Secondly, it opens a new
realm of research on organisational design, adding a higher level of analysis to the study
of how training and other interventions improve an organisation's performance.

IMPLICATIONS FOR QUALITY IN THE ORGANISATION

In a broad sense, the key features in the cases described by Wiggenhorn and Adler
replicate those in the Japanese approach: appropriate skills, deployed across traditional
job boundaries, linked by optimal communication. We might call this "just-in-time skills
delivery".

PERSPECTIVES ON QUALITY IN ORGANISATIONS

Can these approaches be adopted for large-scale changes in the way we organise work?
In the Motorola case, it took ten years and an annual investment of the order of $50
million to change corporate culture, work with schools to improve labour input, and
develop a systemic program of job-related training. These issues concern some
newsworthy companies, but one wonders whether the majority will respond. In Canada
for example, private sector investment in training is extremely low by international
standards, and the results are generally seen as inadequate (Quarterly Labour Market and
Productivity Review 1990).

In Adler's case, the changes in corporate structure are piecemeal, not necessarily rooted
in cultural change, and so risk an abrupt end. As Motorola has learned, significant and
sustainable change requires pervasive changes in attitude and behaviour. This must start
at the top of the organisation and cascade down, so that managers understand and support
change. In one major Canadian company which invests heavily in human resource
development, upper managers initiated a program of decentralising training to teams, but
suddenly changed direction when they felt control slipping away. Such are the risks
which must be taken if our work culture is to change.

My experience with joint- ventures where Japanese and Anglo-Saxon work cultures come
into contact indicates that the most difficult changes involve re-examining the way in
which we organise the flow of information and responsibility. To generalise, sharing
knowledge and responsibility is first-nature in organisations built upon group processes,
team-work, job-rotation. In fact, as indicated earlier, for the Japanese the point in
organising teams is to minimise the risks of labour shortage and inflexibility. On the
other hand, organisations founded upon the principle of expertise, that knowledge is a
private, compensatable property of the individual, tend to create what have been called
"chimneys", knowledge hierarchies wherein each level is privy to only that amount of
knowledge required for a task prescribed by the chimney. The other side of this is that
responsibility for the quality of performance is similarly limited: in short, everyone covers
their own tail.

These characteristics of the way we organize work thus mitigate against the very factors
upon which quality of performance can be seen to depend: common experience,
decentralised skill development, timely sharing of knowledge. If we are to begin to move
towards the greater sharing of knowledge that the current interest in team -building
implies, then significant changes in the way we organise work will be required. One is
to push the knowledge and responsibility required to meet corporate goals down to the
shop-floor. Another is to make sharing information and informal training everyone* s
responsibility, not just that of the official training department. Japanese successes with
a systemic approach suggest that these changes must lead to many others.

References

Adler P S (1990) Shared Learning Management in Science 36(8) August

Becker G (1964) Human Capital: theoretical and empirical analysis with
special reference to education Nat. Bur. Econ. Res. New York

Dicks D J (1986) From Ships to Chips: manpower policy for industrial change in Japan
Aspects of Educational Technology XIX Kogan Page London

Dohse K, Jurgens U and Malsch T (1986) From Fordism to Toyotism? The social
organization of the labor force in the Japanese automobile industry in Politics and
Society 14 pp 45-66

DESIGNING ORGANISATIONS THAr LEARN 9

Dore R, Bounine-Cabale J and Tapiola K (1989) Japan at Work: markets, management
and flexibility OECD Paris

Florida R and Kenney M (1991) Transplanted Organize r. the transfer of Japanese
industrial organization to the US in American Sociological Review 56 pp 381-398

Gertler M (1988) The Limits to Flexibility: comments on the post-fordist vision of
production and its geography in Trans. Inst. British Geog. 13 pp 419-32

Koike K (1988) Understanding Industrial Relations in Myiern Japan MacMillan
London

McMillan C (1979) Human Resource Policies, Labour Markets and Unions: Canada-
Japan comparisons in Hay K A J (ed) Canadian Perspectives on Economic Relations
with Japan IRPP Montreal

Nagatani K (1991) Panel Participant Japanese Studies Association of Canada
Conference Winnipeg October

Nakamura Atsushi (1975) Intrafirm Wage Differentials reprinted in Nishikawa S (ed)
(1980) The Labour Market in Japan University of Tokyo Tokyo

Piore M and Sabel C (1984) The Second Industrial Divide Basic New York

Prescott E C and Visscher M (1980) Organizational Capital in J Political Economy 88
(3)

Quarterly Labour Market and Productivity Review (1990) The Linkage between
Education and Training and Canada's Economic Performance Winter Ottawa

Romain K (1991) Toeing the Quality Line Globe and Mail 9 September p Bl

Wiggenhorn W (1990) Motorola U: when training becomes an education in Harvard
Business Review July - August

2 Quality assurance in a European
context

David Alexander and Jim Morgan. Leeds Metropolitan University, Leeds, LSI
3HE, UK

INTRODUCTION

"In the European Course Credit Transfer System (ECTS), a problem that is only latent in
Erasmus becomes prominent, viz, that of calling all institutions of higher education
universities, and allowing them to be in the same transfer system. The European Council
Decision of 15 June 1987 stipulates that the term university shall be used to cover all
types of post-secondary education and training establishments which offer, where
appropriate within the framework of advanced naining, qualifications or diplomas of that
level, whatever such establishments may be called in the Member States. Admittedly,
there are non-universitary institutions of higher education in each EEC Member State that
deserve a quality label as solid as that of the universities in the strict sense of the word.
But as the Leuven Department of Mechanical Engineering has found out, this equal
fooling can have its dangers, and may lead to legitimate reactions of guarding the
indigenous quality. And this time* there is in the word quality an implication of better or
worse, higher or lower because, within ECTS, the university does not have a choice as to
which partners to accept or reject." Latre and Del Martino (1991) p6.

"This paper has identified some of the specific items that are held to be indicators - they
are often little more than surrogates - of quality. Though they are all deemed to assess
quality in different national circumstances, they themselves vary considerably. There is
no agreement on the purpose of quality assurance, save only as a resource allocation
device or perhaps as a resource withdrawal device. Consequently, it is those features of
an institution's performance which lend themselves to cross-institutional comparison which
are tending to emerge. Often these are of a rudimentary nature, in comparison with the
complexities of the inner life of institutions." CNAA (1991)

MEASURING QUALITY - THE NUMBERS BIT

This ought to be the easy part. Student results in one educational system need to be
translated into ihe parlance of other educational systems. In pass/fail terms this may be
fairly straightforward. But as soon as any form of classification or grading becomes
involved - as in Higher Education it surely must - then considerable difficulties arise.

The BA (Hons) European Finance and Accounting Degree at Leeds Business School
is at present a 3-legged Erasmus supported ICP, involving Leeds, Hochschule Bremen,
and ESC Normandie. Students spend 3 Semesters at "home", 2 semesters at one of the
other institutions in full-time assessed study, and the final semester at home. Successful
students receive 2 separate qualifications* 1 from each relevant country, The formal
mechanism for this is that assessments are taken in Semesters 4 and 5 in the "foreign"
language and cultural context, and also in Semester 6 in the "home" language and cultural
context. This produces a single set of marks, which are then fed separately into 2 legally
different sets of examination regulations. These regulations can have their own
differences. For example, a student who has 2 attempts at an assessment, receiving marks
of 30 and 32, would be regarded as failing the module for UK purposes. For German
purposes the student would have the right of a further attempt. Conversely a student
receiving marks of 30 and 38 could well be regarded as achieving a 40 mark for UK

QUALITY ASSURANCE IN A EUROPEAN CONTEXT

purposes and therefore a pass, but such subjective adjustment could not happen at a
Gentian examination board.

More fundamentally, however, where do the numbers themselves come from? Table
2.1 gives the agreed conversion scales used for students enrolling in 1989 and 1990 on
the same course in the same circumstances. The relationships between the two years are
significantly non -linear. We must remember that these are actual scales producing
numbers which determine the degree classifications of actual individuals. It is clear that
the decision as to which scale to use could influence degree classification or even
pass/fail.

1989 1990

French

CNAA

German

French

British

German

Mark

Mark %

Note

Mark

Mark %

Note

100

1.0

85-100

1.0

90-99

1.0

80-84

1.0

80-89

1.0

75-79

1.3

75-79

1.0

70-74

1.7

70-74

1.3

67-69

2.0

67-69

1.7

63-66

2.3

64-66

2.0

60-62

2.7

60-63

2.3

55-59

3.0

57-59

2.7

50-54

3.3

54-56

3.0

45-49

3.7

50-53

3.3

40-44

4.0

46-49

3.7

38-39

4.3

40-45

4.0

36-37

4.3

38-39

4.3

34-35

5.0

35-37

5.0

32-33

5.0

30-34

5.0

30-32

5.0

2029

5.0

25-29

5.0

15-19

5.0

20-24

5.0

10-14

5.0

10-19

5.0

0-9

5.0

0-9

5.0

Table 2.1 Examination mark conversion scales

How arc these scales prepared? The intention is that, taking one cohort of students with
another, there should be the same number of firsts, upper seconds, etc from each
country's batch of students. Or, to put it the other way round, that a student should have
the same chance of a first whichever his home and foreign institutions.

Given the intention, the detail follows on a rough and ready statistical basis, supported
by discussion and subjective judgement between colleagues from the institutions. The
crucial difference between the two scales, of course, is the minimum pass mark for each
module in France and the UK. There was a widespread view amongst staff that the 1989
scale was producing wrong results. Notice incidentally that the long tails in the German
Note present obvious difficulty. A 5.0 has to be translated into a specific CNAA number
for condonement and averaging considerations. Our German colleagues have no such
problems, as condonement is not possible under their exam regulations.

Leeds Business School is about to extend this programme to a fourth partner, at Pescara

PERSPECTIVES ON QUALITY IN ORGANISATIONS

in Italy. The first stage in the process of producing a fourth column for kaly in the
scales is to establish typical Italian thinking in first-degree assessment and grading.

In the Italian Laurea System results are based on an overall mark out of 110. Each
course is given a mark out of 30 if it has been passed (the assessments are nearly always
orals). A bare pass is given 18, a failure is not given a mark. Resits are possible without
limit (but not to raise a passed mark). The marks from as many as 30 courses are then
averaged, with equal weighting. This average is then divided by 3 and multiplied by 11,
to give a mark out of 110. The thesis is then presented and defended in public, and
marked by a jury of 1 1 , out of ten. This mark is then added to the mark out of 1 1 0, and
the total is regarded as a mark out of 110. The theoretical maximum mark out of 110
is therefore 120, but a total over 110 is regarded as 110. It is this final mark out of 110
which gives the Laurea its "classification".

The task of moving from this starting point, to an extended fair and equitable
Conversion Table, is one we approach with some interest!

BEHIND THE NUMBERS - WHAT ARE WE TRYING TO
MEASURE?

It might appear from the discussion so far that the major problem is to create the best
possible numerical conversion scale. In other words it might seem that there is a
theoretical "right" version of the scales, and that the problem is that we can never more
than approximate to it. This conclusion would be grossly misleading and optimistic
however. A "right" version of scales only exists even conceptually or in theory if there
is a commonality of view as to what it is that we are trying to measure. There is
absolutely no such thing.

Education, educational process and educational measurement are all subjective
processes. They are defined and monitored by human agents. Tutors are entitled to be
as individualistic as learners are. But that is not the point here.

In any particular community or any particular culture the majority of members will tend
to concentrate round the norm as regards any particular defined dimension. In the
educational sphere we might hypothesise, for example, that the majority of an educational
community will tend towards a broad agreement on the issue of the importance of theory
and practice, and of the desirable balance between the two elements. The key issue,
however, is what is meant by the phrase "an educational community". It is suggested
here that, bearing in mind we are talking in a pan-European context, "an educational
community" will often need to be defined fairly narrowly for the above hypothesis to be
valid. This narrowness may well apply in both geographical and discipline focus terms.
"German higher education" is too broad. "Business higher education" is too broad.
"German business higher education" may be narrowly enough defined to justify
generalisations.

We are led to the hypotheses that:

1 different discipline focus areas may, in general, have significantly different
higher education aims and philosophies, and therefore practices;

2 different geographical/cultural regions, within a single discipline focus, may
have different higher education aims and philosophies, and therefore practices.

In the context of student mobility, the latter issue is probably the more significant one in
its effects. By way of illustration we briefly discuss here some perceptions of business
education, and some particular experience with dissertations.

QUALITY ASSURANCE IN A EUROPEAN CONTEXT

The major French Business Schools are some of the most high profile higher education
institutions in Europe. They are characterised above all else by close integration with the
real business world and by a strong emphasis on practical application. This is
encapsulated in the commitment to the case study as being the major learning mechanism.
Work-placements are crucial. All this has a negative as well as a positive side. The
emphasis can sometimes be on practice, rather than on theory and its application to
practice. There is a strong interest in group work, personal skills, and synthesis of
practices. There is less interest in reading, in theory, and in personal intellectual
development. The French Business Schools and their products (they should not be
confused with Economics-related departments at French Universities), have enormous
importance in French business activity.

Germany has no such tradition, there is no longstanding German equivalent. But it is
difficult to argue on empirical grounds that the German economy and business community
have been disadvantaged thereby. The approach to business education in Germany is
much more "academic". In our experience this approach is characterised by emphasis on
libraries, o?> reading, on lecture or lecture and exercise, rather than on seminar or case
study. Work placements may not be involved.

UK practice is probably somewhere in the middle, with named "Business Studies"
degrees attempting (usually very broadly) to adopt the work-related French practices,
whilst more specialist degrees may tend to a more library-based approach.

These generalised differences are well illustrated and demonstrated in the approach to
dissertations we have discovered on the Leeds Business School European Finance and
Accounting Degree. In summary, we can suggest the following:

• Germany - library based, long, detailed, ("Fussnoten Fetischismus"), personal
opinion neither encouraged nor expected;

• France - real-life, empirical base, little interest in theory;

• UK - library or library-empirical, personal opinions and appraisal regarded as
essential;

• Italy - extremely rigorous, theoretical, assessed on a public presentation
(usually called a "defence" which is perhaps indicative).

We might suggest there is very little commonality here as regards educational criteria.
So how can we monitor quality across the whole multi-country programme?

The numbers conversion problem discussed in the previous section can in theory be
eliminated by agreeing a single common marking scale across all partner institutions. But
this does not get us very far unless we can also create a single coherent pan-programme
statement of the desirable characteristics (and their relative importance) of a "good"
dissertation. This means, in other words, doing away with national (ie geographical or
cultural) differences.

The other possibility is to say that these differences are expressed in different
dimensions and are therefore not ultimately comparable. No one dimension can be
translated into the measurement units of the other. (This room has the better shape, that
room has the bene lighting. Which is the better room?) We simply accept - or reject -
that these very different skills, objectives and attainments are of similar quality. (Both
are good rooms.) This can be done by measurement of inputs, the use of surrogates, or
by overt subjectivity. This last may not be at all a bad idea. In other words we are
recognising the inevitability (and perhaps the desirability) of "national" (geographical and
cultural) differences.

There can be no fudging of the issue, no half-hearted compromise. We cannot, with
any semblance of objectivity, cross-monitor quality by an output measure unless we adopt
a single European Education Currency Unit.

PERSPECTIVES ON QUALITY IN ORGANISATIONS

CURRICULUM DESIGN

The characteristics of the system for curriculum design in England may be summarised
as follows.

a) There is no overt state regulation, though the government exercises influence,
or attempts to, through its funding and student maintenance policies. Even
here it is acting through intermediaries - PCFC, UFC, Local Authorities.

b) Responsibility for approval of the curriculum rests with individual universities and
for a little longer with CNAA, BTEC. These bodies provide only a loose
regulatory framework - loosest in the universities and tightest with BTEC.

c) Curriculum design rests with a team of academic staff who normally act
within the assumptions and traditions of their subject or professional area.

d) Approval involves an element of peer, and normally, external peer, review.

By contrast continental Europe is characterised by state regulation of the curriculum and
a lack of external peer review.

In Finland the basic degree (equivalent to a UK Masters) is regulated by Council of
State decrees which lay down the objectives, structures and conditions for awards in 19
fields of interdisciplinary and vocationally relevant study. It is a credit based system with
the basic units of study classified into general, subject and advanced.

However, within this framework the syllabi and assessment of units are a matter for
professors, with the approval of a representative Departmental Council. It is very unusual
for this Council to question, let alone reject, a Professor's proposals.

In Italy the basic degree of Laurea lasts four years. The Ministry of Education approves
the examinations which constitute a particular Laurea. When a new Laurea is proposed
it receives the approval of a university and then has to be enacted by the Ministry.

However, this central control is more apparent than real. The Ministry names the
examinations but individual professors have great autonomy ir determining the content
of the curriculum and assessment. In Engineering, however, there is a national system
in which the first two years are largely common to all universities as a result of pressure
from industry and the professions.

One might conclude from this brief survey that the difference between the devolved
approval systems of the UK and the state regulations of continental Europe is more
apparent than real. Paradoxically, the real contrast is between the practical professorial
autonomy in Finland and Italy and the practice of peer review in England.

COURSE MONITORING

In England this is normally conducted through some form of regular monitoring meeting
by staff (and sometimes students) and through the external examiner/moderator system.

Germany and Italy do not have any formal, regular monitoring. Germany, however,
conducts a formalistic system of assessment with the moderation of examination papers,
double marking and final determination by an elected Examinations Committee. Each
subject is required to produce an annual report on student performance to the elected
Departmental Council which includes student representation. However, there is no
external moderation.

In Italy much assessment is still oral and conducted in theory by three professors, one
of whom is a specialist. In the early years of the Laurea this seems to be increasingly
compromised because of the pressure of student numbers, leaving decisions to the

QUALITY ASSURANCE IN A EUROPEAN CONTEXT

individual professor's judgement. There is no monitoring system. Informal student
pressure is exercised through their choice of examinations to study - by avoiding the
poorly taught or severely examined - or, at the final resort, through protests and strikes.
Again there is no external moderation element.

Once again the obvious contrast is between the autonomy of German and Italian
professors and external peer moderation in England.

CONCLUDING ISSUES

These brief excursions into only some of the realities that lie behind the generalities of
quality assurance systems illustrate the practical issues that have to be addressed in the
context of pan-European higher education. Behind these practical issues lies a more
fundamental difference between the English use of external peer review and the
continental reliance on the professional autonomy and integrity of the professor. Perhaps
even moar fundamentally England has a determination to assure an equivalence between
institutions at variance with continental practice, which accepts the existence of formal
and informal hierarchies.

At the operational level of delivering joint programmes and exchanging students the
essential issue is one of standardisation or equivalence. Arguably the only practical
option is equivalence and, even more arguably, acceptance of the practices of each other's
institutions.

When we look to the future we should also observe that continental European higher
education provides for a mass student market. Can England's expensive quality assurance
procedures survive the arrival of mass higher education? And should it? In the last
analysis quality does depend upon the professional competence and integrity of individual
tutors and professors. Arguably no system assures real quality. And perhaps market
perceptions could, and should, be the final judge?

References

Council for National Academic Awards (1991 ) Models of Quality Assurance in Europe:
discussion paper No 6 CNAA London

Latre G and Del Martino F (1991) Traditional Universities in New Europe: the case of
K U Leuven TEXT Conference paper Toulon September 16-19

3 "What's in it for me? 11 Organisational
culture, rewards and quality

Christopher Hart and Michael Sholbred, University of Central England at
Birmingham, Birmingham, B42 2SU, UK

INTRODUCTION

Quaiity systems generate a quality culture and this is what is at the heart of successful
organisations. Here we have examined some of the literature on successful quality
organisations. We have followed this path because both manufacturing and service
industries spend a great deal of time addressing the topic of cultural change. We believe
that if further and higher education are going to make serious moves towards effective
quality assurance, whether through a QMS (Quality Management Systems) or a TQM
(Total Quality Management) approach, they need to be aware how much the culture may
have to change. This may be highly uncomfortable for senior management. For instance,
a 1989 survey of 3.000 company quality schemes in the UK identified one major
problem:

"Almost all companies have fouM it difficult to achieve cultural change and to get
management, especially lower and middle management, to change their ways of working"
(Develin 1989).

As we move towards quality in HE, everyone will have to make changes.

WHAT ARE ORGANISATIONAL CULTURES AND
CLIMATES?

After identifying no less than 164 different definitions of culture, Kroeber and Kluckhohn
(1952) conclude that:

"Culture consists u patterns ... of ... behaviour acquired and transmitted via symbols,
constituting the distinctive achievement of human groups, including their embodiments in
artifacts; the essential core culture consists of traditional (ie historically derived and
selected) ideas and especially attached values; culture systems may, on the one hand, be
considered as products of action, on the other as conditioning elements of future action/

In the 1980s and 90s we have rather snappier definitions. "Culture is the set of values,
behaviours and norms which make an organisation tick" (Atkinson 1990). Snappier still,
Marvin Bowers* immortal quote that organisational culture is: "the way we do things
around here" (Deal and Kennedy 1982).

An organisation's culture takes years to mould, develop and grow, and years to change.
An organisational climate, on the other hand, is described by Botterill (1990) as: "the
collective current impressions, expectations and feelings of the local work units".

The climate of a unit or department is the "now". It can change instantly. A sharply
worded memo or a derogatory remark in a meeting or overheard criticism can, on
occasion, radically change the climate of interaction. Climatic attitude is the day-to-day
relationship the individual (and sometimes the group) has with others in the organisation.
Unlike climatic attitude, culture is more enduring and stable. Some key characteristics

ORGANISATIONAL CULTURE, REWARDS AND QUALITY 17

of that culture are that it is: commonly held rather than shared, real only in its use, ie
consequences, historically based, learnt, heterogeneous.

What indications are there of corporate culture? These are many and varied and in HE
they might include: specific management styles, "telephone style within the organisation,
willingness to give personal tutorials, how staff spend their lunch breaks, feedback to
students, teaching styles, tidiness in offices, graffiti in the loos, quantity of visual
stimulation on the walls.

All these are possible indications of organisational culture within an educational
environment. They can be positive indications of an organisation with a quality culture.

WHAT IS DIFFERENT ABOUT QUALITY ORGANISATIONAL
CULTURES?

From our own observations and from our researches we are quite certain that there are
some major differences between the culture of organisations which are held to be
successful and which present a quality image and those of other organisations. What we
are now going to describe is surely as true for further and higher education as for any
other service organisation.

Firstly, quality organisations tend to have quite specific quality values, whatever sector
of the market they are working in.

In many organisations values may not be explicit. Perhaps further and higher
education are different in this respect. Staff will frequently be aware of their own
personal value systems and they will sometimes contrast those with the mission
statements of their organisation, or with management actions which seem to belie those
mission statements.

So what kind of quality values underpin quality behaviours? Linkow (1989) suggests
seven core values: 1. customer focus, 2. employee focus, 3. teamwork, 4. safety: for
employees, communities and users-of products and services, 5. candour v 6. total
involvement, 7. process focus: emphasising the continuous improvement of all processes.

This is by no means a complete list. We might want to add to it such values as:
intimacy, integrity, unity, consensus, excellence.

We believe that it is absolutely essential that values in higher education are made
explicit and that there is both the unity of purpose and a unity of belief between
managers, teachers, administrators and students.

RITES AND RITUALS, MYTHS AND LEGENDS

One increasingly common way of examining the culture of an organisation is to look at
its manifestations through such areas as heroes, rites and rituals (Linkow 1989). Heroes
are an important influence on behaviour in an organisation. Heroes are individuals who,
in deed and presence, personify and transmit the culture's values. They provide tangible
cues and act as role models for others to follow. From our own varied experience we
have come across a number of heroes. Comments from colleagues on their heroes
include:

"She's so laid back with the HMJ"
"The guy was a walking libraiy"

"Everything he says carries so much force and conviction"

"She stepped in at the last moment and gave a key note speech better than anyone else
could have dene!"

"He showed a slide of a field and had students enthralled with tales of Eric Blood Axe"

PERSPECTIVES ON QUALITY IN ORGANISATIONS

Heroes are influential within the organisation. Most are situational heroes. They don't
charge out of the trenches but make the right movement at the right time. They are the
stuff that organisational legends and exaggerations are made of. A commonly
acknowledged fact of higher education is the preponderance of anti-heroes over heroes.
Anti-heroes are often the focus of gossip. They are the deviants of the organisation.
Their deeds signify boundaries of what is and what is not currently acceptable behaviour
(Atkinson 1990). When office doors are shut, stories of anti-heroes are often told, such
as:

"She skives off all the time"

"He is never around when you want him!"

M He has got a private business on the side"

"Power has gone to his head"

"She abuses the rules"

"He sleeps with students"

Rites and rituals (Trice and Beyer 1984) are the programmed and routine interactions of
organisations. In higher education we have numerous rituals including the annual
"induction ritual" for new students. This cyclical ritual minimises disruption and shows
newcomers the kind of behaviour expected of them.

Common rituals for members of the Department are, "the staff meeting" or "the Board
of Study". A particular ritual of higher education in recent years has been the "touching
base" meetings ritual. This consists of staff, often from diverse Departments, being called
to meet and "look at" some project or other. Vague indication is given that some
document or other needs to be produced and this cohort will produce it. For instance,
the "Faculty Planning Group" and the "Research Strategy Group" are common groups
throughout higher education. They meet and, without much experience or direction,
manage to produce a document. Once produced, members are seen to have "served their
time on this one".

Rituals in higher education are widely experienced as lacking direction or clear
integration with the values of the organisation.

WHAT'S YOUR OWN ORGANISATIONAL CULTURE
REALLY LIKE?

Values are the bedrock of any culture. They provide a sense of common direction for
all members of the organisation and guide day-to-day behaviour. The following exercise
will allow you to look at the pattern of values in your organisation. It will assist you in
understanding where your organisation has got to on the road to quality values. Go
about the exercise by using the following instructions with figure 3.1

INSTRUCTIONS
Look at the list of values
in the box "zones of relevance'
in figure 3.1

Tick up to 7 values that you
believe represent the values
held by each role holder

COMMENTS

Some represent values
of quality practice

Play yourself and take
the role of the other
Understanding other
people's role is crucial
to quality

ORGANISATIONAL CULTURE, REWARDS AND QUALITY 19

3. Add up the score for each role High scores indicate
placing the total at the end a base from which quality

of each row values can be developed

Low scores indicate need

for change in values

From our experience of working with several institutions of higher education we have
noticed an inverted relationship between the values and expectations of management and
those of staff. Where staff widely practise quality values and expect the kinds of rewards
associated with such values (see Vroom 1964 on "expectancy theory") management has
yet to embrace these values and recognise the expectations they create. We conclude that
the kinds of values held by some management are a major barrier to a quality
organisation. If management does not address these issues we predict that polarisation
will increase, leading to entrenched positions between management and staff. The result
will be internal conflict - the antithesis of quality.

Zones of Relevance
(Degree of attachment to values)

Blackend Institute /& . 'J f . '/. /. >°.^. '/. '/. '/. >?. ' £f
of Higher Education * . ff . /. f. /. g. ^

Roles 12 11 10 9 8 7 I 6 5 4 3 2 1 Score

Roles

Score

YOU

THE DEAN

THE DIRECTOR

Figure 3.1 Matrix of values

WHAT'S IN IT FOR ME?

As further and higher education institutions move towards a quality culture, regardless
of the system that they use to introduce that culture, they will need to reward and support
their staff. If staff are expected to work towards targets, levels of performance and
diversify their activities, they will expect rewards. Management will need to recognise

PERSPECTIVES ON QUALITY IN ORGANISATIONS

this expectation and address it with equity and openness. Contrary to popular belief,
management will not need to attend to motivating staff, but to harnessing the values staff
hold as the foundation for organisational quality.

In commerce, the drive for customer satisfaction is often linked with remuneration. As
yet, further and higher education in this country is only just beginning to grapple with the
introduction of performance related pay for middle management and other staff. If staff
are to work hard towards promoting quality they will surely need both rewards and
recognition. How are these to be achieved? Rewards and recognition might come
through: training and staff development, an improved working environment, improved
mutual respect, some form of financial remuneration.

CONCLUSION

We've suggested that if further and higher education are to deliver quality systems then
they will need to examine their cultures. Quality in education is concerned with having
an appropriate range of services, and getting the service right on each and every occasion.
Clearly, quality costs. But in making decisions about quality, consider the costs of
ignoring quality: poor communication, misinformation, fear and insecurity, lack of
direction, crisis management, lack of information flow, too many conflicting priorities,
no identifiable mission, "make it cheap" attitude, time wasting in all forms, time table
changes at short notice, crisis cycles.

Avoiding the waste and demoralisation of traditionalism necessitates management
change. Senior management needs to get its internal customer relationships right if it is
to provide excellent quality relationships with its external customers.

An appropriate and supportive culture is at the very heart of a quality organisation. The
entire workforce must have some kind of common identity.

"All employees must believe in the company itself, its culture and their own importance
to the overall success of the product or service. This belief must emanate from within the
leadership where the key to corporate culture is PUTTING YOUR PEOPLE FIRST. Your
employees are your greatest and only appreciable asset, and your treatment of them will
be reflected in the quality of both product and service" (Botterill 1990).

Ultimately, it is that sense of corporate belief and shared values which will distinguish
quality organisations in education. That may be what is in it for you.

References

Atkinson P F (1990) Creating Cultural Change in Management Services 34(11) pp6-l0

Atkinson P F (1991) Creating Culture Change: The key is successful total quality
management IFS Ltd Kemptson. Bedford

Botterill M (1990) Changing Corporate Culture in Management Services 34(6) pp 14-18

Deal T and Kennedy A (1982) Corporate Cultures; the Rites and Rituals of Corporate
Life Penguin London

Develin N (1989) What's Wrong with British Quality? in Quality Today November
pp 44-50

3 j

ORGANISATIONAL CULTURE, REWARDS AND QUALITY 21

Kroeber A L and Kluckhohn C (1952) Culture: a Critical Review of Concepts and
Definitions Vintage Books New York

Linkow P (1989) Is Your Culture Ready for Total Quality? in Quality Progress 22(11)
pp 69-71

McClelland D C and Burnham D H (1974) Power is the Great Motivator in Harvard
Business Review 54 pp 100-110

Trice H M and Beyer J M (1984) Studying Organisational Culture Through Rites and
Ceremonials in Academy of Management Review 9 pp 653-659

Vroom V (1964) Work and Motivation J Wiley New York

Walsh S P (1990) Successful TQM Training in TQM Magazine 2(3) pp 165-169

4 Quality and the academic
administrator

Catherine D Payne, Leeds Metropolitan University, Leeds, LSI 3HE t UK
INTRODUCTION

It is self-evident that the quality of a product or service is underpinned by the efficiency
and effectiveness of all the operations that contribute towards the product or provide the
service. Thus quality initiatives which seek to improve efficiency and effectiveness,
particularly TQM, have an impact on all groups of staff but each group may view some
of the issues differently. Whilst it is recognised that all staff are involved to a greater or
lesser extent in administrative functions, the academic administrator can be seen as the
facilitator between the teaching staff as the deliverers of the primary service (courses) and
management as the providers of policy and strategy. Here I shall draw on research into
the efficiency and effectiveness of Course Monitoring and Review process in the School
of the Environment, Leeds Metropolitan University.

There are many questions raised by consideration of TQM in this context. I shall focus
on three only:

• how to establish the internal customer/supplier relationships;

• the use of performance indicators;

• the application of TQM principles in a non-TQ environment
INTERNAL CUSTOMER AND SUPPLIER RELATIONSHIPS

The principle raison d'etre of a TQM organisation is to ensure that its product/service is
fit for its purpose and satisfies customer requirements in order that they will repeat their
custom and encourage others to be customers thus ensuring the continuing enterprise of
the organisation, hopefully at a substantial profit. Within the education context we know
that our principal customers - students - require a good quality education that will fit
them for their chosen career. It is the Institution's responsibility therefore, to ensure that
courses are fit for that purpose both through the curriculum and delivery. However,
Atkinson (1990) argues that the focus on the external customer must not be at the
expense of ignoring the internal customers and finding out their requirements.

As part of the research into Course Monitoring and Review, interviews were conducted
with staff in various functional roles including managers, teachers and
administrative/clerical staff. A question about the external customer and internal
customer formed part of the semi-structured interview. Whilst everyone understood very
clearly who was the external customer(s) - primarily the students (but also employers, the
professions. Government) - the concept of an internal customer was not so clearly
perceived. However, as Leeds Metropolitan University is not as yet a practising TQM
culture this is perhaps not so surprising.

Who is our Internal Customer/Supplier?

John Oakland (1989) states that: "The next person who checks [uses] your work will be
your customer" p 243.

QUALITY AND THE ACADEMIC ADMINISTRATOR

Like many of the principles of TQM this is very simple to state. It is suggested that
the reality is much more complex. The many inputs and outputs to the process of Course
Monitoring and Review typically include: employers, validating bodies, professional
groups, HMI, external examiners, students, staff, committees, boards, student services,
academic standards. These may be said to represent the myriad internal customer and
supplier relationships that exist for this process alone. If we superimposed on this the
numerous other activities that we are involved with we would see that, whilst some
customers and suppliers would be removed, others would be added. Indeed even this list
has failed to include those support staff who provide the domestic arrangements for a
review.

Professor Oakland (1989) uses his Quality Chains diagram to illustrate the links through
an organisation from external supplier to external customer (see figure 4.1).

Outside organization

Supplier/
Customer

, External customer

Supplier/
^ Customer

Supplier/
Customer

Supplier/
i Customer

Supplier/
Customer

External supplier

Outside organization
Figure 4.1 The quality chains

It could be argued, however, that the customer chains obtaining within HE institutions
do not and cannot take such a simple linear form. For instance, a senior administrator
in a Faculty works with managers, other administrators and teaching staff. Sometimes
our relationship to one or other of these groups is as a supplier and sometimes as a
customer. Often they can both be in the same process! An example from my own
experience would be: to produce the School Examination Timetable the teaching staff
have to supply me with their requirements. I piece the jigsaw together which necessitates
liaison with a central Polytechnic administrative office - I supply information to them,
they make accommodation bookings for me (customer). I produce a timetable for my
customers, the teaching staff (previously suppliers), and ultimately for the external
customer the students.

C kJ

PERSPECTIVES ON QUALITY IN ORGANISATIONS

What is suggested by this example therefore is that the notion of internal customer and
supplier fails to take into account that these relationships flow in both directions, at least
certainly for the administrator who is caught in the middle! This suggestion is not
intended to be an argument against the value of establishing internal customers and
suppliers and finding out their requirements but to suggest that as we do not work in a
production line outfit, a different interpretation is perhaps required. For instance, quality
chains may be more appropriately applied on an activity basis as opposed to a role basis.
Figure 4.2 shows how this interpretation has been applied to the activity of examination
timetabling.

EXAM TIMETABLING

Me as Customer

Me as Supplier

Teachers

Quality of

Awards & Exams

Service

Awards & Exams

dependent

Draft/Secretary

upon

TT wordprocessed

accuracy

and

Teaching Staff

timeliness

of data.

Students

Figure 4.2 Internal customers /suppliers of exam
timetabling process

This approach helps unravel the complex links of a non-linear chain. Once unravelled
we can more easily answer the key questions:

• Who are my customers for this activity?

• What are their requirements?

• Who are my suppliers for this activity?

• What are my requirements?

PERFORMANCE INDICATORS

The extent to which we do or do not meet requirements is the performance indicator (PI)
of the quality of our service in our particular role or function.

Once the internal customer/supplier link is established the next stage is to establish the
precise requirements in that relationship. We are probably all too familiar with how the
requirements of a particular role or task are so often fogged by expectations on the one
hand and assumptions on the other. We have probably all heard and used the expression
"But I thought you were supposed io be doing that." When roles and requirements
remain undefined it is inevitable that this will happen.

TQM gurus use the expression "right first time" (RFT). If this is the ultimate
performance indicator why is it so difficult to achieve, bearing in mind that Performance

QUALITY AND THE ACADEMIC ADMINISTRATOR

Indicators (Pis) should be achievable? Is it an appropriate PI given the nature of our
tasks and the environments in which we work? The days of tradition and stability are
over, replaced by a high-speed culture that is constantly changing, leading to the
phenomenon of "grey stress" (Bedell 1992) as we come under pressure to adapt and
perform. Does this culture permit the time and space to perform RFT?

Not everyone is convinced that TQM is the best thing since sliced bread. RFT would
make all our lives easier - no more redrafting of notices of meetings because you got the
date wrong, because you were interrupted by the telephone and someone came to your
desk wanting just a quick word; no more redoing the student numbers spread sheet for
the nth time because some enrolment forms have been discovered on a staff member's
desk three months after enrolment; no more redrafting the Course Document because it
has been decided to change the sequence of sections. Yet these events are the realities
of our daily lives. Indeed it is as much a part of our service to internal customers, as to
external customers, to respond to requests and change. Is RFT not then a rather
sanctimonious view of an ivory-tower theorist? Add to this the patronising comments
from senior management such as "we never have the time to get it right first time but all
have the time to get it right second, third time" and RFT as a PI can really stick in your
throat.

What is the nature of this PI and others within our real environment? Are they not in
fact a stick with which to threaten rather than motivating objectives? Certainly at the
macro level they are used as such by the funding bodies. If objectives and targets are not
met then fu.ding is reduced. In this regard Pis take on the quota function of the
production line. The answer to these criticisms of course is that it all depends on the
culture in which Pis and all the other principles of TQM are applied - the subject of my
final section.

Ignoring the arguments against Pis it is contested that they are still a very problematical
area for the academic administrator. For Pis to be non -subjective they have to be
measurable, ie. subject to SPC (statistical process control). What sort of Pis could be
applied to the administrator?

A few suggestions might be: turnaround time - how long for a specific task?; 24-hour
response time for standard information; accuracy; sensitivity to different perspectives;
value as an information source; use of secretaries.

For Pis to be realistic and achievable they must be looked at in terms of the internal
customer/supplier relationships and the number of those that exist. If you have a
reputation for being good then you attract more customers but how many customers can
be realistically and effectively dealt with before the service is reduced and your reputation
tails off - as with a business over-extending itself? It may actually be necessary to
reduce customer voiume in order to improve quality. (Heresy!!)

There is another difficulty consequent upon applying Pis in a non-TQ environment.
Whilst they may be legitimate to measure the performance of an organisation, in our
present culture they become personalised - they become focused on an individual rather
than on the structures/teams in place to support that individual in their activity. Taking
account of the above limitations, the reality of our present (non-TQ) lives could counter
each of the suggested Pis as follows:

• turnaround time

... four weeks for an Exam Timetable - what if the Dean wants a breakdown of
student numbers by domicile in the meantime?

* 24-hour response time for standard information

... what is standard? If its not in "the book 1 ', is it not standard? What
happens when the person who can give you the information is away till next
Tuesday?

PERSPECTIVES ON QUALITY IN ORGANISATIONS

♦ accuracy

... no argument on this provided we work in an environment that is conducive to
concentration and checking

♦ sensitivity to different perspectives
... how do you measure sensitive?

♦ value as an information source
... how do you measure value?

♦ use of secretaries - this is a real one!

... how many times do we send a piece of work back because we have
redrafted it as opposed to there being typographical mistakes to be corrected?

Ideally - in a TQM environment - none of these provisos should apply. In the meantime
the reality is that they do.

TQM PRINCIPLES IN A NON-TQ ENVIRONMENT

Readers will probably know that TQM is a philosophy which sets out to achieve optimum
efficiency and effectiveness by reducing waste and error through the application of certain
principles and operating methods to achieve quality in every aspect of an operation to
satisfy external customer requirements.

So far I have focused on just two of the principles/operating methods required for a
TQM culture - establishing internal customer/supplier relationships and the use of
performance indicators. However desirable, it is suggested that these principles cannot
be applied in isolation of an all-encompassing TQ culture. I shall now explore some of
the key features that are required for this culture.

Leadership and Commitment

Quality is a strategic issue and an attitudinal position. The decision to go down that road
must be led by the Senior Managers of an organisation for it is from this level that
attitudes in an organisation are inculcated. It is through the example and commitment of
Senior Managers to quality that the whole organisation is enabled to adopt a quality
ethos.

Resources

The provision of resources for education and training is crucial - to educate people in
ways of working in a quality environment and training to practise error prevention rather
than error detection; in the use of problem-solving techniques and tools.

Another valuable resource is time - time to do the job RFT. Time management courses
may help but consideration also needs to be given to the actual management of staff
resources to support the various activities.

Communication

Open communication is a crucial feature of a TQM culture. In order for employees to
feel committed to an organisation's objectives they need to know and understand what
these are. Decisions are made with regard to objectives - how to implement them, why
they nrght change, how they bear upon an employee's place and operation of work.

QUALITY AND THE ACADEMIC ADMINISTRATOR

Employees are the implementation channel through which an objective is met. In a TQM
culture, however, communication goes further to become an interactive process leading
to participation.

Participation

An organisation may have very good channels of communication - newsletters, up-to-date
notice boards, regular policy statements - but if people are not involved in the decision
making processes a sense of belonging and commitment is not cultivated. A non-
participative style of management perpetuates an "us and them" syndrome. My research
showed evidence of this attitude at Leeds Metropolitan University - between teaching
staff and management, teaching staff and administrators, administrators and managers.
Each perceived the other group as taking decisions which affected their lives but without
reference to them. "Old style" management might consider that it is the prerogative of
management to make "managerial decisions". TQM does not sustain this prerogative.
Instead it recognises that participation has a lot to do with an individual's sense of power
and control. Powerlessness undermines confidence and motivation and does not inspire
a sense of responsibility. Participation brings about a sense of involvement, identity and
responsibility - for the task and the objective.

Team Building

Establishing a culture of open communication and participation can be a foundation for
team building. The value of team building has often been stressed in management
handbooks. In TQM, teams may come under the guise of "quality circles", "problem
solving teams". Whatever their guise it is suggested that they can achieve more than their
purpose at hand if they are cross- functional and/or cross-departmental. They help build
on participation and involvement. They help build communication. A suggested quasi-PI
was sensitivity to another's perspective. I can iiave great empathy with fellow
administrators. I know "where they're coming from", what their problems arc. This
helps establish a strong cooperative spirit. Teams can help establish empathy across
different roles and functions, it enables us to get to know each other better and our
different perspectives. This increased awareness reinforces the principle of internal
customer and supplier.

These selected basic principles of TQM are in themselves not that original in
management theory. But it is the combination of them in a specific context that makes
TQM different and so very difficult to achieve. It is difficult to implement because it is
a very long term and continuing process, requiring wholehearted commitment and.
probably, an initial outlay of resources. More significantly however, it requires a
fundamental change to most organisations' corporate culture: from a closed culture that
is secretive and dominated by coercion and intimidation, resulting in staff being defensive
and non -critical of performance, to an open culture where fear is driven out, barriers
broken, targets eliminated. Where people are self-critical and take pride in their
workmanship - these are the values which provide the impetus for improvement.

CONCLUSIONS

My conclusions are simple. Quality can be a personal standard. Right first time can be
a personal standard within the constraints of an individual's environment. TQM is posed
as the panacea to lift those constraints so that quality can be applied across the board.

28 PERSPECTIVES ON QUALITY IN ORGANISATIONS

But does TQM pose more questions than it answers? A few have been suggested from
the administrator's point of view, there are many more view points. TQM is different
from our usual experience, it is another change:

References

Atkinson P (1990) Creating Culture Change: the key to successful total quality
management IFS Ltd UK

Bedell G (1992) Tension mounts as the slow lane speeds up The Independent on Sunday
8th March

Oakland J S (1989) Total Quality Management Butterworth-Heinemann Oxford

5 Total quality management in an
education and training context

Barbara Ellis, Education and Training Consultant, Arlesford, Hampshire, S024
0HH f UK

DEFINING QUALITY

Philip Crosby, an American quality "guru", defines quality as, "Everyone in the Company
from the doorman to the managing director, routinely doing things right".

This is a helpful, but in one respect a puzzling, definition of quality for Further
Education and Training. It is in taking what is helpful, and in grappling with what is
puzzling, that further education establishments can find a direction to move forward on
a coherent and planned approach to assuring and improving quality.

I imagine that no-one in education would quarrel with the assertion that everyone within
the organisation should be involved in the drive for quality. Students attend or interact
with colleges in a variety of ways and have a complex range of expectations and needs
which they require the college to fulfil; these may range from the achievement of defined
learning outcomes, to personal development, social satisfaction, counselling and guidance
and will undoubtedly include the expectation of courteous and efficient service in all of
their dealings with the college. Clearly, everyone employed by the college will affect the
quality of the student's experience.

Neither is there initially any difficulty with the notion of "routinely doing things right".
At the first level of understanding this expresses a commitment to delivering each part
of our service on a right first and every time basis. Anything less would imply an
acceptance of sloppy work and inferior standards.

However the problem arises when we examine the notion of the meaning of right in
education and training. Our core technology, the enabling of learning, is a complex and
interactive process and a simple exhortation to get things right begs the fundamental
question - What is right in education and training?

There are numerous and frequently changing interpretations of what constitutes a quality
learning experience; these may be subject to political dogma, changing fashions in
education technology and pressures from stake holders other than the learner. A tempting
solution to this dilemma would be to limit our quality assurance to those elements of our
service which would be easier to describe and quantify; it is right and laudable that such
aspects of our service are included in any quality improvement process but to ignore the
fundamental issue of the quality of learning would be to relegate quality to the periphery
of our activities and would fail to capture the imagination of our teaching staff.

Given that learning is an interactive process and that, in the literature of Total Quality
Management, quality is seen as meeting the requirements and expectations of the
customer, a way forward becomes clear - we consult those most intimately involved, the
learners. By a process of negotiation between the suppliers (the staff of the college) and
the customers (the students) we can arrive at an explicit and realistic set of quality
requirements that we can guarantee to achieve.

But unlike other service industries, the quality of our "product", the learning outcome,
is not wholly within our control. We can only control our processes. The learner's
contribution is vital and therefore should form part of the stated quality requirements.
Thus, by negotiation we will arrive at a quality contract or a quality entitlement owned
by the learner and the college. This contract will guarantee certain services, processes
and behaviours so that the student and the college staff will fully understand what to

PERSPECTIVES ON QUALITY IN ORGANISATIONS

expect from each other. The aspiration would be negotiated quality requirements with
each learner but at first it may only be possible to arrive at negotiated quality
requirements for groups of students.

Thus the quality we assure is realistic within resources, meets our customer's reasonable
expectations, is owned by the customer and the supplier and is achievable. By stating
specific requirements we can measure achievement and plan to improve on any
shortcomings.

Quality is performing to requirements each and every time.
PLANNING TO ACHIEVE QUALITY

Total Quality Management is concerned with people, systems and culture.

It is a truism that people are any organisation's most valuable resource but, in the
context of quality improvement and assurance in education and training, it is worth
repeating. Management's task within TQM is to create a situation in which the
commitment, flair and experience of the staff can be released and guIUed to meet the
agreed requirements of the customer. It is essential that senior managers themselves are
trained in the philosophy and practices of TQM prior to committing to a programme of
quality improvement. The genuine involvement of all in negotiating quality requirements,
planning to achieve those requirements, identifying problems and offering and
implementing solutions will be exciting but challenging and it's all too easy to retreat into
hierarchical and authoritarian behaviour when the rhetoric becomes the reality of genuine
involvement and problem identification. Management commitment must be visible,
genuine and ongoing.

Management commitment is demonstrated by;

• generating the quality policy through consultation

• publicising the quality policy

• allocating money and time to the quality process

• attending quality team meetings

• talking with staff both informally and formally about quality

• modelling the quality culture

• valuing and respecting people (all of them, all of the time)

• insisting on clear quality requirements and documented procedures

At the beginning of the quality improvement process, managers will need to win their
staff's commitment to, involvement with and awareness of quality by providing an
opportunity for them to agree and make explicit the shared values of the organisation.
Secondly, they will need to set up systems for team working and open communication.
The notion that quality is achieved by the requirements of the customer is equally
applicable to customer/supplier relationships within an institution as it is to the external
customer and supplier. Therefore teams will be most effective if based on these internal
customer/supplier relationships. The teams will be able to work together to agree quality
requirements for specific services or functions, plan the work processes to ensure that
these requirements are met, identify where errors or problems are occurring, devise and
suggest solutions to these problems and act to eliminate the error. They will need to feel
secure that this work will be regarded as positive and commendable and to know that
there are formal and safe channels of communication by which problems and solutions
can be communicated to line and whole college managers and decision making bodies,
Having established these quality improvement teams and channels of communication,

TQM IN EDUCATION AND TRAINING

managers will need to ensure that the team members have been trained in the skills and
techniques of work process analysis, problem identification and solving and collaborative
group working. It will also be advantageous to establish a team of staff who have an
overarching leadership, advice and support function in the quality improvement process.
The members of this team will need to be representative of all of the working groups and
management levels within the college and will need in depth training in quality
philosophy, systems and techniques. In support of these activities there will be a need
to provide systems to document defined quality requirements and working procedures.

Expert advice and training on assurance and monitoring systems should be sought.
Wherever possible these functions should be in the control of the quality improvement
teams but with some impartial auditing. It is self evident, but sometimes overlooked, that
any audit process must involve customer consultation on their level of satisfaction on the
achievement of quality requirements. The quality loop will then ensure that corrective
action is taken where a failure to meet a requirement has been identified.

An essential prerequisite for success in the quality improvement programme is the
creation of an organisational culture that is supportive of the aims and processes of total
quality. The first step for this will have been taken by involving all staff in the
identification and expression of shared organisational values. There are some cultural
characteristics which are essential to enable the process to succeed. Perhaps most vital
of these is a real commitment to a no blame culture. Open communication is also a vital
ingredient with everyone believing that a genuine attempt will be made to understand and
to trust the intentions of the communicators. The culture will also be one in which it is
expected that people will act; education is all too prone to analysing problems and
spending an rnordinate amount of time and energy on finding the intellectually and
ideologically pure solution only to move on to the next problem before corrective action
has been taken. Tnus at all levels there will need to be a commitment to act, or at the
very least to respond even if only to explain why a suggested solution cannot be
implemented, within a specified and short time scale.

Thus a total quality culture will have: shared values, commitment to always getting it
right, open and explicit communication, time for teamwork, training in quality, total
involvement, sensitivity to others' needs.

ONE COLLEGE'S EXPERIENCE OF IMPLEMENTING TOTAL
QUALITY

Background Information

Mid Cornwall and Saltash Colleges had, supported by a county wide project, developed
systems and expertise in some aspects of quality assurance through a process of customer
consultation based on student questionnaires adapted by course teams from work
undertaken in Avon. The students were consulted, in detail, on their satisfaction with
access arrangements (everything up to settling into the course), on the quality of their
learning and assessment experiences and the validity of their course for the next stage of
their lives (further or higher education or employment). Employers and/or further and
higher education would also be consulted at the validity stage. The information gained
from the questionnaires was used by the course team in their course review processes and
as an aid to action planning for improvement.

A number of full time courses had implemented this approach and the desire was to
spread the process to all full and part time courses.

PERSPECTIVES ON QUALITY IN ORGANISATIONS

The college management team had consciously decided not to seek course evaluation
feedback from the customer consultation processes as they wished, at least in the pilot
phase* to ensure that course teams owned the process and the information and that staff
did not feel threatened or defensive.

However, after the first year of the project, the course teams themselves felt that they
would like to have clear ways of communicating information and requests to
management at least about those issues which could be resolved by management or whole
institution decision making groups.

There was also a feeling by key members of the management team that the time was
right to implement a whole college quality system both in response to an internal desire
to do so and in response to external pressures. The College wished to develop and
implement a quality approach which matched its culture rather than adopt a prescribed
quality management system such as ISO 9000. They were attracted by what they had
learned of TQM and by the notion of continuous, or incremental quality improvement.

Some funding was available from a Regional Department of Employment project.

Beginning the Process

The College's management team decided to use a consultant to assist them with the initial
stages of investigating TQM, to provide some management training in TQM and to help
facilitate the introduction of their chosen approach to the whole staff of the College.

The first activity was a development day for the College management team in which
they explored the essential concepts of TQM and used TQM techniques to generate a
draft College Quality Policy and draft • ;ollege Quality Aims.

The second phase was to consult the College Governors who had a crucial role in
management commitment and an evening session for managers and governors was held
in which a presentation on quality systems in general and in education was made. The
focus was on TQM as providing a framework which fitted College culture, which could
if required embrace ISO 9000 and which could build on the work undertaken by the
course teams in integrating regular and rigorous customer consultation into course review
processes. During structured discussions, following the presentation, governors supported
the approach that was recommended. Additionally a high degree of experience in
industrial applications of TQM was identified and their assistance was enlisted with the
next phase of the programme.

The third phase was to launch the TQM programme with the whole College staff. An
afternoon and evening session was organised to which all full time staff and part time
employees who worked more than ten hours per week were invited. Management
commitment to the programme was effectively demonstrated by the large investment of
resources that such an event entailed, by the closing of the College for the afternoon and
by the high profile that they took during the proceedings.

Presentations by two College Governors with industrial experience of TQM, by the
consultant and by the College Principal and Vice Principal (Development) were made and
questions sought. Workshop sessions followed in which staff were asked to comment on
the draft Quality Policy and the draft Quality Aims and to identify key areas within the
College that should be the subject of the first phase of the quality improvement
programme.

Following this session with all staff the suggestions for areas for quality improvement
were circulated and staff were invited to volunteer to be involved in the work. The
College Quality Policy and Aims were approved by the Board of Governors and
circulated to all staff.

TQM IN EDUCATION AND TRAINING

A structure for teams was agreed and established with the overarching role being taken
by a Quality Steering Group and with four Quality Improvement Groups to address the
four areas for whole College improvement that had been identified. It was decided that
a regular Quality Newsletter would be produced and distributed to staff.

The College also devised and administered a Quality Audit in the form of an attitude
survey with ali staff. It is intended to use the results of the survey to inform management
but also as a benchmark against which the results of the whole College Quality
Improvements Groups' work can be measured.

Training in some of the tools and techniques of TQM and in group working was
undertaken by the College management team. They have begun to cascade this training
to members of the Quality Improvement Groups.

Some Outcomes from the Programme

The first action, the senior management development day, took place in October 1991.
Identifiable outcomes to date include:

• A College Quality Policy

• College Quality Aims

• Four teams to address quality improvement issues that relate to the whole
College and which were identified as priorities at the whole staff event:
Communication, Management, Accommodation, Course Evaluation and
Review

• A Quality Steering Group

• A staff attitude survey, the results of which have been communicated to all
staff

• Extension of the customer consultation and course review processes with the
intention of spreading this to cover all College courses

• The Quality Improvement Group for Communication has recommended that
details from management meetings should be more widely publicised and that
senior staff should make themselves more accessible. As a result the
Principal and Vice Principals will attend meetings by invitation and will be
available for informal contact. Agendas and summary notes of Heads of
Department Meetings are posted on notice boards and widely circulated to aid
communication and accurate and timely information flow

• Four editions of the Quality Newsletter have been published

• The Quality Improvement Group for Accommodation is reviewing computer
programmes for time tabling and room use. A system for reporting and
rectifying "minor" repairs/replacements h?s been devised and implemented

• The Quality Improvement Group for Management has reviewed the College
committee structure and has prepared recommendations for a more effective,
efficient and accountable use of committees

• The Quality Improvement Group for Course Evaluation and Review has
received the results of an audit devised by the Curriculum Committee and is
using these to evolve a systematic approach for the whole College

• Senior staff have been trained in aspects of quality improvement

• A large number of College staff have been consulted and involved. Some are
taking an active part in whole College quality improvement initiatives

PERSPECTIVES ON QUALITY IN ORGANISATIONS

The Governors have approved a Mission Statement for the College and
strategic aims for the College for the next five years. These have been much
influenced by the information gained and the culture established from the
quality initiatives

The College has discussed the Investors in People programme with DCTEC
and has registered an interest in an initial assessment visit

6 Performance management in a
competence framework

Elaine Sauve, TDA Consulting Group Ltd, Brentford, Middlesex, TW8 OHF, UK
INTRODUCTION

In the current economic climate and with 1992 and the open European market upon us,
companies are becoming increasingly aware of the importance of investing in training.
It is, however, recognised that training per se is not always a good thing. Training needs
to be targeted. Corporate goals are identified and the training required for individuals to
help the company meet those business goals can be designed and introduced.

Tliis approach has the advantage of being cost effective in helping the company meet
its objectives but is also motivating for individual staff who feel that their training
requirements are being met, helping them to cany our their jobs more effectively, whilst
also recognising their current skills.

Nationally there is a thrust towards the development of standards of competence in all
occupational sectors and the introduction of training programmes based on these national
standards. This may or may not be linked to the acquisition of National Vocational
Qualifications (NVQs).

Most companies appear to accept this approach for craft and technician grades and for
clerical and administrative support staff. There is a tendency to feel that it is not possible
to identify standards of competence for managerial and professional level staff. This is
at best a questionable premise, probably based on a misconception of the definition of
"competence". A competence-based approach should not merely be a list of tasks but
should reflect broader issues such as task management, health and safety, and the ability
to deal with organisational environments, relationships with other people and unexpected
events (NCVQ 1991). It is therefore not only possible to identify occupational
competence for managers, it is positively desirable in order to ensure targeted, cost
effective training.

It is probable that some of the resistance to the "competence" model is due to the
intimidating amount of jargon which is associated with it. Some of the jargon associated
with NVQs includes: competence, standards, unit, element, performance criteria and range
statement. Introducing these specialised terms into a company training and development
programme can be intimidating for all concerned, be they line manager, trainer or
employee undergoing training. I will demonstrate here the relevance of competence in
a performance management system and the relationship between established management
terminology and "competence" terminology. A performance management model will be
described, which is generic but which can be tailored by individual organisations and has
been introduced in a number of companies as a key contribution to their total quality
approach.

WHAT IS COMPETENCE?

Some of the concerns about the introduction of NVQs, particularly from managers, is a
perception that "competence" refers to the possession of practical or manual skills. This
was perhaps reinforced by the early accreditation of NVQs, many of which were largely
associated with practical skills. In fact the NCVQ definition of competence, whilst it
certainly does recognise the need for the performance of tasks, is much broader. An
individual must also possess the relevant knowledge and understanding to perform

PERSPECTIVES ON QUALITY IN ORGANISATIONS

effectively in the work situation.

Competence then, in NVQ terminology, is much broader than is often perceived and
applies from trainee to senior management.

At this stage it will be useful to give some broad definitions of NVQ terms, before
discussing the performance management model and why performance management is an
important factor in the establishment of total quality.

NVQ Definitions

• Unit of Competence - discrete work area

• Element of Competence - describes what a trainee should be able to do

• Performance Criteria - specify the standards of performance

• Range Statement - details of the range of applications or contexts across which
the performance criteria apply

• Essential Knowledge - underpins performance

WHAT IS PERFORMANCE MANAGEMENT?

There are probably as many different performance management models as there are
definitions of "competence". Probably the most familiar one is the annual appraisal
system. Often the appraisal is based on the subjective judgement of the individual line
manager, who considers personal attributes such as: contribution to the team, co-
operation, attitude, effort and promotability.
The Performance Management Model described here differs in that it is:

• an on-going cyclical model rather than an annual interview

• based on clear objectives, agreed between job holder and line management

• based on continuous review of progress

• not dependent on an annual performance review

The model is essentially a three stage cyclical model - see figure 6.1 .
STAGE ONE: JOB CLARIFICATION

This stage involves line manager and job holder working together to agree the objectives
of the job and identify areas of training and development required to enable the job
holder to meet the objectives. From this a personal development plan can be produced,
detailing:

• clear objectives

• training and development requirements

• agreed timescales

• agreed monitoring and review process

STAGE TWO: MONITORING AND REVIEW

This is an essential part of the performance review cycle and involves the lino manager
in taking responsibility for reviewing progress against the agreed objectives and providing
constructive feedback. There may be a requirement to re-negotiate the personal
development plan at this stage.

MANAGEMENT IN A COMPETENCE FRAMEWORK

S l Alih ON K

JOB CLARIFICATION

job Roll
Elements of Competence
(Objectives)

ST AC J K THRKK

ANNUAL
PERFORMANCE
REVIEW

st\c;f. two

MONITORING

(Assosiment)

Figure 6.1 The performance review cycle

STAGE THREE: ANNUAL PERFORMANCE REVIEW

This may be seen as part of the on-going review process or it may be that the
organisation's practice requires an appraisal interview. Either way, the "annual review"
should present no unpleasant surprises to the candidate. He/she will be aware of progress
and of steps which need to be taken to meet objectives.

In describing the performance management model, terms which will be familiar to
human resource and training managers have been used. It is our belief that there is no
conflict between this model and the competence framework and below we suggest some
broad "equivalences" of terminology. We recognise that this is probably an over
simplication. Nonetheless we believe it provides a platform for employers to begin to
relate to the national jargon attached to competence terminology.

TERMINOLOGY

COMPETENCE PERFORMANCE MANAGEMENT

Functional Analysis
Key Job Role
Units of Competence
Elements of Competence
Performance Criteria
Outcome Based
Assessment

Job Clarification

Job Purpose Statement

Key Results Areas

Objectives

Success Criteria

Outcome Based

Monitoring

PERSPECTIVES ON QUALITY IN ORGANISATIONS

WHAT ARE THE ADVANTAGES OF A PERFORMANCE
REVIEW CYCLE?

It is suggested that advantages for the organisation are:

♦ Corporate objectives are clarified and publicised

♦ People are working towards the achievement of corporate goals

♦ Overall performance is raised

♦ Identification of resources required to achieve objectives takes place

♦ Means for monitoring progress are provided

♦ Training and development needs are identified

Advantages for the individual are:

♦ Clarification of the job, purpose and goals

♦ Increased motivation

♦ A basis for a two-way discussion is formed

♦ A basis for monitoring progress and formative feedback is provided

♦ Resource requirements and constraints are identified

♦ Development needs are identified

♦ Links may be made with National Standards of Competence and Nationally
recognised qualifications

THE LINK WITH TOTAL QUALITY

More and more companies are beginning to investigate "Total Quality Management" as
a vehicle to increase organisational efficiency and effectiveness. There are a number of
TQM models and initiatives, but most stress the need for: clear corporate goals and
objectives: clear individual goals and objectives linked to the corporate goals: investment
in relevant training and development; on -going staff monitoring (assessment) against
agreed objectives.

A recent initiative which makes explicit these features is * Investors in People'.

An Investor in People:

♦ Makes a public commitment from the top to develop all employees to achieve its
business objectives

♦ Regularly reviews the training and development needs of all employees

♦ Takes action to train and develop individuals on recruitment and throughout their
employment

♦ Evaluates the investment in training and development to assess achievement and
improve future effectiveness

The Confederation of British Industry (1991) published Weld Class Targets, which
outlined a series of progressive achievement targets for individuals and organisations,
which must be attained if Britain is to maintain a competitive edge in world markets.
The introduction of a structured performance management system will help with that task.

MANAGEMENT IN A COMPETENCE FRAMEWORK

References

NCVQ (1991) Guide to National Vocational Qualifications National Council for
Vocational Qualifications March

CBI (1991) World Class Targets Confederation of British Industry

7 Quality in further education: an
unchanging agenda

David Shepherdson, West Sussex Advisory and Inspection Service, Chichester, West
Sussex, UK

INTRODUCTION

"Politicians' time-scales are always much shorter than anyone else's; there are always new
agendas for LEAs and we are in a state of perpetual change in education. However, one
agenda will never change, and that is to provide the best educational opportunities for all
... and that is what we must do" (Sir Roy Harding 1990).

The focus here is on the unchanging agenda of quality development from a Local
Education Authority (LEA) perspective. The West Sussex Initiative is based on the
author's experience of working with the LEA quality development team in Birmingham,
and it is primarily concerned with developing Total Quality Management (TQM) systems
in all four Further Education Colleges in West Sussex. It is concerned with a belief in
the self -evaluating professional working within the self-evaluating institution. The
Advisory and Inspection Service is supporting the development of systematic training in
monitoring and evaluation techniques for all college staff, through a programme of
workshops. The process is designed to equip whole institutions to achieve "self -critical
academic maturity 11 within a specified timescale.

THE INITIATIVE

The overall aim of the initiative is to link the demands for accountability with the desire
for improvement; often they are seen to be mutually incompatible. Improving can be a
statistical exercise, fulfilling requirements in terms of performance indicators but, when
linked to the qualitative process of developing a narrative related to classroom practice,
the two processes can be both developmental and powerful, A keynote of the Initiative
is to gain the commitment of identified teams of staff in FE colleges to the idea of
monitoring and evaluation as a positive and educative process in itself. Too often, quality
is associated only with audit, review and inspection. If quality is genuinely in-built and
integral to the whole institution, the integrity of the process may eliminate the
fault-finding, problem-solving focus of many quality systems.

The identified teams are asked to work through 20 tasks in the handbook over one year.
This is supported by systematic staff development and accredited training in evaluation
techniques for team leaders. The teams identify 'quality gaps' in the area of provision
for which they are responsible. These 'gaps 4 are then converted to quality targets that
arc tackled by the team itself. Notionally each team will identify 3 targets to be met each
term. In order to address these targets, the team formulates an action plan identifying
what is to be tackled, by whom, how and where. This provides the basis for a bid to a
central college group (the Curriculum Review and Development Group) which will then
approve or modify the bid for funding. The funding may take the form of release time,
equipment or materials. This forms the basis of reciprocal contracting - where the team
contracts to deliver on its own targets and the Review Group contracts to provide support
for the team to meet its targets. The Senior management team would contract to deliver
on its own targets in the same way as all other teams.

QUALITY IN FURTHER EDUCATION

All targets and timescales are 'public knowledge* and conform to the College strategic
framework for quality. This means that the College Development Plan can be enacted
through the contracting process and the regular review of progress each term.

IMPLEMENTING THE INITIATIVE

The first stage is to agree some key principles that provide the focus of improvement.
Many teams include classroom practice, the quality of teaching and learning or the
interaction between lecture and students. These concepts need to be dis -aggregated into
a series of developmental tasks. At this stage, it is useful to begin the process of
identifying critical success factors or criteria for making judgements about the success of
the improvement process. Although a necessarily lengthy process, it is essential to ensure
a commitment through ownership - the team decides what the issues are, and what the
criteria for success will be. The accredited training programme provides the necessary
support for key individuals to lead the programme as 'professional evaluation*.

It is important that the central group that receives bids from teams undertakes a
monitoring and evaluation role for the whole college. This Curriculum Review and
Development Group is normally representative of all levels of staffing and operates
rotation of membership to ensure full representation over several years. The CRDG remit
is based on the College Development Plan and Quality Development strategy for the
institution. Team progress is monitored by brief half-termly reporting and end of term
reviews.

The reciprocal contracting of the TQM system extends to learner contracts as an integral
part of "entitlement". This replicates a work contract for students and would include
enhanced opportunities for learner support and study skills programmes.

METHODOLOGY OF THE INITIATIVE

The methodology of the initiative is based on the work of David Kolb and D A Schon.
This model of experiential learning and the reflective practitioner, exemplified through
the research and development function, is not usually part of the FE College system. If
time for 'reflection' is built into the teaching and learning programme, it becomes
possible to identify needs and meet planning requirements within "normal practice". The
teams therefore use a variety of monitoring and evaluation techniques, each suited to its
own purpose. The widespread use of questionnaires is generally found to be of limited
use. Both students and staff can become "anaesthetised" to their use and the quality of
some information is doubtful. By using a variety of methods, it is possible to triangulate
on issues, and gain a much closer insight into the origins and nature of the issues. Many
of the so-called problems identified by some quality systems are products of the system
itself rather than being symptomatic of real underlying course culture.

Evaluation techniques can be built into learning programmes, using tutorials, seminars
and workshops. The common complaint is that "we don't have time ... (for quality)."
Teaching hours are being reduced and many staff are under considerable stress as a direct
result of increased administration. A possible solution lies in revisiting teaching
methodologies, timetabling and room allocations. It becomes necessary for the team to
have increasing ownership of the means to deliver the curriculum that meet their
identified needs and the needs of the students more effectively.

PERSPECTIVES ON QUALITY IN ORGANISATIONS

Traditionally colleges have been driven by SSRs and the requirements of resource
allocations based on student numbers. Increasingly, the notion of performance related
funding may produce a more efficient and effective way of delivering learning
programmes that will incorporate "self -critical academic maturity 1 '. These are the issues
that can be picked up very quickly by the 'empowered* course team. Their initial
solutions to perceived problems may be crude, but they offer a genuine R and D function
that many commercial operations value highly. The emphasis on discrete elements of
college operations eg. marketing, recruitment, quality, tends to fragment and dispel
responses such as - "its someone else's responsibility". You cannot delegate or
subcontract quality and the fundamental approach of TQM lies in a set of values and
attitudes that embody commitment to continuous improvement. In the same way, quality
is not a project, nor is it a standard to be achieved. This serves only to limit achievement
and leads to a "lip-service 1 ' model of the way that things should be done.

QUALITY IMPROVEMENT

The "steps" to quality improvement begin with a view about "what do we mean by
quality". Programme Area teams may have a different view to that of the management
teams; providing interesting perspectives on the college's view of quality. Agreeing
common principles is part of the first reflective stage. It is based primarily on experience
{as in the Kolb model), and may be a function of a series of critical incidents, concerns,
problems, good ideas or hunches.

These form the basis of the keynote principles to be outlined in the college Quality
Development Action Plan, that outlines the "mission" and incorporates commitment to
specific quality attributes.

Secondly, decisions about the composition of teams will provide a basis for identifying
areas of "common effort". In itself an interesting exercise, this can identify the "black
holes" of provision and the "team that never meets" or the "team that never was". Team
forming provides a basis for m:>re collaborative working and for improved
communications and shared resources. Indeed, the benefits of this single exercise often
promote a more integrative approach to all aspects of the college provision and lead to
consideration of structured responses (for example, in relation to modularisation).

The third stage is for each team to identify their own response to "quality gaps", as
perceived by the team. These gaps should become targets for improvement, recognising
that in some cases there are resource implications for management attention. Initially,
it is useful to identify smaller (incremental) issues for improvement, rather than medium
or long term targets. If all teams engage in the process, by contracting to deliver, the
whole college can move rapidly to a stage of critical mass. At this point the management
strategy should incorporate a moderating influence, ensuring a degree of realism and
consistency in both targeting and methodology. This is important to ensure that diversity
of approach does not preclude a corporateness of purpose.

It would be possible for a system to be "bought in" - but this does not generally
engender ownership or indeed, commitment by all. Most alternatives are superstructures
rather than infrastructures - built on, rather than built in. It is essential to grow the
system within the organisation rather than impose a structure for accountability; this is
because the reaction to the exogenous system is often to subvert or at least to ignore it.
Systems for accountability can often be counter productive, generating whole sub-systems
and consuming large amounts of time and energy.

QUALITY IN FURTHER EDUCATION

CONCLUSIONS

The advantages of TQM and the "home grown" model of quality improvement mean that
ownership, empowerment and commitment go hand-in-hand, leading to real progress
towards commonly agreed goals. There is no reason why this should not meet external
demands for accountability and, additionally, provide the basis for systematic
improvement.

The real benefits are usually in terms of creating genuine enthusiasm and motivation;
creating some space for reflecting on what business we really are in, and finally creating
the basis for a continuous quest to provide the very best learning opportunities for the
benefit of the students.

There are several key points that are embodied within the West Sussex Initiative. The
unchanging agenda is concerned with continuous improvement with the aim of improving
the quality of the learning experience, increasing participation in training and education
and improving attainment. Above ail, quality is judged by the customer and requires us
to deliver consistently above expectations. In this way it is quite feasible that educational
ideals and economic realities are not necessarily mutually exclusive.

8 Defining quality in higher education -
the stakeholder approach

Alison Burrows and Lee Harvey, University of Central England at Birmingham,
Birmingham, B42 2SU, UK

QUALITY IN HIGHER EDUCATION - THE LAST DECADE

During the last decade qualify has become an important issue in higher education for two
separate, but related, reasons. In the first place there has been ~ drive for greater
accountability by the government for the way in which higher education spends the funds
allocated to it. Society is no longer prepared to take on trust that higher education is
providing value for money. Higher Education is being asked to demonstrate that it is
spending public money efficiently and effectively. This parallels approaches to other
public services, such as Health (Cave et al 1990).

In part, this demand for greater public accountability came from a sense that higher
education had not been responding effectively to the needs of a modern industrial society.
So, in the 1985 Green Paper "The Development of Higher Education into the 1990s" we
find:

"there is continuing concern that higher education does not respond sufficiently to
changing needs. This may be due in part to disincentives to change within higher
education, including over-dependence on public funding, and to failures in communication
between employers and institutions"
and:

"The Government believes that it is vital for our higher education to contribute more
effectively to the improvement of the performance of the economy" (DES 1985).

The government's commitment to ensuring greater public accountability for public
expenditure on higher education has continued throughout the 1980s. Its stance is
demonstrated by the following extract from a letter from the Secretary of State to the
Head of the PCFC:

"I shall however expect to see two key features. Hie first is a means of specifying clearly
what polytechnics and colleges are expected to provide in return for public funds. The
second is a systematic method of monitoring institutional performance. I attach particular
importance to the latter, since without measures of performance, the Council will have the
means neither of satisfying itself that institutions are providing what has been promised
at acceptable quality, nor of making comparative assessments of institutions as a basis for
future allocations of grant" (Baker 1988).

This drive for greater accountability has not been restricted to the United Kingdom but
should be viewed as part of an international trend. Similar developments have occurred
in many other countries (Neave 1986; Goedegebuure et al 1990; Teather 1990).

The second important development that has influenced the quality debate in higher
education in the United Kingdom has been the move away from an elitist system of
higher education towards one of mass participation without a corresponding increase in
finances. This reduction in the unit of resource in higher education and the increase in
numbers entering the higher education system, many without the traditional entry
requirements, has led some to argue that quality must inevitably suffer.

QUALITY AND THE STAKEHOLDER APPROACH

The debate has gained added impetus through the publication in 1991 of the White
Paper 'Higher Education - A New Framework 1 and the subsequent Bill on Further and
Higher Education which was enacted in March 1992. The new law heralded significant
changes in the structure of higher education in the United Kingdom. All of the
polytechnics were granted degree awarding powers and the Council for National
Academic Awards, a major quality assurance body for the polytechnics and colleges
sector, was wound up. A single higher education funding council was established for
England with similar arrangements for Wales, Scotland and Northern Ireland. The new
legislation required the funding council to establish a Quality Assessment Committee and
to "secure that provision is made for assessing the quality of education provided in
institutions ..." (Further and Higher Education Bill 1992). The institutions have been
encouraged by the government to set up their own Quality and Access Organisation to
focus on the audit of quality assurance systems.

DEFINING QUALITY

What is quality in higher education and how do you go about assessing it? Quality is a
concept, like freedom or justice, of which people have intuitive understanding but whose
meaning they find difficult to articulate. Does it mean the same to everybody or do
people use the same word to mean very different things? The Quality in Higher
Education (QHE) project is looking at precisely these issues. The Project was launched
in September 1991. It is sponsored by a partnership of education, government and
business. There are currently 27 sponsors in the United Kingdom including the
Department of Education and Science, the funding councils, accrediting bodies, major
employers, employers' organisations, higher education institutions, research and training
organisations. It is independent of any single organisation or institution but its physical
base is the University of Central England. Each of the sponsors is represented on the
project steering group which meets about once every three months. Sponsors are
expected to take an active part in determining the scope and development of the project.

The ultimate aim of the project is to develop a methodology for assessing quality in
higher education. However, before you can develop techniques for assessing something
you ha\e to be clear what it is that you are trying to assess. The first year of the project
is being spent establishing a clear set of criteria forjudging quality in higher education.

An underlying hypothesis for the first phase of the project is that quality in the context
of higher education is a relative concept. Its definition varies according to:

• which aspect or dimension of the higher education process is the focus of
attention;

• who is making the assessment;

• the purpose for which the assessment is being made.

THE STAKEHOLDER APPROACH

In order to test this hypothesis the research team has identified a range of interest groups
or "stakeholders" in higher education, eight in all: students; employers; 'government*,
subdivided into ministerial departments (DES, DTI, DE); funding councils; teaching staff
in higher education institutions; managerial staff in higher education institutions
(including representative bodies such as CDP, CVCP, and administrative and support
staff); accrediting bodies (including the professions); assessment bodies (HMI, the Audit
Unit and the new Assessment Unit).

5°

46 PERSPECTIVES ON QUALITY IN ORGANISATIONS

Our aim is to find out the views on quality of each of these groups by looking at:

• the focus of interest of each group - is the focus on the input, process or output
of higher education; is it on teaching or research?

• the underlying definitions of quality used by the groups and their political and
philosophical underpinnings.

The stakeholder approach is an analytical tool which other researchers have used to look
at quality in higher education (Vroeijenstijn 1991; Yorke 1991). Primarily, it has been
used in the Netherlands to try and establish a set of performance indicators for assessing
quality which would have wide credibility (Dochy et al 1990). The crucial difference
between this research and our own is that, at this stage, we are not looking at indicators
of quality but at the criteria for judging quality itself. We are trying to find out what
constitutes teaching quality rather than how you might measure that teaching quality is
present (which might involve, for the sake of argument, looking at the degree
classifications of graduates).

To use the stakeholder approach effectively means that there should be no assumptions
about how a particular group might define quality nor about its main focus of interest.
The project does not concentrate specifically on the quality of teaching and learning or
research or institutional quality - anything may be considered relevant.

At the end of the first stage of the project, we aim to have a set of quality criteria
ranked in order of preference, which reflect the views of all the stakeholders. At one end
of the spectrum each stakeholder group might think about quality in a different way and
its focus of attention might also be different. M ^e likely, however, there will be some
convergence and overlap of views on particular issues.

RESEARCH METHODS

A variety of methods is being used to discover the views of the different stakeholders.
These have been combined into three broad groups for practical purposes.

Staff and Students in Higher Education

A questionnaire based survey is being conducted covering sixteen institutions selected
according to territorial, cultural and structural criteria. Within each institution, a
representative stratified sample of students (approximately 400) and random sample of
400 staff (teaching and non -teaching) is being surveyed.

To design the questionnaire, we undertook a literature search to identify the quality
criteria and measures currently in use by bodies such as HMI, PCFC, UFC, CDP, CVCP,
CNAA, BTEC, NATFHE and the AUT. Using the research conducted by the Student
Satisfaction Project at Birmingham Polytechnic (Mazelan et al 1991) and parallel work
by the project team on employers' views, we identified around 300 criteria. These were
distilled down, by merging overlapping criteria, to around 100 different elements.

We then attempted to specify more clearly the "definitions" of quality. "Definition" was
considered too grandiose and so we reverted to the rather less precise "ways of thinking
about" quality. From this we identified nine in total:

• Meeting the requirements of the student.

• Enhancing the knowledge, abilities and skills of students.

• Empowering students to effect changes in their education.

QUALITY AND THE STAKEHOLDER APPROACH

• Fulfilling the stated intention (or mission) of the institution.

• Providing mechanisms to ensure that students get what has been offered.

• Striving for excellence in all aspects of the institution and programmes of study.

• Making efficient and cost effective use of educational resources.

• Checking that standards have been met.

• Providing a distinctive, special and "high class" education.

Staff and students are asked to indicate how important they think the criteria are for
judging quality in higher education and which "way of thinking about quality" seems
most appropriate to them.

Government, Funding Agencies, Accrediting Agencies

This work is ongoing - a review of published material has been conducted which will be
followed up by interview-based verification.

Employers

The following methods were selected and have been used: interviews with organisations
representing employers; a short employer questionnaire; two employer seminars.

Interviews with employer representatives on the steering group took place to help
determine what methods should be used to find out the views of employers about quality
in higher education and the character of the employer questionnaire and seminars. This
source was backed up with two in-depth interviews with organisations representing
employers: the Confederation for British Industry (CBI) and the Council for Industry and
Higher Education.

As a result of the interviews, the research team reached the conclusion that a focused
approach, concentrating specifically on the outputs of higher education relevant to
employers, was most likely to yield results.

Higher education can be viewed &s a process which has two main outputs: an educated
workforce and research (Cave et al 1991). The research team has sought the views of
employers on these outputs in two ways: a short questionnaire on graduate cruitment;
two seminars, one focusing on quality in relation to an educated workforce and the other
on quality in relation to research, consultancy and sponsorship.

The discussions during the two seminars have been drawn together to produce a position
statement on employers' views on quality in higher education and a set of criteria for
judging quality.

The project steering group itself provides another resource as the majority of
stakeholder groups are represented. The project has been able to make use of the group,
via discussions within meetings, depth-interviews with members and contacts with other
key informants.

FUTURE DEVELOPMENT OF THE PROJECT

During the second phase of the project, quality assessment techniques will be tested
within higher education institutions. The criteria identified in the first stage will be used
as a basis for developing quality indicators. A methodology will be developed
incorporating these indicators. Although it is difficult to be clear about the methodology
at this stage, it is likely that the research team will seek the cooperation of institutions
to monitor and evaluate methods which are already being developed and used. In the

PERSPECTIVES ON QUALITY IN ORGANISATIONS

case of methods that are not currently being utilised in UK higher education, a selected
number of institutions will be invited to take part in the testing and evaluation exercise.

It is likely that certain approaches to quality management, such as BS5750 and Total
Quality Management, will also be evaluated. The objective here will be to see whether
these approaches can be successfully adopted for use in the context of higher education.

How successful these techniques are in measuring or managing the quality criteria
identified in phase one of the project will be the focus of the evaluation.

References

Baker K (1988) Letter to the Chairman of the PCFC, 1st November

Cave M, Kogan M and Smith R eds (1990) Output and Performance Measurement in
Government: state of the art Jessica Kingsley London

Cave M, Hanney S. Kogan M (1991) The Use of Performance Indicators in Higher
Education - a critical analysis of developing practice Jessica Kingsley London

DES (1985) The Development of Higher Education into the 1990s Green Paper Cmnd
9524 HMSO

DES (1991) Higher Education: A New Framework White Paper Cm 1541 HMSO
DES (1992) Further and Higher Education Bill

Dochy FJRC, Segers M S R, Wijnen W H F W (1990) Preliminaries to the
Implementation of a Quality Assurance System Based on Management Information and
Performance Indicators in Dochy FJRC. Segers M S R, Wijnen W H F W (eds)
Management Information and Performance Indicators in Higher Education: an
international issue Van Corcum Assesn/Maastricht pp 69-94

Goedegebuure L C J. Maassen P A M and Westcrheijdcn D F (1990) Quality
Assessment in Higher Education in Goedegebuure L C J, Maassen P A M and
Westcrheijdcn D F Peer Review and Performance Indicators - quality assessment in
British and Dutch higher education Utrecht Uitgeverij Lemma B V pp 15-45

Mazelan P. Brannigan C. Green D. Tormay P and O'Shea J (1991) Using Measures of
Student Satisfaction: the implications of a user- led strategy of quality assurance in higher
education Broadcast 18 Winter pp 4-5

Neave G (1986) The All-Seeing Eye of the Prince in Western Europe in Moodie G C
(ed) Standards and Criteria in Higher Education SRHE & NFER/Nelson Guildford
pp 157-170

Teather D (1990) Performance Indicators in Australian Higher Education: the context and
an appraisal of the 1988 report* in Dochy FJRC. Segers M S R. Wijnen W H F W
(eds) Management Information and Performance Indicators in Higher Education: an
international issue Van Corcum Assesn/Maastricht pp 103-118

QUALITY AND THE STAKEHOLDER APPROACH

Vroeijenstijn A I (1991) External Quality Assessment: servant of two masters? Paper
presented at Hong Kong Council for Academic Accreditation conference on Quality-
Assurance in Higher Education 15-17 July Hong Kong

Yorke M (1991) Performance Indicators: obsen'ations on their use in the assurance of
course quality Council for National Academic Awards Project Report 30

Section 2: Implementing Quality in
Organisations

The contributions here are largely in the form of case studies that describe in some detail
how quality approaches have been implemented in specific organisations. Taken together
they represent a significant range and type of organisation.

At one end of the scale, in "TQ implementation - cultural issues and training", we find
implementation in a very large multi-national company described by Brian Hurley, with
its problems of worldwide permeation and scale. John Oakland and Leslie Porter
describe the application of their European model of quality in the context of a single
university in "Developing the European quality model",

Robert Newton, in "Strategic quality management", argues for and takes a more strategic
view of quality management in his description of the implementation of quality
approaches in his Training and Enterprise Council,

The strategies that were employed in quality initiatives in a former polytechnic are
explained by John Heap and Hazel Solomon, who also present a balance sheet of the
"Profits and pitfalls in establishing a qualytechnic". In a significant change of emphasis,
Patti Mazelan and her colleagues focus specifically on the student and describe their
work to elicit views of students that might inform the monitoring of quality in "Student
satisfaction and perceptions of quality".

Douglas Edgar, in "Experiences in the design and conduct of enterprise audits", explores
the methods and strategic approaches that can be taken in the complex, and often
controversial, process of educational auditing. In "Monitoring the quality of quality
control systems" , Henry Ellington and Gavin Ross move into the realms of meta quality
control as they describe and examine systems in their own higher education institution.

9 TQ implementation - cultural issues and
training

Brian Hurley, formerly Quality Implementation Manager, BP Chemicals Ltd
INTRODUCTION

BP Chemicals, one of four main businesses in the BP group, took 20,000 employees, at
90 locations worldwide into a Philip Crosby based total quality process in 1987.

The early successes and lessons highlighted the need to reinforce and recommit
everyone to take full ownership. A team was set up in 1990 to catalyse and focus this
recommitment to quality implementation.

Tne team, led by the author, and assisted by outside training consultants, developed a
new in-house training package and process, built on the Crosby phase, but customised to
BP Chemical's needs. A BP group initiative for overall culture change reinforced the
emphasis on combining hard and soft factors in the integrated training process then used
to launch the BP Chemicals recommitment, which was called "Towards 2000 with Total
Quality".

The training package and processes demanded follow-up into "business as usual" by
requiring the line and senior management role models to lead the change process by
personal involvement in training activities, as the platform for new ways of thinking and
working. This raised issues and barriers, created opportunities and benefits which are
outlined here.

Total quality implementation as the foundation for underpinning a culture focused on
customer satisfaction, and the need to continue our value improvement of all work
processes, is hard work but not optional.

CONTEXT

The BP Group in 1991 comprised the main businesses indicated in Figure 9.1.

Employees

Exploration and Production 10,200

Refining and Marketing 56,000

Chemicals 20.400

Nutrition 20.600

Miscellaneous and Corporate 9.600

Total Worldwide 116.800

Figure 9.1 BP business and employee levels (1991)

BP Chemicals had been a loss making part of BP in the early 1980's but by focusing on
core businesses, and reducing costs and people numbers during 1980 to 1984. we became
profitable. We alone in the BP group embarked on Total Quality (TQ) ui late 1987 to
establish a culture of continuous improvement focused on "Internal" and "Exicrnar
customer satisfaction.

IMPLEMENTING QUALITY IN ORGANISATIONS

We are organised into business areas, cross matrixed with a set of essential functions,
which characterise our company's approach to the way we operate. A snapshot of our
recent business performance and scale for 1989, 1990 and 1991 is indicated in Figure 9.2.

1989

1990

1991

Turnover(£m)

3,503

3,164

3,064

Profit <£m)

540

129

NIL

ROCE (%)

NIL

Capital Employed (£m)

2,143

2,081

2,436

Employees

25,000

20,400

20,000

Locations

Figure 9.2 BP chemicals recent business indicators
STARTING THE TQM JOURNEY

The principal reason for starting TQ was to secure competitiveness by a strategy based
on a vision that we needed to:

• Compete continuously for customers, ie. be preferred suppliers.

• Create a culture of continuous improvement of all work, as a process.

• Improve the participation, job satisfaction and value of all employees.

• Focus on process management

In short, we needed to improve our effectiveness at adding value in a series of chains
both internally and especially across the external supplier and customer interfaces.

Our initial priorities were the internal processes, believing we had to learn ourselves
before offering TQ to our customers, and demanding it from our suppliers.

Our model of TQ embraces three key elements: sysiems assurance guidance, team work,
techniques and tools.

Various parts of our organisation had been pressed to quality assurance (1SO9000) and
some techniques by customer demand. Our thrust in 1987 was based on a training
programme for all employees, using Philip Crosby's Total Quality philosophy and
training package which is founded on the four absolutes of:

• conformance to requirements

• system of prevention

• standard - defect free

• measurement - price of non -conformance.

1 will not dwell on quality assurance (BS5750/ISO9000) or the techniques and tools of
quality except to say they are essential to provide a hard disciplined and systematic way
of applying the philosophy of TQ, to analyse, improve and measure process variability
and capability.

Our experience with the Crosby approach in 1987/1988 was long on philosophy, and
a fine start giving us a common language, in many national tongues; but we had to
develop our own answers to the key question "How to do it?"

TOTAL QUALITY IMPLEMENTATION

RECOMMITMENT

In late 1988 we asked ourselves, "how well are we meeting the requirements?"

Not only did we not have clear answers, but many parts of the organisation had mixed
messages, or they said they had. We conducted a series of workshops asking three basic
questions: "Where are we now?"; "Where do we need to be?"; "What do we have to do
to get there?"

We had learnt some lessons:

♦ Clear demonstration of commitment at the top is essential

♦ TQ implementation can result in stress at all levels

♦ Fear and insecurity can be outcomes

♦ People don't like to admit problems and failures

♦ Problem prevention is not as much fun as problem fixing.

Also we realised the need to focus our quality improvement on a project by project basis.

A major requirement was to take ownership and, to this end, we had started and rapidly
completed a totally new in house customised training package and process. It was
founded on enhancement of the basic Crosby philosophy but added our own examples
and two special new features.

We emphasised the use of good quality assurance and quality tools and techniques to
focus on team projects - the hard issues. We also realised the need to address the main
cultural barriers - the soft issues.

So, the two special features were the need to address culture and training in a TQ
context.

What is culture? We discovered tLat everyone had their own view but no one could
define it in terms suited to a multi-national industrial organisation which covered several
countries each with its own ethnic and social culture. We did produce a model, which
viewed culture as a product of process, structure and behaviour.

We recognised that our recommitment process needed to address the behavioural aspect
in particular.

We used an in-house team, assisted by external training consultants, in a joint project
which developed a comprehensive modular training package.
Our training was based on 3 main premises:

♦ that all work is a process and the horizontal value chain is the focus for teamwork
and leadership.

♦ good prevention systems hold gains by preventing regression.

♦ to continuously improve, people must be engaged to drive an innovative dynamic,
which we called the Four Stage Cycle of Continuous Improvement - the core of
our TQ approach (see Figure 9.3).

CULTURE

TRAINING

IMPLEMENTING QUALITY IN ORGANISATIONS

But the follow-through depends entirely on the line team leaders. To assist in this we
used our own line managers as trainers, which required a huge investment in selection,
training and now support and maintenance. Conflicts of time and line management
participation become important - but we believe these pain barriers are unavoidable if
ownership is to be achieved.

Quality has to become part of "business as usual" to facilitate the changes from
segmental, functional and often vertically focussed people, overburdened by systems, to
a business and work approach based on innovation and external customer requirements.

As we were about to deliver our new package, the BP Group initiative of "Project 90"
gave new emphasis to the key factors which should be addressed to stimulate and get best
effect from TQ: process, structure and behaviour.

We had recognised behaviour in our new training modules but Project 1990 re-
emphasised this in terms of a BP group vision: "The most successful oil company" and
some explicit values related to a variety of stakeholders: employees, suppliers,
community, customers and shareholders.

We had begun restructuring and delayering. On the behaviour side, we have resolved
that our employees and managers must become capable of a number of essential
behaviours: open thinking, personal impact, empowering and networking. These are now
incorporated into our staff appraisal, development, reward and recognition processes.

1. Define requirements

4. Foiiow-up

2. Assess
& plan

3. Implement Improvement

Standards

ADD

Methods

INPUTS

Product

— \

Service

Information

Facilities

Training

OUTPUTS

Product
Service
Information

C
U

T
0
M
E
R

Process management

Prevention not appraisal

Figure 9.3 Cycle of continuous improvement

TOTAL QUALITY IMPLEMENTATION

TRAINING AND CULTURAL ISSUES

Reference has been made to the new training elements addressing behaviour and
leadership. The training process in all of this was deliberately interactive and the Board
were the first to use the new material by setting aside two whole days.

We chose to build the training on well established material, some of which was not new
to many of our managers. They still need reminding of the 110 role models in our
20,000 strong organisation.

The training process was founded on line management creating the need for training,
taking a lead in it and the support function satisfying that need - with good material and
a process for training line managers to be the trainers. The fundamental lesson we got
over about training, was that it is the beginning of a process and team leaders must own
it and follow up if the training effort is to be valuable. In short "behaviour breeds
behaviour".

The many elements that go to provide a TQ culture and the essential behaviours were
identified in our material which was based on a range of theoretical perspectives:

♦ Team roles and group dynamics: leader, worker, achiever, reflector, maintainer.

♦ Teams and action -centred leadership: forming, storming, norming, performing.

♦ What makes good leaders: drive, dedication, competitiveness, honesty, realism and
maturity.

♦ Effective leadership: vision, communication, trust, self-knowledge and
positiveness.

♦ Continuum of style: tells, sells, consults, shares and delegates.

We have made a good start! Four years along our journey we had been reinforced by a
BP group initiative that reinforces the process focus with the need to change behaviour.

So how is it in the real world of a large organisation; coping with recession? Here are
some of the painful issues we continue to address and which require commitment and
participation from the top.

Do the role models really value wealth creation and team work or has short term profit
and self interest aggravated the class barriers to team work? In business terms, have we
replaced "captains" of industry with "pirates"?

1 am grateful to Rosabeth Moss Kanter (1985) for listing some relevant dilemmas and
paradoxes: process needs management, participation by command, top orders middle to
do, empowerment with resources, delegation not abdication, need for a 25 hour day.
participation threatens status, what's in it for me, teams equalise individuals, belief in
team synergy.

The changes we seek will only occur if we all perceive a need for a better world and
a way to get there. We recognise there are behaviour issues which have to be tackled.
Jind top management must lead the way. We can summarise the key forward issues:
behaviour, line management ownership, staff selection, staff appraisal, training, promotion
policy, recognition, continuous education, internal marketing and leadership participation.

CONCLUSION

Is it worth it? That is not the real question; we have no choice. Some of the issues are
national, social and economic ones, but we have to bite the bullet. Total Quality seems
to be the right vehicle. But it is worth it too! Despite most of what I've said about pain
and barriers, there are gains and successes which prove the worth of embarking on TQ

IMPLEMENTING QUALITY IN ORGANISATIONS

as the process for underpinning all enterprises .

Let me finish with some good news, which encourages us to conclude that our journey
so far has been successful. Two hundred and fifty quality line trainers, 90% certified to
ISO 9000, a common language, changing behaviour, own training material/processes,
improved supplier and customer alliances, quality cost savings approximately £120m over
3 years, all bear witness to the progress made.

Finally I mentioned the recession and business performance. You will note that in 1992
we just broke even. We believe that proves the value of embarking on TQ. In the last
downturn of the early 1980's we made huge losses during a less severe recession.

References

Kanter R M (1985) The Change Masters: corporate entrepreneurs at work Unwin
Paperbacks London

10 Developing the European quality model

John S Oakland and Leslie J Porter, University of Bradford Management Centre,
Bradford, UK

INTRODUCTION

The University of Bradford's Management Centre is one of Europe's oldest and largest
business schools. The Management Centre pursues an extensive programme of research
that has earned an international reputation in many fields of study. It is a fully integrated
business school, providing comprehensive and innovative programmes of management
education for undergraduates, postgraduates and post-experience students. The interaction
of students with experienced managers and staff from diverse educational and industrial
backgrounds provides a stimulating and creative environment.

The Management Centre has a matrix structure with seven Professors, responsible for
the subject groups of economics, financial management, international business, marketing,
organisation behaviour, production and operations management, and total quality
management. Non -professorial Programme Chairmen manage the undergraduate*
postgraduate (MBA), doctoral, and executive development (EDP) programmes. There is
also a Director and Assistant Director of the Centre who are both professors. The
Director, Assistant Director, Programme Chairmen, and some Professors also sit on an
"Executive Committee", which is the decision-making body of the Management Centre.

There are approximately eighty full-time academic and related staff, fifty non-academic
staff, five hundred undergraduates, three hundred full-time and part-time MBA students
and thirty doctoral students on site. Almost two thousand executives and managers from
industry, commerce and the public sector pass through the EDP each year on various
types of short course programmes, including the Executive MBA.

fhe establishment of the industry funded European Centre for Total Quality
Management (ECTQM) in 1987 was a major innovation. The ECTQM is actively
involved in research, teaching and advisory work in all areas of quality management. A
wide range of students, including those on executive development programmes,
experience TQM training and education at Bradford. The Centre is a member of EFQM
and the TQM group are involved in many EFQM activities.

In the summer of 1990 it was decided to introduce TQM into the Management Centre
itself using the expertise of the European Centre for TQM. The 1990*s will be a very
challenging phase in the life of the Management Centre. Business schools generally
operate in an increasingly competitive environment and the Management Centre's future
success will depend on the commitment of all its staff to the improvement of its
processes,

AWARENESS AND ORGANISATION FOR TQM

The seven Professors held two TQM strategic planning workshops to examine the
feasibility of introducing TQM into the Management Centre. A new mission statement
was developed and the factors critical to the achievement of the mission were identified.
There were seven critical success factors (CSFs) and associated critical processes were
listed.

A model of TQM, figure 1 0.1 . covering the main areas of the criteria of the European
Quality Award was adopted for implementation at the Manager icnt Centre.

IMPLEMENTING QUALITY IN ORGANISATIONS

Fig'jrc 10.1 The Bradford TQM model

The consensus view was that TQM at the Centre should involve:

• Everyone striving to meet customer requirements - internally and externally.

• Managing business processes.

• Continuous improvement in everything we do.

• Open good communications.

• Participative management style and involvement of everyone.

• Documented, auditable systems for the way we do things.

• Teamwork, built around our processes.

• Training and education in the identification of processes and the tools and
techniques of improvement.

• Empowering people to act wherever improvements can be made.

It was recognised that the necessary culture change will happen slowly and that the
Management Centre's total quality culture would not be built overnight. There was a
strong belief among the Professors that the TQM approach could be used at the Centre
in a disciplined approach to never-ending improvement.

It was also recognised that to devise and implement a TQM process takes considerable
time and dedication and it must be given the status of an executive project. It was also
essentia! that any TQM initiatives be fully integrated into the Management Centre's
operating philosophy and management systems. The Professors suggested to the
Executive Committee that the Centre should embark seriously on a TQM implementation

DEVELOPING THE EUROPEAN QUALITY MODEL

programme, but pointed out that the commitment would be continually questioned and
be weakened, perhaps destroyed, by any failure of the 'senior management* to support
the initiatives.

THE QUALITY COUNCIL

It was decided that a Quality Council (QC) should be formed to guide the TQM process.

Purpose

The main purpose of the QC was to develop the Centre's culture into one of Total
Quality, thereby ensuring that we identify, understand, and achieve our "mission".

Charter and Responsibilities

The members of the Quality Council were drawn from various levels in the organisation.
The Council works within the strategic framework laid down by the Professors and the
Executive Committee. The membership is the Director, two Professors (including the
Professor of TQM as facilitator, and chairperson) two Programme Chairmen, a Lecturer,
a Secretary and a Computer Officer (technician).
Their responsibilities include:

• Updating the Mission Statement

• Identifying the Critical Success Factors

• Providing overall strategic direction on TQM for the Management Centre

• Establishing plans for TQM implementation

• Setting up Process Quality Teams and Quality Action Teams to make
improvements

• Reviewing progress and plans for quality improvement

• Revising plans for the development of TQM and process improvement

The Council acts as a senior problem-solving group. It holds meetings monthly,
following the meeting of the Executive Committee to review quality strategy,
implementation, progress, and improvement.

First Tasks

The QC reviewed the previous mission statement and found it to contain suitable
components for the development of a new mission statement and critical success factors
(CSFs). These were set down as follows, including a list of "Stakeholders":

Mission Statement

To be a growing centre of excellence in teaching and research in the disciplires or
management.

To improve the practice of management, world-wide.

IMPLEMENTING QUALITY IN ORGANISATIONS

Critical Success Factors (CSFs)

1 We must have a UFC research rating of 5.

2 We need a demonstrably excellent reputation for an innovative practical
approach.

3 We must have products and services which meet current and future market needs.

4 We need financial independence.

5 We need an excellent infrastructure.

6 We must have a stimulating and rewarding work environment.

7 We must achieve a critical mass of quality staff.

8 We need quality inputs.

These were deemed to be all necessary and together sufficient, for the Mission to be
achieved. Some are directly measurable (eg UFC research rating) , some are aspirations,
hopes or fears.

Stakeholders

These are: students; staff (academic and others); state; industry/business/public
organisations; professions; university.
Current Aims were derived from the Mission and the CSFs and are as follows:

To achieve a University Funding Council (UFC) research rating of 5 in 1993. To
improve the management processes in the external world by research, teaching and
advisory work.

To be the 'number one* UK business school, in terms of the services and products
offered to targeted markets.

Be a total quality business school which fully involves and develops all employees
and revolves around never-ending improvement of processes to meet customers"
requirements completely.

To empower employees and teams to act in making continuous improvements.

The Quality Council also identified 28 critical processes. These are the activities which
must be carried out especially well in order for the CSFs to be achieved. The Council
then produced a 'process quality matrix* in order to prioritise critical processes for
improvement. This identifies which processes have a high impact on each CSF. Again
the necessary and sufficiency rule applies. The matrix also showed the subjective quality
ranking given by the Council to each process on the scale:

A Excellent Performance

B Good Performance

C Fair Performance

D Bad Performance

E Embryonic Processes

A second matrix showed the results of this work - a plot of the quality of each process
against the number of CSFs. showing the critical processes in most urgent need of
attention.

7* '«

DEVELOPING THE EUROPEAN QUALITY MODEL 63

All 28 processes were grouped under seven headings and these main groupings given
generic titles. Seven Process Quality Teams were set up to manage these processes.

PROCESS QUALITY TEAMS (PQTs)

The PQTs own and manage the process improvements.

Purpose

To define certain business processes and to set them up to run perfectly.

Each PQT Chair will form a team of approximately 8 staff. Their responsibilities will
include:

• Breaking down and describing the assigned critical processes

• Prioritising and selecting processes for improvement

• Setting up quality action teams (QATs)

• Reviewing and supporting QAT activity,

QUALITY ACTION TEAMS (QATs)

The actual detailed improvements will be carried out by QATs.

Purpose

To define and improve any particular process assigned by a PQT.

Figure 10.2 Organisation for TQM

7£"

IMPLEMENTING QUALITY IN ORGANISATIONS

Composition

6-10 members of staff, selected by a Team Leader, representing all those involved in the
assigned process. The team leader will be selected and asked by the PQT to form the
team, preferably using a flow chart of the process. It is hoped that every employee at the
Management Centre will be involved in quality improvement through a QAT (or PQT)
at some point. Their responsibilities will include:

• Drawing a flow chart of the process to identify its customers and suppliers

• Identifying measurement points

• Measuring and comparing results with requirements

• Improving the process and documenting it

This teamwork structure, which is shown in Figure 10.2, should ensure the widest
possible involvement in the TQM process and its effective implementation.

INTRODUCTION AND TRAINING

Everyone took part in a one day workshop to explain what had been done by the Council
and the plans described above. There were several workshops to accommodate all staff,
led by the Professor of TQM.

Further training, on process improvement methods, will be provided to the individual
teams on an "as needed" basis. It was expected, however, that academic staff at a
management centre should be familiar with techniques such as flow charting, force-field
analysis, Pareto analysis, brainstorming, cause and effect analysis, and the use of simple
data presentation methods.

The Quality Council prepared a very brief force-field diagram of the pressures for and
against TQM at the Management Centre (Figure 10.3).

TQM

Pushing For

If we don't do it there will be disaster -

It will affect promotion

Quality Council

Professors committed to TQM -
PQT's have chairs

Share ideas with AUT -

Improves efficiencies and effectiveness ■

Quality Action Teams to be formed -
- wjih equal opportunities

Pushing Against

« : We are doing bits of it anyway

What about promotion?

Existing Management Structure

* AUT will be against

* Current Appraisal Schemes

* Why do extra work?

4 No equal opportunities

Figure 10.3 Force field analysis

DEVELOPING THE EUROPEAN QUALITY MODEL

It was agreed at the senior level that only by measurement would people know when the
Centre had succeeded in its Mission and in meeting its aims. To measure the start point,
a survey of the staff's perceptions of quality at the Management Centre was carried out.
The survey addressed all the key areas of the Bradford TQM model. Many positive
things showed in this survey, eg people taking pride in the Management Centre, but the
survey also highlighted many areas for improvement. Deficiencies in teamwork,
communication, systems and measurement were identified. The most alarming statistic
was that 88% of the respondents did not feel that the senior staff had established a clear
policy and strategy to improve quality. Clearly this illustrated that the ground was fertile
for a TQM initiative.

The TQM training programme commenced with a teamwork workshop for the executive
committee and the Professoriate - see Figure 10.4.

The senior team profile had previously been exposed to the Myers-Briggs Type
Indicator (MBTI) which is a psychological measure that attempts to explain the central
aspects of people's personality, both to the individuals themselves and to their colleagues.
An understanding of the MBTI measures helps to promote effective teamwork. This
profile was reviewed at the workshop together with the TQM concepts and models. The
action plan for implementation of TQM at the Management Centre was also reviewed.

Four TQM workshops have been run to date and 85 members of staff have received the
first stage of their TQM training. The objectives of the workshops were:

♦ to enable staff to share information on their work and associated quality
problems,

♦ to develop staff understanding of the concept of TQM,

♦ to have fun,

♦ to start building more effective teamwork for quality improvement.

Strategic planning
Workshop

TQM strategy

TQM teamwork
Workshop

Staff TQM
Workshops

-^Staff awareness7"X
4 understanding and )
involvement

Figure 10,4 The training programme

? 73

IMPLEMENTING QUALITY IN ORGANISATIONS

The outcome of these workshops was that staff were encouraged to participate in the
TQM process at the individual and team level. A team approach to continuous
improvement around the Management Centre's key processes was launched by the
Quality Council, once a core of staff had been trained. All staff have been asked to do
some work on improvement immediately on an individual basis, using the tools and
techniques of TQM learnt on their workshops.

THE WAY AHEAD

The first phase of the TQM programme was concerned with obtaining the commitment
of the Professoriate and executive committee to the principles of TQM. This commitment
to TQM has been reinforced by the following statement from the Director of Bradford
Management Centre, Professor David Weir:

"We are entering a very challenging competitive phase in the life of the Centre, and we
are committed to creating a quality environment. Our success over the next few years
depends on our ability to liberate the enormous positive energy within the Centre to meet
our customers' requirements. Universities, old and new, all over the UK, Europe and the
World, will have to adopt never-ending quality improvement processes to succeed. We
want to be one of the first Departments to use TQM to gain a clear competitive advantage
over our competitors."

A strategy for implementing TQM was developed at a strategic planning workshop. A
teambuilding workshop was used to weld the Professoriate and executive committee into
an effective senior management team. A Quality Council has been formed to guide the
programme. The commitment to TQM was communicated to all the Management
Centre's staff through a series of TQM workshops. The development of a TQM culture
has started to evolve through the training programme - see Figure 10.4.

How will the necessary improvements be effected in these areas? Some of the harder
tools and techniques of TQM will undoubtedly be called into play. The role of quality
management systems such as ISO 9000 can make a significant contribution to
improvement. The review of existing foimal and informal procedures and the adoption
of best practices is a fundamental step in the TQM process. It is important to produce
a documented system of what is actually done. The danger in all improvement activities
is that people prefer to improve processes first before writing down what is done. This
results in an indisciplined approach and progress is usually not sustained. Measurement
and recording systems required by a good quality system will also result in more effective
operations. Many organisations in the educational sector are considering the contribution
of ISO 9000 to quality improvement. The role of teamwork and the tools o<" TQM will
also make a major contribution to the programme.

The combined effects of good systems and people working in teams using the tools and
techniques of TQM sustain the process of continuous improvement. The process quality
teams and quality action teams will be empowered to drive the improvement process
forward.

Initiating and sustaining a TQM programme in a large and well established academic
institution raises many interesting questions. Hopefully, the TQM initiative at Bradford
is fully integrated into the Management Centre's operating philosophy arid management
systems. The TQM awareness level of all staff has been developed by a systematic
training programme. As the programme evolves, further development of the quality
model may be required. The contribution to quality improvement of the vari ms elements
of the European Quality Model will be evaluated. The ultimate success of the programme

DEVELOPING THE EUROPEAN QUALITY MODEL 67

will depend upon people working effectively together, with the shared values of TQM,
to achieve the shared aims critical to the future of the Bradford Management Centre.

The TQM work will now be extended to the main University and work has started on
workshops for the Vice Chancellor, Registrar, Pro-Vice Chancellors, Deans, and Heads
of Departments,

11 Strategic quality management: if I pull
it up the flagpole will anyone salute?

Robert James Newton, Greater Peterborough Training and Enterprise Council
(GPtec), Peterborough, UK

INTRODUCTION

In recent years there has been a proliferation of articles on organisational development,
very few of which have drawn the essential links between quality, strategy and policy
which this paper addresses. As any reader of Business Week would know some of the
highly proclaimed, quality conscious companies identified by Tom Peters and colleagues
in 1983 have since experienced major problems. As Tregoe and Zimmerman (1989) have
so accurately proclaimed, the problems "did not stem from operational inadequacies but
from strategic deficiencies. These companies failed because their vision of quality was
flawed".

To avoid this strategic trap, into which I expect many more British companies to fall,
this paper advocates the understanding and use of the technique commonly referred to as
Strategic Quality Management (SQM). Nowhere is the use of SQM more appropriate
than in the Training and Enterprise Councils (TECs) who are not only in the constant eye
of their local communities but are under the rigorous scrutiny of various government
agencies. Furthermore in preaching the significance and importance of quality assurance
to local businesses each TEC must be seen to be practising it within its own organisation.

Like Tregoe and Zimmerman, 1 do not wish to criticise the prevailing wisdom regarding
operational excellence and the use of various quality operational systems to bring this
about, but to emphasise that the last thing a company headed in the wrong direction needs
is to get there more efficiently. This paper therefore addresses an apparent imbalance in
the literature in respect of quality debate about how organisations, like those of the
Training and Enterprise Councils, can improve company performance through the project
management triad of strategy, policy and quality.

STRATEGY

As Stephen Connock (1991) has pointed out, Strategic planning processes have increased
in importance in organisations in recent years, so too has the meaningfulness of strategic
notions such as vision and mission. Very rarely, however, does an organisation take
pains to explain its vision of quality and how such a vision determines the
interrelationships of strategy and policy. The particularly interesting thread of values
which weaves its way through this project management triad is ignored in this paper, not
for its lack of priority in quality debate, but because its significance in organisational
development is paramount and is deserving of a paper in its own right.

For Training and Enterprise Councils, the very effective trap which they must avoid at
all costs is to allow the obvious intensity of operational pressure to overwhelm their
strategic vision. In order to preserve the long term, strategic well-being of an
organisation, Training and Enterprise Councils must go beyond the quarter by quarter
operational outlook.

For Greater Peterborough Training and Enterprise Council (GPtec), strategy is the
description of what the organisation is responsible for delivering and how we intend to
determine priorities in the light of available resources and conflicting commitments. The
vision of quality shared (in value terms) by the senior management group of GPtec is that

STRATEGIC QUALITY MANAGEMENT

it is a strategic business management function which will help us, and other companies
with whom we work closely, to change our cultures. In this respect, our SQM
programme began with the senior management team taking responsibility for laying
down objectives in the form of policy and strategy.

The central thrust of GPtec's approach to implementing our total quality policy is,
therefore unequivocally, a strategic one. Continued success of our operation depends as
much on the quality of our strategic thinking and on how well this thinking becomes
imprinted on every decision made within the TEC, as it does on the effectiveness of our
internal day to day operations. At the risk of using definition as explanation, for GPtec
SQM is the process by which we can translate our very clear vision into action. Or put
differently, by the use of SQM techniques we are able to take the strategic direction
formulated by the renior management group and translate this direction into a quality
operation.

In committing ourselves to quality the usually hidden relationship between strategy and
structure becomes manifest - structure follows strategy. Thus, the quality of strategic
decisions must be management's first priority in bringing about effective organisational
structural change. In putting quality of strategy above all other management processes,
GPtec has shifted away from a dominant Civil Service culture of cell-based operation to
a team-driven culture commensurate with the much more business-like framework of the
TEC marketplace. In the development of our SQM organisational culture we have shifted
from a hierarchy based on status towards one based on service, explicitly recognising the
notion of both internal and external customer satisfaction.

IMPLEMENTING SQM

The major lesson within the domain of quality, and in particular the SQM context, is that
organisational restructuring is an integral part of the strategic management process and
cannot and should not be divorced from it if quality - defined simply as meeting customer
requirements, - is to be an overarching strategic aim.

As Miller and Innis (1990) have so lucidly put it. "There is little point in having a well
worked out strategy which commits the service ... to the improvement of quality if this
message is not thoroughly understood throughout the whole organisation" (p 34). As the
senior management group of GPtec therefore, our first essential task was to raise the
awareness of all our staff about the concept of SQM and how we firmly believed that
quality improvements continually derive through staff's commitment to it. To this extent
our guidirg principle in achieving this outcome was to recognise that. "In involving
people in the debate on quality and on strategy, it is therefore important to specify
precisely the nature of the issues, the assumptions behind the approach and type of
response expected from the people being communicated to?" (ibid p 36). For this reason
we chose the communication process of open forum debate.

To contextualise the debate and to give the feeling to staff that the meeting was but the
start of an explicit use of SQM principles we devised a company SQM project called
'MORALE'. To foster understanding that the senior management group were fully
committed to this SQM project and to convey our desire for far more than lip service to
be paid to it. the implications were explained in detail to all staff.

Tne starting point for the presentation was that ownership and genuine empowerment
of staff is. in essence, the major focus of a strategic quality management approach.
However, for this to be not only said but implemented it was pointed out how a SQM
stance to organisational development suggested the use of what we now call a quality
project management system. It is a firm belief of the senior management group of GPtec

IMPLEMENTING QUALITY IN ORGANISATIONS

that such a quality system necessitates the use of modern management techniques and
information technology as major ways of preventing poor performance and assuring
quality of service to all our customers, whether they be internal or external. Furthermore,
this project management structure will avoid the decision taken by so many other
companies, often by default, to maintain an imbalance of executive time and effort spent
between finance and strategy.

The essential purposes of the company Morale project are: to involve ail staff in the
strategic challenge facing all TECs; to commit staff to the view that short term gains are
no substitute, in quality terms, for long term security and getting the TEC firmly
established; to inform staff of strategic decisions made by the senior management group
only after which such a company project can begin; that the largest failure cost of all
occurs when bad quality strategic decisions are made.

Strategic quality management conforms to the precept that the real objective of any
quality assurance project is to get decisions made correctly by all the project team
members involved. The Morale project is GPtec's attempt to implement a quality
assurance programme which runs on the twin engines of strategy and policy at action and
decision making levels, involving all staff. Hopefully, as is the intent behind the project,
we will avoid the sorts of quality problems which beset most organisations namely; lack
of top management support, lack of training, lack of discipline, lack of resources and lack
of time.

The introduction of a project management system as a team managed operational
mechanism, permits the full involvement of all staff with clear understanding that quality
of performance is measured against team determined project objectives which must be
strategically aligned to GPtec's corporate and business plans. Quality of operation cannot
be gauged without reference to strategy and policy - the driving forces of service
delivery.

The significant involvement of each and every member of staff in a project area team
with joint responsibility for writing performance objectives and establishing quality
standards gives the greatest possible sense of ownership that staff can feel. Taking this
one step further and allowing teams to set targets based on an analysis of GPiec's annual
business plan, should also help in improving strategic decision making at operation team
level.

The essential staff training and deveiopment profile, that so depicts any quality
organisation, will emerge from the strategic base of this SQM project system and
hopefully result in the growth of individuals as well as in the growth of the TEC itself.
Future decision making by senior TEC management may thus be influenced in respect
of strategic plan development. Decision making is a process that is dependent upon the
quality of the input as much as the quality of the process itself.

By the use of management auditing it should be possible to measure the effectiveness
of this SQM project - and evaluating vision and mission is a rare phenomenon in British
enterprise today. Pursuing a SQM approach to translating vision into action is about
articulating the long-term human resource goals, core values, key behaviour and
underlying philosophy which derive from, support and complement the business mission
and strategies of GPtec.

This paper suggests that greater weight of emphasis should be given to quality discussion
at a strategic level. When quality, policy and strategy arc brought together in a company
management model, operational quality flows from it. The quality philosophy

CONCLUSION

STRATEGIC QUALITY MANAGEMENT 71

underpining GPtec's SQM model is to practice what we preach. Our quality tenet is: if
it doesn't add value its a waste of time. Our quality slogan is: if it is not in the strategy
we don't do it.

References

Connock S (1991) Human Resource Vision: managing a quality workforce Institute of
Personnel Management

Miller A J and Innis R B (1990) Strategic Quality Management: the strategic
management of a quality further education service Crown Copywright

Smith R A and Tobia P M (1989) Putting a Winning Strategy to Work Simon and
Slater Publications

Tregoe B B and Zimmerman J W (1989) Vision in Action: putting a winning strategy
to work Simon and Schuster New York

* 79

12 Profits and pitfalls in establishing a
qualytechnic

John Heap and Hazel Solomon, Leeds Metropolitan University, Leeds, LSI 3HE t

The University, like other academic institutions, has a sophisticated quality assurance
system for its academic provision. Many of the supporting functions - personnel, finance,
etc - had been provided, at least in terms of policy and structure, by the local authority.
Incorporation of the Polytechnic as a higher education corporation required the institution
to take over and develop many of these services. Initially, attention was focused on
simple operational provision - we needed administrative, personnel and financial systems
in place and the timescale to provide them was very short. The move to encompass such
systems within a quality framework simply resulted from :

• a recognition that the total experience of students was significantly -Influenced by
such systems - in some cases (such as catering) almost as much as it is by the
academic provision

• dissatisfaction with the nature and level of service provision - arising, in part, from
the haste with which they were, of necessity, introduced

• a critical mass of staff who were aware of the wider quality revolution taking
place within industry and commerce.

The Assistant Director of the University who has responsibility for academic quality
called together a number of people who he knew had an interest in, or experience of,
quality matters. He established a quality steering group to evaluate various approaches
to quality assurance and to attempt to identify a direction for the university to improve
the quality of its non-academic services. This group included the deputy chair of the
board of governors, who had recently undertaken an exercise within his own
(manufacturing) company to apply for and receive BS5750 accreditation. The main
debate centred on the difference between procedurally-based approaches (such as BS5750)
and culturally-based approaches such as Total Quality Management (TQM).

At the time, there was no clear drive for quality improvement coming from the senior
management of the institution and it was felt that TQM. in particular, could only be
successful if such a drive existed. Additionally, the steering group was concerned about
the nature and pace of change that the staff of the university had already been subjected
to over the previous two or three years.

Accordingly, it was felt that a procedurally-based approach would :

• allow quality issues to be raised and discussed in a relatively 'hard -edged' manner,
without the need for cultural change or significant resource input

• form the basis for education of the university management and for later
consideration of cultural change to initiate a more broadly- based quality movement

• assist in the process of service definition for many of the still relatively new
systems and procedures.

INTRODUCTION

ESTABLISHING A QUALYTECHNIC

The concept of a university-wide initiative was sold to the senior management team
through a series of informal meetings of the more senior members of the steering group
with the senior university managers and culminated in a seminar for the entire senior
management team of the university. Even after this process, it is fair to say that there
were reservations amongst some members of the management team about the relevance
of industry-based quality approaches to a higher education institution.

It became obvious that certain areas of operation were suitable for immediate
application of a procedural ly-based approach. But, allowing for the fact that most of
those within the institution that had knowledge or experience of quality matters had jobs
that would allow little time for direct involvement, resources were needed to prioritise
and co-ordinate work in different areas.

Short course provision was identified as an area in which some form of quality
accreditation would have immediate marketing benefits and a bid was made to the
PICKUP agency for funding to support the establishment of BS5750 accreditation for the
short course provision. The success of the bid resulted in the appointment of a quality
development manager whose brief was to examine the short course area and to make
recommendations which may result in eventual BS5750 accreditation. Since many of the
supporting activities of short courses are also supporting activities of other parts of
university provision, this actually resulted in a wider consideration of quality generally.
The Quality Development Manager is thus involved in a number of specific projects
within the university initiative.

This may sound straightforward. However, the nature of the debate on quality within
the university meant that there was a some confusion amongst many people (including
senior managers) about the nature o f quality systems and about BS5750 in particular.
There was certainly resistance in some quarters to the idea that existing systems and
procedures could be described as quality systems or procedures and a commonly held
view that any accreditation system must be bureaucratic and cumbersome.

The real brief of the quality development manager was therefore to begin the process
education and of attitude change that would result in positive work towards quality
improvemeni, and then to act as driver for the short course accreditation project and as
facilitator for other projects. This is bast considered in three main phases.

PHASE ONE

The process began with data gathering around the institution, interviewing key managers
and a cross-section of staff to gain a greater understanding of their attitude towards
quality. This sensing and networking was also useful in gaining access to areas for future
work. Taking into account the size of the organisation, a realistic plan was put together
of what could be achieved with estimated timescales - see figure 12.1.

Pitfalls

• Pace and diversity needed to be controlled

• Wider quality debate concerned with the applicability of BS5750 to education

• No quick results likely

• Funding and budgets

74 IMPLEMENTING QUALITY IN ORGANISATIONS

Profits

A genuine concern for quality was evident

Lecturing staff are very conscious of the need for quality

Enthusiasm to see improvements

SHORT TERM

MEDIUM TERM

LONG TERM

To demonstrate results

♦ To raise the profile of the
Quality Initiative

♦ To demonstrate what
effect procedures
can have

♦ Pilot BS5750 in

the Language Centre

♦ Preparation of Quality
manual

BS5750 for short
courses in the
Schools/Faculties

Quality Improvement
Projects

Widespread implementation
Quality Systems

Quality Improvement
Projects

Quality Polytechnic

C-1 2 months 1 -3 years 7-1 0 years

Figure 12.1 Aim and approach

PHASE TWO

Most of the short course co-ordinators were interviewed to gain an understanding of the
current operations and to determine which area would be appropriate for a pilot study.
After careful consideration, the Language Centre was chosen as the pilot area. This
satisfied the criteria of being containable and had a good chance of success due to the
existing systems and enthusiastic management.

At this time it was also considered whether it would be beneficial to use consultants.
It was felt that this would increase the likelihood of success due to their experience of
applying the standard within an educational context.

Pitfalls

• Cost of consultants

• Diversity of approaches for short course provision

• Pressure to increase short courses to generate income

• Limited incentives for staff to run short courses

• Concerns that BS5750 was too bureaucratic or would impose a system.

Profits

• Increased chance of success by using consultants

• Increased chance of success by selecting the pilot area wisely

• Enthusiastic management in the pilot area

ESTABLISHING A QUALYTECHNIC

PHASE THREE

This phase included application of the standard to tailored language courses and
involvement in ongoing quality projects within central services, eg finance, reception and
personnel.

The approach taken for the pilot was to assess current practices in the light of the
requirements of the standard and produce an audit report. All Language Centre staff were
involved in the results of the report and the proposed approach.

Drafting of procedures was then begun with substantial involvement from the Language
Consultant w r ho manages the Language Centre. Once implemented, practices were to be
audited against the procedures and modified.

Pitfalls

• Applying the standard to education

• Labour intensive

• Controlling the pace of all the quality projects whilst providing the support
required.

Profits

• Accepted practices are questioned and improved

• Staff became more involved in making systems more effective

• Transfer of learning for future application of the standard

• Budget was allocated for the quality initiative

PRESENTATION TO MANAGEMENT

To serve as a progress report and to encourage ownership which would be essential for
assessment purposes, a presentation was given to management on the requirements of
BS5750 and progress made. This presentation served to not only achieve the above aims,
but also to encourage a debate around the direction of the quality initiative and the need
to address cultural issues.

CONCLUSION

The strategy adopted to establish a Qualytechnic at Leeds has been successful to date.
The aim was to begin with a systems approach which would lead to a greater
commitment to quality. This has been demonstrated following the presentation to senior
management. The systems approach has also benefited the Language Centre by
questioning and improving current practices. These lessons can now be transferred to
other short course providers.

13 Student satisfaction and perceptions of
quality

P M Mazelan, D M Green, C R Brannigan and P F Tormey, University of
Central England, Birmingham, UK

INTRODUCTION

The purpose of this brief paper is twofold. First, to provide a context in which to view
the emerging importance of student-based information. Second, to briefly describe some
recent research which illustrates how student-based information may be gathered and
interpreted to reveal patterns of student satisfaction. These trends are valuable inputs to
managerial decision-making and may be used to inform innovations in quality policy.

PERCEPTIONS AND EXPECTATIONS OF QUALITY

The quality debate in higher education traditionally focuses on whether profit-orientated
private sector concepts can be readily transferred to the public sector. This is a complex
issue but even the limited view of the student, as a customer or client, goes against the
grain of many in HE. They are very reluctant to empower a group who are traditionally
treated as passive recipients of education.

Both the broad consumer perspective, that service quality can be equated to the extent
to which the perceived level of service delivered matches consumer expectations (eg.
Lewis and Booms 1983) and the more specific view that service satisfaction equals
consumer perception minus consumer expectation, is not without problems in the HE
context. However, the implication that satisfaction will result if perceptions are greater
than expectations has a great deal of intuitive appeal.

This balance between consumers* perceptions and expectations as a measure of service
quality has been fairly widely-discussed (eg. Churchill and Suprenant 1982; McMillan
1986: Parasuraman et al 1986) and appears to be useful. Nevertheless antagonists to the
quality initiative often argue that in the HE context it is futile to attempt to define quality
in this way because student expectations are poorly developed, variable and unpredictable.
Validly measuring student perceptions and expectations is not a simple matter but can be
approached in a systematic way to reveal useful information.

To complicate matters further, antagonists to the quality initiative often point out that
different stakeholders in HE (ie. students, teachers, managers and employers) will all have
different sorts of expectations from a particular institution or service. Consequently they
claim that different stakeholders will have disparate ideas about "quality" jeopardising any
attempt at valid assessment. Clearly there is more than one client group to be considered
in examining the perceived value of higher education, but this does not mean that these
various perceptions cannot be systematically studied.

Despite these conceptual caveats, two things are clear. First, the most important client
group are the "front- line" consumers, the students themselves. Second, and equally
important, it is evident that "quality" (however defined) will become the principal means
by which higher education institutions will attempt to differentiate themselves from their
educational competitors.

STUDENT SATISFACTION AND QUALITY

THE WORK OF THE SSRU

Until recently, students' views of their academic experience had been of interest only to
a handful of educational evaluators but increasingly the value of this information has been
recognised. The realisation that students' attitudes and perceptions are not only
interesting but may play a central role in shaping the decisions of those who manage our
higher education institutions has widespread implications.

Since its inception in 1988, the Student Satisfaction Research Unit (SSRU) at
Birmingham Polytechnic has produced an abundance of information related to student
perceptions of educational quality and their satisfaction with their educational experience.
Our interest in quality and our attempts to measure it through student satisfaction have
three broad aims. First, we are attempting to identify and examine the components of
satisfaction and the components of quality and to consider how these two domains are
interrelated. Second, we are investigating whether there is a common core to students'
perceptions of satisfaction and quality. Third, we are attempting to model student
satisfaction and quality of education and to represent this model in ways which are
designed to inform management decision and policy-making.

Nowadays, collecting information about student satisfaction in educational
establishments is a growth industry. Information of this kind has the potential benefit of
guiding managem :.n decision -making with respect to the planning and provision 0i
educational services. Those HE managers who can clearly identify and subsequently
measure client-centred quality will be able to capitalise on student-based information to
support their claims for resources.

Eliciting Areas of Student Concern

One approach to directly eliciting students' views of the Polytechnic which has proved
to be particularly productive over the last few years is a form of Nominal Group
Technique which we have dubbed the "Group Feedback Strategy" (GFS). Essentially,
this takes the form of a structured participative discussion with groups of students in
order to provoke a span of student-nominated indicators of satisfaction and dissatisfaction.

Experience has shown that engendering an atmosphere of frank and open
communication encourages students to talk freely and honestly about what their education
means to them. These sessions, facilitated by experienced group leaders, generally result
in eliciting a wide and diverse range of satisfiers and dissatisfiers which are used on an
annual basis to update the design of the instruments used to survey student satisfaction
in the Polytechnic. The GFS exercise is used primarily as a device to validate the content
of the questionnaire survey prior to it's annual administration, although the procedure
itself produces some interesting findings. Some results of the GFS session are described
in the next section.

Three factors were used to define the sample frame:
a Faculty - 8 categories in all (Health and Social Sciences, Built environment etc.)
b PFC Classification * 3 categories (Postgraduate, Degree and Other)
c Mode of Attendance - 3 categories (FulUime, Part-time and Sandwich).

16 classes were representatively sampled from this frame and participated in GFS
exercises, enabling the views of 265 students to be obtained.

It is worth noting that the overwhelming majority of ■ tudents welcomed the opportunity
to express their opinions in a confidential climate and were delighted to have some input
into any future decision-making process.

IMPLEMENTING QUALITY IN ORGANISATIONS

Areas of Student Satisfaction and Dissatisfaction

The outcomes of the GFS procedure were consensus, but wide-ranging, lists of satisfiers
and dissatisfiers elicited from students in each of the 16 participating classes. In order
to provide a framework for analysing these unstructured statements, each "good" or "bad"
statement was allocated to one or more of 44 core activities which had been identified

through synthesising categories used in our previous work. These core Ivities are

reasonably exhaustive of students' concerns and are clearly associated with student
satisfaction and dissatisfaction. The 44 core activities themselves represent four broad
areas of student concern. The four broad areas are shown in Table 13.1 together with a
mean percentage satisfaction score for each area (derived from the relative proportion of
good and bad statements allocated to each area).

| 1 Teachmg/Learning Activity Area

36.9% |

i 2 Personal/Interpersonal Activity Area

76.5%

[ 3 Using Facilities Activity Area

17.6%

| 4 Spending/Budgeting Activity_Area

4.6%

These results are interesting because they suggest that students have a natural tendency
to regard some aspects of their educational experience as innately and potentially more
satisfying than others. The Personal and Interpersonal area which includes such activities
as "broadening my experience and outlook", "socialising with other students" and "talking
to staff freely and openly" is seen in a very positive light (77%) whereas Spending and
Budgeting has little potential for satisfaction (5%). Teaching and Learning Activities can
provide some satisfaction (37%) whereas Using Facilities generally tends to be a focus
for discontent (18%). In other words, students tend to stress negative aspects or
grievances when considering their finances or using polytechnic facilities. In contrast,
students are inclined to be more positive when considering their study experiences and
even mow enthusiastic when reflecting on their personal and interpersonal experiences.

Ratings of Importance

In addition to expressing their opinions about their likes and dislikes, students were also
required to rate each of the 44 core activities with respect to two aspects of importance.
The first rating was of the importance of each activity to enhancing the effectiveness of
study; the second rating was of the importance of each activity to enhancing students'
personal sense of well-being. Part of our purpose was to investigate the relationship
between these twin aspects of importanc e in order to provide information about students'
priorities.

Table 13.1 Areas of student concern

STUDENT SATISFACTION AND QUALITY

The results revealed that students see some activities as consistently more important
than others. Some activities were seen as critical to both study effectiveness and personal
well-being. Some other activities were perceived as relatively unimportant in both
domains; yet others were seen as differentially important to effectiveness of study or
well-being. The two ratings of importance appeared to be tapping different underlying
student perceptions. Table 13.2 below shows the five most and the five least important
activities together with their associated correlations between ratings of importance to
effective study and importance to personal well-being.

Most Important Activities for Effective Study Rank r

Understanding the content of lectures 1 0.30

Receiving well organised and structured tuition 2 0.24

Receiving feedback on assessed work 3 0.40

Receiving guidance and supervision by staff 4 0.41

Using time productively 5 0.45

Most Important Activities for Enhancing Wcll-Bcing

Paying my way/spending/budgeting 1 0.69

Increasing my self confidence 2 0.67

Having a base to leave personal property 3 0.59

Receiving well organised and structured tuition 4 0.24

Receiving feedback on assessed work 5= 0.60

Broadening my experience and outlook 5= 0.40

Least Important Activities for Effective Study

Joining societies and groups 1 0.66

Using the accommodation service 2 0.77

Using student services (e.g. medical etc) 3 0.80

Using canteen facilities 4 0.59

Buying acceptable (cost, range & quality) food... 5 0.60

Least Important Activities for Enhancing Wcll-Bcing

Using the accommodation service

1 0.77

Joining societies and groups

2 0.66

Accessing relevant information from Faculty

learning Centres

3= 0.64

Using student services (e.g. medical etc)

3= 0.80

Using Faculty Learning Centres

5= C.6j

Accessing relevant information from IT facilities

5= 0.66

Table 13.2 Most im/x>rtant and least important activities

Mapping Satisfaction and Importance Ratings

Both satisfaction and importance are essential ways of considering students' perceptions
of their educational activities. Polcyn (1986) has suggested that importance and
satisfaction can be represented as quadrants on a graph and each quadrant has different
organisational implications. Figure 13.1 shows the four key areas of student concern
mapped onto an importance/satisfaction graph.

'87

IMPLEMENTING QUALITY IN ORGANISATIONS

Higher Importance

♦ Spending/
Budgeting

+ Teaching/
Learning

Quadrat A:
Hjjh Importiooe
LovSariR&ctkn

♦ Personal/
Interpersonal

Quadra* B:
High Importance
Ui^h Samfaoicc

Lower

Higher

i 1 1 1 1

Satisfaction .

♦ Using

Facilities

H 1 1 1 1

Satisfaction

QadranrC:
Low Importance
Low Satijfacrioc

Quadra D:
Low Imparuocc
High Sukitaioa

Lower Importance

Figure 13.1 Importance 'satisfaction map

The graph shows thai both the Spending/Budgeting Area and the Teaching/Learning Area
fall within Quadrant A (ic. High importance and low satisfaction). Consequently, both
areas may be regarded as those where management may consider intervention a priority.
Spending and budgeting is an area of increasing concern to students at Polytechnics and
this is reflected in its position on the graph. Personal and Interpersonal Concerns fall
within Quadrant B (high importance and high satisfaction) on the graph and this reflects
students repeatedly- voiced opinion that being able to approach teaching staff and relate
to them in a free and open manner is of great importance to their general levels of
satisfaction. From the management perspective, the graph shows that it is necessary to
maintain course organisation so that opportunities for good quality interaction between
staff and students are optimised. Using Facilities can be seen to be the least important of
the four broad areas and it can be seen to fall in Quadrant C (low importance and low
satisfaction). Often it seems that good student facilities are weighted as more critical by
management than they are by students themselves. The graph shows that no area falls
within Quadrant D (low importance and high satisfaction) and this is probably an
outcome of the GFS procedure itself which is designed to target issues of high importance
rather than focus on issues of little or no importance.

CONCLUSIONS

These days, cost and quality arc two fundamental concerns in the quest for greater
accountability in higher education and this has heralded public sector consumerism as an

STUDENT SATISFACTION AND QUALITY

increasingly compelling force for change. As Pollitt (1988) has pointed out, the consumer
approach can appear in many different guises. At its weakest it can range from being a
visible but harmless cosmetic. Alternatively, it may manifest itself as a means of
providing information to consumers or managers. It may even consist of real attempts
to empower service users or inform HE managers. In the context of higher education,
this new public orientation is particularly significant in stressing the importance of
obtaining the opinions of stakeholders in the process. User feedback on quality of
educational services is increasingly acknowledged as critical in understanding what
quality means, although much work remains in developing appropriate methodologies and
techniques to measure and use this feedback to maximum effect.

One of the dangers in attempting to improve quality through students' views is that
expectations about change may become elevated to unattainable levels. If resource
constraints mean that managers are unable to respond to students suggestions for
improvements, then skepticism and distrust are likely to become additional
stumbling-blocks to change. It is our view that these dangers can be minimised if clear
and practical methodologies and models are derived which link student satisfaction
information, performance indicators, and management decision -making.

References

Churchill G and Suprenant C (1982) An Investigation into the Determinants of
Customer Satisfaction Journal of Marketing Research 19(6) pp 491-504

Lewis R and Booms B (1983) The Marketing Aspect of Service Quality in Berry L L,
Shostack G L and Upah G D (Eds.) Emerging Perspectives on Services Marketing
American Marketing Assoc Chicago

McMillan J R (1986) Including Satisfaction Data in Health Care Marketing Information
Systems in Cooper P D (Ed) Responding to the Challenge: health care marketing comes
of age Academy for Health Services Marketing American Marketing Assoc Chicago

MaisterDH (1985) The Psychology of Waking Lines in Czepicl J A, Solomon M R
and Suprenant C F The Sen-ice Encounter: managing employee -customer interaction in
sen ice businesses Lexington Books Lexington MA

Morris Committee Report (1990) Performance Indicators: report of a committee of
enquiry Chaired by Dr A Morris Polytechnics & Colleges Funding Council

Parasuraman A, Zeithaml V and Berry L L (1986) SERVQUAL: a multiple item scale
for measuring customers ' perceptions of senice quality Marketing Science Institute
Cambridge MA

Polcyn L J (1986) A Two Instrument Approach to Student Satisfaction Measurement
College & University 62 pp 18-24

Pollitt C (1988) Bringing Consumers into Performance Measurement: concepts,
consequences and constraints Policy and Politics 16(2) pp 77-87

14 Experiences in the design and conduct of
enterprise audits

Douglas Edgar, Glasgow Caledonian University, Glasgow, G4 OBA, Scotland
INTRODUCTION

In determining an audit methodology to be employed there are a number of questions
which have to be considered, questions such as:

Is it to be "top down" directed by management or "bottom up" directed by the staff
involved (ie who "owns" the audit)?

What is its purpose? Is it to attract funding support or identify a present position or
raise EHE awareness or identify those courses or groups of staff in need of enterprise
development?

To what extent is it to be quantitative (allowing comparisons) or qualitative (allowing
insights) in nature?

Is it to be course driven (via course committees) or subject driven (via subject specialist
groups)?

The audit undertaken at Glasgow Polytechnic, which was a pre-cursor to our main five
year EHE project, tended towards a top down model in that its approach and
methodology was directed by a management steering group. It was largely subject driven
through subject specialist groups. It was essentially qualitative in nature, although a
decision was made during the study to use a quantitative approach in questionnaires to
course organisers. It was concerned with awareness raising and position identification.
It commenced in mid session, late January 91, and concluded in November 91.

No attempt was made by the audit team to define enterprise in the form of some verbal
statement. However, the steering group's view of enterprise was about giving students
greater responsibility for their own learning; about developing their transferable personal
skills: and, through these, enhancing the students' economic and social worth.

The audit involved three major activities:

• A series of structured interviews, concerning current teaching and assessment
methods, with subject area staff to examine ways in which opportunities were
given for students to exercise responsibility for their own learning and develop
core transferable personal skills.

• A series of structured interviews on the placement experiences of a sample of
students (and their supervisors) to attempt to identify the characteristics of an
"enterprising" placement.

• The completion of a questionnaire by course organisers of degree courses, with
follow-up interviews, on the teaching and assessment of core transferable personal
skills. (These core transferable personal skills were identified as basic IT
keyboarding skills; study/learning skills; and communication skills.)

From the conduct of this audit six main issues were identified in its evaluation:

• The consequence of the use of the term "audit" and ownership of the audit.

• Time and timing of the audit.

• The role of the audit in raising EHE awareness.

• The choice and role of the audit team.

DESIGN AND CONDUCT OF ENTERPRISE AUDITS

• Qualitative v quantitative approach.

• Course driven v subject driven approach.

USE OF THE TERM "AUDIT" AND OWNERSHIP OF THE
AUDIT

The approach adopted was very much a top down approach directed by management as
opposed 10 a bottom up approach "owned" and directed by the staff involved. Essentially
management wished to identify where we were, in terms of the penetration of enterprise
development in the curriculum, and to identify examples of good practice. The audit
team planned and implemented an audit strategy with no prior consultation with course
teams or subject areas.

At the outset we found considerable concern in the minds of subject area colleagues
about the audit and a certain element of suspicion. This resulted in an over cautious
approach being adopted by some subject areas. For example the view was expressed on
a number of occasions, that subject areas would have to be careful how much they
disclosed lest they be considered as "enterprising enough" and lose out on any future
funding. This caution regarding disclosure took a little time to overcome and, to assist,
we wished to be as flexible as possible in terms of how we interfaced with subject area
representatives in order to gain maximum co-operation.

Consequently, whilst the agenda for the structured interview remained the same, the
actual procedures adopted, from one department to another, differed quite widely. This
resulted in some departments resolving to appoint coordinators with whom we interfaced,
and others writing their own departmental reports in order to assist the audit team.
Whilst this was welcomed and did have many advantages - it passed ownership of the
audit to departments - it did create some problems for the audit team in terms of
controlling the audit. In particular, it became impossible to keep this part of the audit
running to its pre planned completion time and there was an over- run on this aspect of
the study.

Wc believe such a top down approach can be criticised. It does, for example, perceive
the audit as an event rather than as an ongoing reflective review process; that is one
which could allow, in the right organisational culture, a commitment to action research
through team building and the development of a shared vision by staff about where the
Polytechnic wishes to move. Pennington (1991) encapsulates the problem well: "Audits
tend to be done by experts to others: by contrast review is something everyone is capable
of undertaking". Essentially it has to do with working with people, not working on
people.

Those holding this viewpoint would argue that if an institution seeks meaningful change
in its practices (in this case enterprise development in the curriculum), then such a bottom
up approach is desirable allowing course teams a feeling of confidence and competence
to undertake an audit themselves and letting them determine, on the basis of a set of
shared understandings, how best to manage this audit.

Given our experiences of undertaking the audit, particularly our experiences of working
with those departments who by volunteering to write reports for us took "ownership" of
the audit, we are sympathetic to this view. Letting subject teams determine, on the basis
of their shared set of understandings, how best to manage the audit seems the way to
create a climate for meaningful change. There are, of course, very clear costs associated
with such a process. It is developmental, can be lengthy and even indeterminate in terms
of its outcomes and therefore can be difficult to control in the project management sense.
But the arguments for this approach lie in the very fact that it is developmental and, as

IMPLEMENTING QUALITY IN ORGANISATIONS

Pennington states, "its momentum is sustained by the improvements staff themselves
identify as necessary".

We have been able to determine where we are in terms of enterprise development in the
curricula. Whether the audit, conducted the way it was, achieved that momentum towards
team building, meaningful change and a shared vision about where the Polytechnic
wished to move is an open question.

TIME AND TIMING OF THE AUDIT

The audit was plaimed to commence in late January. In three respects the audit seriously
over- ran its initial planned time. The first was agreeing the methods to be employed.
It took longer than planned for the management steering group to agree its preferred
strategy. The second was the study via subject area groups due to the methods adopted.
Hie third was in the time required for the data analysis, given the loosely structured
interviews approach adopted during our study with subject groups.

Bell (1991) reports that in the Hatfield Polytechnic audit one full session was allowed.
Our original timescales were far too optimistic for the qualitative route we took and the
commencement of the audit, in late January and straddling the summer leave period, was
ill-timed.

We believe audits should commence at the beginning, or close to the beginning, of an
academic session and thai a realistic period of time should be allowed for consultation
in the planning of the audit by any steering group established for that purpose. The
comments by Pennington, on the need for the rate of progress to be "acceptable to those
who are responsible for delivery", seem particularly apt.

ROLE OF AUDIT IN RAISING AWARENESS

There was a general lack of awareness within the Polytechnic of the Government's EHE
Initiative. This gave the audit team problems. Bell indicated similar problems and
suggested that, from his experiences, earlier briefings - both of papers and presentations
- would have assisted. This Glasgow audit fell into the same trap. Whilst an attempt at
awareness raising was made, through letters from the Vice-Principal and presentations to
heads of department, from our experiences we would suggest that this is inadequate.
Ewins (1991) indicates that a successful procedure adopted at the Polytechnic of Wales
was the holding of one day workshops involving all academic staff in the institution, and
a scries of workshops to key communicators. We believe this should also have been our
approach at the outset. Nevertheless, the audit has played a major part in raising
awareness and creating the right kind of climate for the main project. Indeed that may
be the audit's single most important outcome.

CHOICE AND ROLE OF AUDIT TEAM

The decision to appoint a full time audit assistant from outwith the body of the academic
staff, and without any previous experience of teaching in higher education or in the
process of the management of degree level courses, had its advantages and its
disadvantages. Its advantages lay in the audit assistant being a new face • every member
of staff she met resulted in a new professional relationship and was not influenced by the
perceptions of previous encounters between the parties. She was not a permanent
member of staff from any one constituency within the Polytechnic and could, therefore,
be seen as independent. Bell reports that Hatfield appointed a Senior Fellow who had,

DESIGN AND CONDUCT OF ENTERPRISE AUDITS

from his previous post, extensive knowledge of the Polytechnic and considerable
credibility with staff. On occasions the audit assistant did comment upon her status -
both in terms of her lack of "well knownness" within the Polytechnic and the grading of
her post - as influencing the priority academic staff perhaps assigned to dealing with her
requests for interviews and information returns.

For the audit to be worthwhile it requires to be perceived as an ongoing review process
and not as an "event". Whether such a review process needs an external audit team - ie.
external to the review groups - depends very much on the perceived role of that team.
We would suggest two possible roles exist: the first a facilitating role, the second a
reporting role. As facilitators their main task would be to raise awareness via the
mechanisms of workshops and the like, and in performing a co-ordinating and
communicating role between review groups. In their reporting role their main task would
be to draw together information on the activities and outcomes of review groups and to
consolidate these into meaningful statements which both the community of review groups,
and possibly a wider community beyond that, could consider.

As for personnel, there are clear advantages in having senior personnel well versed in
the processes and procedures of the institution to undertake such roles. What is critical,
however, is the independence and credibility which these personnel are able to bring with
them to these roles.

QUANTITATIVE v QUALITATIVE APPROACH

Bell, has commented on the problems of having a vague definition of enterprise and the
consequences this poses in questionnaire design. Earl (1991) also drew attention to the
limitations of questionnaires in terms of their ability to elicit qualitative information. In
this audit the management steering group was concerned about the well known
psychological reaction to questionnaires and that staff should not be discouraged from
participating by any complex questionnaire. However Ewins points up some of the real
problems in undertaking a less structured survey approach. He found in his audit thai:
replies did not always address the questions asked, the lack of response did not always
indicate that a feature did not exist in the course, responses were not always consistent.
This experience has been shared by the Glasgow audit team. We see a problem here.
On the one hand we have supported the bottom up, action learning approach espoused
by Pennington, yet on the other we have experienced the same problems as Ewins in
using structured (or loosely structured) survey methods. Pennington acknowledges that
the process he advocates "may appear to be lengthy and overly complex" and certainly
the experience of this audit team and Earl would indicate that structured interview
methods do require very considerably more time than would at first be imagined. By
contrast, we found the use of a questionnaire with course organisers later in the audit very
efficient and had little difficulty in working with them towards its completion.

We think we were right not to use questionnaires in the initial stage. However, we did
underestimate the problems of control and, subsequently, data analysis when we
encouraged departments, or groups of staff within them, to write their own reports. On
balance we accept the value of this (ic. transfer of "ownership") and, whilst we conclude
that it does impact very significantly on the time required to complete such an activity,
it can be justified. On the other hand we think it a mistake to assume that the use of
questionnaires will somehow, automatically, "switch off" staff. This was not our
experience with course organisers. Perhaps preparation for, and timing of any use of,
questionnaires is thti key to this issue. We believe, therefore, that there is a place for
both quantitative and qualitative approaches in such audits.

IMPLEMENTING QUALITY IN ORGANISATIONS

COURSE DRIVEN v SUBJECT DRIVEN APPROACH

This audit differed from the others mentioned in this report in that it approached
information gathering via subject area boards. We took this approach for two reasons.
First, efficiency - data on the teaching of a subject could be collected once from the
teaching group, rather than repetitiously through sundry course committees. Second, it
was felt that course organisers were unlikely to have the depth of detailed knowledge
about the teaching methods on their course to be able to answer the audit team's
questions without extensive referrals to subject specialists. Our experiences tend to
confirm this latter concern.

Conducting our study via subject areas created a level of "corporate consciousness"
amongst subject areas (and departments) about their teaching and assessment methods and
encouraged at least some subject groups into positive action. One group, for example,
managed to have their departmental report published in a professional journal! On the
other hand it left course boards largely untouched by the audit and so it may be that their
"corporate consciousness" has suffered as a consequence. This could be to the
disadvantage of the main EHE project at Gla; gow which is planned to work, essentially,
through course boards. On balance we conclude that, given the audit was a pre-cursor
to the main project and that we tend towards the philosophy espoused by Pennington (that
audit is a process of review and not an event), we would have been better conducting the
audit via course boards. It might have better set course boards "rolling" on the kind of
process review adopted by Pennington at Teesside. However, whilst something would
have been gained, something might also have been lost.

CONCLUSIONS

To .summarise from the experiences of this study and others, the following conclusions
are drawn:

a The term "audit" is inappropriate: "review" is a more appropriate term to describe
such processes.

b Reviews should be conducted "bottom up" (ie. ownership should reside with those
directly involved) and not "top down".

c They are useful mechanisms for raising enterprise initiative awareness, but this
must be positively planned as part of the audit.

d The role of the audit team should be to facilitate, co-ordinate and communicate
between course review groups rather than an investigative role.

e Qualitative (ie structured interview) methods are extremely expensive on human
resources and make for difficult analysis but allow ownership of the audit to pass
more easily to course review groups than quantitative methods.

f Audits are better conducted via course groups (ie course driven) than subject
groups if meaningful change is to be effected.

ERLC

DESIGN AND CONDUCT OF ENTERPRISE AUDITS 87

References

Bell D (1991) Conducting an Enterprise Audit at Hatfield Polytechnic

Earl S (1991) Conducting and Enterprise Audit at the Robert Gordon's Institute of
Technology

Ewins G (1991) Conducting an Enterprise Audit at the Polytechnic of Wales

Pennington G (1991) Conducting an Enterprise Audit at Teesside Polytechnic

Papers in the proceedings of the EHE Audit Methodologies Workshop April 91 (Copies
of the proceedings available from Educational Development Service at Glasgow
Caledonian University)

15 Monitoring the quality of quality
control systems

Henry I Ellington ard Gavin T N Ross, Robert Gordon University, Aberdeen,
Scotland

INTRODUCTION

"Sed quis custodiet ipsos/Custodes?" - Juvenal

Nearly 2000 years ago, the Roman satirist Juvenal made an extremely profound
observation. He pointed out that it was all very well posting guards to look after
something, but that the efficacy of such a measure would always be in doubt unless
further measures were taken to ensure that the guards did their job properly. In other
words, Juvenal asked. "Who will guard the guard? themselves?"

Somewhat more recently, the Vice Principal of Aberdeen's Robert Gordon Institute of
Technology (RGIT) asked himself a similar question about the quality system that had
been set up within RGIT in order to meet the accreditation requirements of the Council
for National Academic Awards (CNAA). In 1989, RGIT was accredited by CNAA to
validate and review its taught degree courses and was scheduled to be considered for re-
accreditation in 1996. The Vice Principal was only too well aware that any College
seeking such re-accreditation would be required to demonstrate that its quality system had
fully satisfied the standards set by the Institution itself, and that the responsibilities and
obligations under CNAA accreditation had been properly met. He therefore proposed to
RGIT s Academic council that a "meta-quality-control" system be set up within RGIT to
help ensure that this did in fact happen, and to provide the necessary evidence. We
describe here how this was done.

RGITS QUALITY ASSURANCE AND QUALITY CONTROL
SYSTEMS

RGIT is one of Scotland's Central Institutions, having over 4,300 full-time and 1,300
part-time students enrolled on over 80 different degree and other courses ranging from
Fine Art to Mechanical Engineering.

Academically, it is divided into four faculties (Design, Health & Food, Management,
and Science and Technology), each headed by an assistant principal who acts as
Executive Dean. Overall executive responsibility for the academic work of the institute
and the quality thereof rests with the vice principal, to whom the four assistant principals
report in respect of their faculty-related activities.

In January, 1990 RGITs Academic Council produced a quality handbook defining the
procedures that were to be adopted for the quality assurance and quality control of all
Institute courses - a handbook that has since been considered with interest by a number
of other HE Institutions.

Quality-assurance procedures within RGIT are based on a rolling programme of formal
validation and review events, with all taught courses being subject to rigorous validation
when they are first set up. All of RGITs Schools (academic departments within
faculties) and other academic units arc subject to periodic reviews and. within this school-
based system, the individual courses arc also reviewed. RGIT is wholly committed to
the principle of external peer review, as developed over the last quarter-century and more

MONITORING QUALITY CONTROL SYSTEMS

by the CNAA. For this reason, the panels responsible for conducting all such validation
and review events include a majority of external members - usually two academics and
tuo practitioners. RGIT is almost unique in that the panels are externally chaired, a
policy that the Institute believes adds greatly to the authority and overall credibility of its
quality- assurance system. The panel also includes two internal members, normally the
dean of the faculty concerned and the chairman of the faculty advisory committee which
handles the preliminary internal stages of the quality- assurance procedures.

It is also worth noting at this juncture that RGIT deliberately did not set up an
Academic Standards Committee or its equivalent, which again makes it unusual among
CNAA-accredited institutions. The Institute's thinking in not so doing was that it wished
the concern and the responsibility for quality to be built into every facet of the normal
systems of academic management. In other words, quality procedures would be based
and owned at the appropriate operational level. This did, of course, mean that there were
no "quality police" in RGIT. Rather, each member of the academic community was
charged with a self -responsibility to be actively involved in and personally aware of, the
issue of quality at all levels of academic activity,

RGITs annual quality-control procedures, on the other hand, are the responsibility of
its internal hierarchy of committees. These range from course panels and school
committees at operational level, through the faculty boards, which then report accordingly
to Academic Council. The Council has delegated responsibility from the Governing Body
for the overall planning, co-ordination, development and supervision of the academic
work of the Institute. The Institute's quality-control procedures involve carrying out
annual internal critical appraisals of the operation and success of all RGIT courses in
meeting their stated aims and objectives.

HOW RGITS INTERNAL QUALITY AUDIT TEAM WAS
FORMED

In the Spring of 1990, RGITs Vice Principal submitted a discussion paper to academic
council setting out the rationale for the establishment of an internal quality audit system
in RGIT. Such a systeir n'ould monitor the various quality- assurance and quality-control
systems described in the last section, thus effectively "guarding the guards". Academic
council subsequently agreed with the Vice Principal's proposal that responsibility for co-
ordinating the new "meta-quality-control" system should rest with the Institute's
educational development unit, since this had an institute-wide brief for promoting and
supporting academic development work of all types. It was also neutral in course-related
matters, as it did not itself offer courses leading to a CNAA award.

In subsequent discussions between the Vice Principal and the Head of the educational
development unit, it was agreed that the latter would set up and lead an institute- wide
internal quality audit team, which would report to the Planning and Review committee
through the Vice Principal. The team would include a representative from each of
RGITs four faculties. It would also include a representative of the Registry, which is
responsible for the management and co-ordination of RGITs quality programme. Its
terms of reference would be "to establish appropriate mechanisms for the appraisal of the
Institute's quality-assurance/quality-control procedures and their effectiveness 1 '.

It was also originally envisaged that the team would plan and implement formal
internal-appraisal events dealing with the Institute's quality procedures in 1991, 1993 and
1995, in preparation for the full CNAA re-accreditation event that was (at the time)
scheduled for 1996. Whilst subsequent external developments (notably the Government's
decision to disband CNAA in the summer of 1992 and grant full degree-awarding powers,

IMPLEMENTING QUALITY IN ORGANISATIONS

together with university status to institutions such as RGIT) have removed the original
need for this programme, it is intended that RGIT will still organise internal appraisal
events. The team has, however, concentrated its efforts to date on carrying out audits of
specific aspects of the Institute's quality-assurance and quality-control procedures.

THE FIRST QUALITY AUDIT CARRIED OUT BY THE TEAM

RGITs internal quality audit team was formally established at the start of the 1990-91
session, and immediately embarked upon the first task that the planning and review
committee had asked it to address - carrying out a critical examination of the Institute's
annual course appraisal system. Quality control at RGIT involves each course panel
carrying out a detailed critical appraisal of the previous year's operations and submitting
a report to the appropriate faculty board using a standard RGIT proforma. The faculty
boards then report to Academic Council on the outcome of this exercise with respect to
their own portfolios of courses.

The quality audit that was carried out by the team took place between October, 1990
and March, 1991, It involved critically examining all annual course appraisals that had
been completed in RGIT since the Institute obtained CNAA-accredited status in 1988 ie.
the annual appraisals for 1988-89 and 1989-90. The bulk of the work was carried out by
the faculty representatives on the team, who were asked to seek answers to the following
two key questions:

Ql "Is self-criticism as an ethos operating widely and deeply throughout RGIT?"
Q2 "Are issues identified by Course Panels subsequently being addressed?"

The evaluation was carried out using two specially-designed proformas, the first of which
was completed in every respect for every RGIT course that was subject to annual
appraisal (55 at the time). The second was used to provide a summary of the findings
in respect of courses operated by each faculty. Since the faculty of Science and
Technology had operated almost half the Institute's courses during the period under
review, the work on its courses was split between two people.

Once the faculty members of the internal quality audit team had finished their work,
they submitted their completed proformas to the team chair together with a short report
on their overall findings. The resulting material was then circulated to the team and an
overall report on the team's findings, conclusions and recommendations was produced.
This was submitted to the planning and review committee in March, 1991.

With regard to Question 1, the team came to the conclusion that an overall ethos of
self-criticism could not be considered to be as fully developed throughout RGIT as was
desirable. Substantial differences between individual faculties, schools and even courses
were also identified. In broad terms, the probiem seemed to be that critical appraisal
tended lo be reactive rather than proactive, since the annual course appraisal procedure
then being operated had clearly not been successful in encouraging proactive self-criticism
on the part of course panels. It was felt that there were two possible reasons for this
unsatisfactory state of affairs. Firstly, the guidance given within the standard proforma
had perhaps not been explicit enough. Secondly, a more intensive staff development
exercise designed to promote a more sophisticated understanding of the underlying
philosophy of "a self-critical academic community" would probably have been a good
idea. As a direct result of the criticisms and recommendations made by the team, RGITs
planning and review committee decided to carry out a radical review of the procedure.

9§

MONITORING QUALITY CONTROL SYSTEMS

With regard to Question 2, the team concluded that the issues that we^ identified by
course panels were in most cases being appropriately dealt with - faculty boards playing
a key role in this regard. The main exceptions appeared to be issues connected with
resourcing, which tended to be noted by the faculty boards but subsequently not always
fulfilled - largely due to the general reduction in unit costs that is a characteristic of
higher education today.

The team's more detailed findings in respect of each faculty's portfolio of courses and
the individual courses therein were passed back to each dean of faculty so that
appropriate action could be taken where necessary.

SECOND QUALITY AUDIT CARRIED OUT BY THE TEAM

As soon as it had completed the report on its first audit , the internal quality audit team
started work on the second task that the planning and review committee had asked it to
address - carrying out a critical appraisal of all validation and progress review events that
had been conducted in RGIT since the institute received CNAA accreditation.

Work on this task was carried out between April 1991 and February 1992, using
essentially the same approach that had been employed in the first audit. It involved the
various faculty representatives conducting detailed critical examinations of the thirty five
validations and progress reviews that had taken place in RGIT over the period in
question. In order to standardise the procedure, two specially-designed proformas were
again employed. The first proforma was completed in respect of each separate validation
or progress review, and involved determining whether the detailed procedures set out in
RGIT's quality handbook had been properly adhered to. The second proforma was used
to provide a summary of findings in respect of the events held within each faculty. To
help standardise the process, faculty scrutineers were provided with detailed guidelines
on how to complete the two proformas.

Once the faculty members of the team had finished their examinations, they again
submitted their completed proformas to the team chair together with a short report on
their overall findings. The resulting material was again circulated to the team for
discussion and an overall report on the team's findings, conclusions and recommendations
was again produced. This was submitted to the planning and review committee in
February, 1992.

On the whole, the team's findings were much more positive and satisfactory than had
been the case in respect of the earlier audit. It concluded that virtually all the validations
and progress reviews that had been carried out had been well organised and conducted.
Where problems had arisen, mainly in respect of four events in the faculty of Health and
Food, these were almost invariably due to the fact that the course panels concerned had
no; properly followed the Institute £ Adelines.

The team did, however, make a number of specific recommendations as to how RGIT\s
quality-assurance procedures could be simplified and improved. Firstly, it recommended
that the amount of documentation required in respect of both validation events and
progress review events should be drastically reduced, since the faculty scrutineers had
been unanimous in concluding that the present amount was both excessive and
unnecessary. This has long been a familiar refrain in CNAA member institutions and it
is not without a certain irony that , within the freedom of an accredited institution, this
should still be a problem. From an ins 'tutional standpoint - as from CNAA in the past -
it coHd be argued that the excessive dc umentatton is not in fact asked for, but rather
that it indicates a lack of synthesis and selection of material on the part of the Course
Panels. In other words, the blame lies mainly with the validated rather than with the

IMPLEMENTING QUALITY IN ORGANISATIONS

validators. Secondly, the team recommended that new course validation events should
be kept separate from school review events. If incorporated within such reviews for
logistical reasons, they should constitute clearly-id .itifia, *s f discrete events with separate
reports. Thirdly, it recommended that the role O' tne faculty advisory committees (which
scrutinise the documentation before each validation or review event) should be
strengthened, thus enabling any major problems to be identified and addressed at an early
stage. Fourthly, it recommended that a comprehensive staff development programme
should be implemented in respect of the Institute's quality- assurance and quality-control
procedures, so that everyone involved will be better prepared to carry out their respective
roles effectively. These various recommendations are all being addressed at the time of
writing and it is expected that they will all be implemented eventually.

THE PROPOSED FUTURE PROGRAMME FOR THE TEAM

Now that the internal quality audit team has successfully completed its first two audits,
it is intended to thank the original faculty representatives for their efforts and allow them
to demit office. Replacing them with new representatives will not only spread the work
load, but will also enable new people to benefit from the not inconsiderable staff
development opportunity that serving on the team provides. The EDU and Registry
representatives on the team will be unchanged, thus providing executive continuity.
RGITs planning and review committee has also identified the next two key issues that
it wishes the team to address. The first is the effectiveness of the process by which
matters relating to quality assurance and quality control are recorded in RGHT
documentation minutes of meetings. The second is the extent to which RGIT is
succeeding in meeting the CNAA criteria for accreditation, which are likely to be adopted
by RGIT even after the demise of CNAA.

CONCLUSION

On the basis of the work that has been carried out so far, the two authors of this paper
are in no doubt that the establishment of RGITs new internal quality audit system has
been a great success. The system has proved to be both practicable and effective, with
the internal quality audit team being able to complete both lis initial tasks more or less
to schedule (both took slightly longer than planned). The first two quality audits have
produced a large amount of extremely useful information about the effectiveness and
efficiency of RGITs quality-control and quality- assurance procedures. But most
important - the audits have identified specific ways in which both sets of procedures can
be made even more effective and efficient. The au f, iors therefore commend the RGIT
system to other colleges that are thinking about introducing "meta-quality-control"
procedures, and will be happy to provide more detailed information on request.

100

Section 3: Quality :~i Courses

The conference enabled participants to explore the applicability of different perspectives
on quality to educational contexts and practices. Many of the contributions were in the
fonn of case study reports, providing detailed and vivid data and making it possible to
explore issues of quality arising in specific initiatives and settings. Participants valued
the opportunity to hear what o f I,crs were doing, and in discussion to clarity for
themselves some of the issues to be addressed.

I Musallam, M Brophy and M Schilling in "Improving the quality of a national
curriculum: the role of textbooks" report the preliminary results of the evaluation of a
national curriculum reform programme which aims to raise the quality of basic and
secondary education in Jordan, The curriculum reform strategy relies heavily on the
introduction of new textbooks, but even with in-service training there appears to be a mis-
match between the use of the books and the aims of the reform. The reformers,
curriculum developers and teachers do not have a shared understanding of the meaning
of some of the basic reform concepts, and use different criteria in judging the quality of
a textbook. The pitfalls of "top-down" reform are illustrated. The medium does not
convey the message!

In "Developing quality in education: a way forward" R J D Rutherford demonstrates
how the principles of total quality management can be applied directly and successfully
in education. He identifies core elements and key questions in the systematic and
continuous process of developing quality, and illustrates how these have shaped the
design of a course for teachers which prepares them to engage in quality development in
their own schools. "Practising what he preaches", the author describes some numerical
performance indicators for the course.

"Quality in course design" is explored by Eric Roper in the context of a professional
development scheme for educators working in all sectors of the education system. The
scheme is examined in relation to three perspectives on quality: fitness for purpose, how
it sets out to meet participants' needs, and how it attempts to empower participants to
take charge of their own professional development. Scheme processes and structures are
described, emerging issues in relation to quality in course design are discussed, and ways
forward in assuring quality are identified.

Louise Gibbs in "Quality horses for quality courses" reminds us that an important
element of quality is the matching of students to courses. She a^-csses the question of
what a "quality" student is and suggests that "if there can be unsuitable students for
courses, can there not also be unsuitable courses for students?" She identifies lessons to
be learned by the new generation, market-led higher education from adult continuing
education, explores some of the reasons why students have dropped out of music courses
and reports on a training programme for course advisers. She also raises the notion of
the place of intuition in advising prospective students.

In a provocative paper, "Recognising quality in engineering education", D C Hughes and
R G S Matthew consider both the purpose of education and course design features in the
context of engineering education at degree level. They outline the components of the
"engineering method" and ask how students would recognise that they were on a quality
course - one which allowed them to adopt a deep approach to learning and to develop
skills in using the engineering method.

16 Improving the quality of a national
curriculum: the role of textbooks

I Musallam, M Brophy and M Schilling, The Ministry of Education, Hie British
Council, Jordan, and Die University of Liverpool, 169 3BX, UK

INTRODUCTION: THE CURRICULUM REFORM PROJECT
IN JORDAN

Human resource development has been a long-standing priority for the Government in
Jordan and investment in the education and training system has served the country well
in the past. In the seventies and eighties, for example, large numbers of educated
Jordanians were employed in the Gulf region and their remittances were a major
economic factor in the national economy. Today enrolments at the basic, secondary and
higher education levels are amongst the highest in the world and it has been estimated
that at present every third Jordanian citizen is involved in some form of education or
training (General Directorate of Planning 1989).

The nineteen fifties, sixties and even seventies were periods of rapid expansion in
education. This expansion was often at the expense of quality, so that by the mid-eighties
there was a growing awareness of the need for reform of the education system itself.
Research into the quality of education, especially science education, revealed a picture
of overcrowded, under-resourced classrooms and ineffective teaching methods ( Abu
Sardaneh 1983; Ismail 1981; Abdalla 1990).

In his speech at the opening of the National Assembly in 1985, His Majesty King
Hussein called for the launching of a national programme for education reform and in
September 1987 under his patronage a National Conference of Educational Reform
(NCER) was inaugurated with the express purpose of developing a plan of action.
Following from this conference a ten-year national reform programme was developed
and is currently being implemented with assistance from a number of organizations
including The World Bank, The British Overseas Development Administration and The
British Council.

The primary purpose of the reform is to raise the quality of basic and secondary
education and so improve the nation's manpower base so that it will be better able to
meet both the domestic needs and those of the region's scientific and technological labour
markets. To achieve this the Government established National Subject Panels to revise
the curricula.

These teams were charged with developing curricula whose content and approach would
emphasize:

1 fostering creative thinking by the use of materials and approaches which
promote critical thinking, problem-solving, analytical and information
processing skills

2 enhancing student achievement by building flexibility in curriculum especially to
allow for the ability range of pupils

3 modernizing curriculum content and making it relevant to current conditions and
issues such as environmental and health issues

4 re-orienting classroom teaching methods to emphasize activity-based
experiences and applied learning

5 providing teacher guidance to encourage greater creativity and flexibility in
implementing the curriculum

102

QUALITY IN COURSES

6 enhancing the occupational and vocational orientation of students
• 7 designing continuity and integration into the curriculum of different grades and
subjects.

THE ROLE OF TEXTBOOKS IN THE REFORM

Two basic processes are being used to implement the curriculum reform, namely the
development of new textbooks and A/V materials and the in-service training of teachers.
Text books are seen as having a pivotal role in the reform process. In fact the project
has been categorized as a textbook-led reform and a considerable part of the in-service
training is devoted to training teachers on the use of the new books.

Year Grades Levels to be Implemented

1991
1992
1993
1994

9
10
11
12

Table 16.1 The schedule for implementing the reform

Although the basic curricula for all five subjects were prepared some of the texts were
not completed on schedule and so only twelve of the fifteen books proposed for
introduction in September 1991 have been implemented. Nevertheless the project has
started well. Almost 1.5 million new books have been distributed and are in use in
schools.

The plan of action for the reform called for the new texts to be evaluated during their
first year of use. so that they could be revised during the second year and "final" editions
introduced in the third year.

To organize and carry out the evaluation the Ministry of Education established a
Evaluation Division in September 1991 based in the General Directorate of Curriculum
and Educational Technology (GDCET).

DEVELOPING THE EVALUATION STRATEGY

The first questions that the staff of the new division had to address were. "What was the
purpose of the evaluation ? What was it that was to be evaluated and why?" It was
difficult to get a consensus on this, partly because of the organizational structure within
the Ministry of Education. Development of the curriculum and the text books is the
responsibility of the GDCET while in-service training is the responsibility of the Training
Centre. The Evaluation Division is based in and staffed by members of GDCET and one
view was that it should only e* aluate the work of it's own general directorate, i.e. the
curricula and the text books. A second view was that textbooks could not be evaluated
in isolation from how they were used in the classroom. And how the books were used
must inevitably depend on how teachers were trained to use them.
A compromise was reached in that the strategy eventually accepted was designed to

103

ROLE OF TEXTBOOKS

evaluate three things.

1 The books themselves

What could be called their "technical" aspects (accuracy, printing quality,
illustrations).

2 The suitability of the books

Eg. how well they matched the level of the pupils, the needs of the local
community and the conditions in schools.

3 The way the books were used

Eg. were they used didactically with the teachers reading and explaining the
concepts or were they used by the pupils in group work or for following
activities.

THE STRUCTURE AGREED

Subject
Grade

Maths

Arabic

Science

Islamic
Studies

Social
Studies

Table 16,2 New texts introduced into Jordanian schools in September J 99 J

Twelve new texts were introduced into the schools in September 1991 and the Evaluation
Division is responsible for evaluating them during 1991-92. Just as there had to be a
compromise on the purposes of the evaluation so too was there a compromise on the
structure used. Many different personalities and groups were involved in the discussions,
and the specialists from the different subject areas and the "customers" who were to
receive and act on the evaluation feedback had varying opinions on what information
would be most useful to them. In the end after a three month cycle of proposal,
discussion and revision a two stranded approach was agreed on and approved by the
Minister and Committee of Education. The two strands are the continuing and the final
phase.

The Continuing Phase

For each of the books being evaluated twenty-five teachers have been chosen from five
representative regions of the country. The teachers are asked to keep daily teaching logs
for one of the twelve books, using a prepared reporting format.

A supervisor from each region visits one teacher each week to interview them about
their experiences as recorded in the daily logs and to observe a lesson. Structured
interview forms and detailed observational schedules are provided. The supervisors
received a three-day training session on interview and observational techniques. Video
films were used to establish standardized and agreed procedures.

From the continuing phase it was intended that we should get information for each

10 4

QUALITY IN COURSES

book on the teaching experiences of twenty-five teachers who had taught it throughout
the year. If each teacher was interviewed and observed once every five weeks then in a
thirty-week teaching year we should have information from 150 (25 x 6) interviews and
150 lesson observations.

The Final Phase

To supplement and provide some cross referencing for the on-going strand it was agreed
that during the last month of the academic year questionnaires would be sent for each
book to teachers in 5% of the schools in the 23 regional educational directorates. The
questionnaires were designed to provided additional evidence about the suitability and use
of the books rather than to elicit responses about technical problems.

A parents* questionnaire has also been designed and will be distributed to the parents
of 150 pupils over all regions following each book. Both the teachers* and the parents*
questionnaires were piloted during February and are scheduled for distribution in May.

THE RESULTS

A number of the books are being published in two parts, one for each semester, so it was
possible to carry out a preliminary analysis of some of the texts based on the results from
the continuing phase collected over the first semester. Tne books have been generally
well received by the teachers and are considered to be a major improvement on the old
text. Although from the results obtained so far it is clear that the teachers and
supervisors are generally more inclined to comment on the "technical" aspects of the
books, the misprints and mistakes, rather than on their suitability and use. A variety of
"errors" have been identified for each book including printing, cultural and contextual
errors.

The types of errors identified varied from book to book but followed a logical pattern
in that in the lower level books there were relatively more difficulties with language and
illustrations and fewer with exercise questions. Specific language difficulties were often
to do with vocabulary.

Language

Illustrations

Exercises

Total

Grade 1 Arabic

101 (45)

93 (41)

33 (15)

227

Grade 5 Science

58 (28)

64 (31)

83 (40)

205

Grade 9 Maths

15 (24)

16 (26)

31 (50)

(percentages in brackets)

Table 16.3 Comparison of types of problem reported for three books

One of the main concerns was the level of dissatisfaction expressed with the illustrations
in the books for the lower grades especially over the lack of clarity and the numbers of
contextual and cultural errors. Exercise questions in the books were generally seen as
being too difficult. Although in light of this there were surprisingly few reports about the
books failing to cater for the range of pupils' abilities and their individual differences.

105

ROLE OF TEXTBOOKS

THE USE OF THE BOOKS

One disappointment for those involved in the reform has come from the reports and
observations about the way the books are used. A major aim of the reform is to
encourage more pupil-centred and activity-based learning and teaching. It had been
expected that with the introduction of the new texts and in-service training there would
be a move away from didactic methods towards greater pupil involvement and that this
would be reflected in the way the books were used. Unfortunately this does not appear
to have happened so far. The reports on the science book, for example, show that the
books were still predominantly being used by the teachers. The main pupil usages were
for doing exercise questions and home work. There wasn't a single instance of that book
being used for group work in science.

IMPLICATIONS FOR THE EVALUATION
Problems of Reliability

One problem which the results have highlighted is the difficulty experienced in achieving
a shared understanding of the meanings of some of the key concepts of the reform,
concepts such as "problem-solving" and "catering for individual differences". Although
the supervisors received specific training on using the observation schedule there was still
a wide variation in the way that they reported seeing these concepts being implemented
in the classroom. For example one supervisor reported that there was no evidence of
problem -solving activities in any of the twelve lessons he observed while another reported
that he saw it in nine of the eleven lessons observed.

Region

Total %

"Yes"

24 45

"No"

29 55

Tabic 16.4 Reports of problem-solving activities observed in lessons

Region

Total %

"Yes"

48 98

"No"

1 2

Table 16.5 Reports of observations of teachers catering for individual differences

100

QUALITY IN COURSES

At present we are unable to say if this was a real difference (ie. in the teaching) or
simply a difference in the supervisors' interpretations. There was more agreement
amongst the supervisors about teachers catering for the individual differences of the
pupils. K^re the supervisors noted that the teachers did this in all but one of the 49
lessons observed. But the explanations of what was accepted as "catering for individual
differences" varied greatly. Since these results a second round of training workshops has
been held and modifications made to the instruments to try to achieve a greater degree
of "mutual" understanding.

The Difficulty of Defining "Quality" for Textbooks

When starting the evaluation one of the questions asked was "What are the qualities of
a good textbook? What characteristics should it have?" A recent study of a series of
textbooks in Texas found that they had an average of over 700 mistakes per book yet
the publishers claimed that 'everyone agrees these are the best textbooks they've ever
seen" (The Guardian 1992;.

So how important is accuracy as a criteria for a good textbook?

As part of our study we surveyed teachers and curriculum workers to see what they felt
were the most important qualities that a good textbook should have. Using the opinions
of a group of regional supervisors a list of the ten most important criteria for good
textbooks was derived. A sample of forty-five teachers were then asked to rank each of
these criteria in order of importance. The results are shown in Table 16.6 below.

Criteria Teachers (45) Curriculum

Specialists (34)
Rank Mean Rank Mean

Relates to local community

(8.1)

(4.4)

Suits ability level of pupils

(8.0)

(8.5)

Meets curriculum objectives

(6.7)

(9.3)

Ideas/concepts logically

sequenced

(6.2)

(6.6)

Clear presentation of

ideas/concepts

(6.1)

(7.3)

Clear printing

(5.6)

(3.7)

Correct/no mistakes

(3.8)

(3.0)

Has activities which can

be done

(3.7)

(3.9)

Uses a variety of approaches

(3.6)

(4.6)

Good clear illustrations and

diagrams

(3.5)

(3.7)

Tabic 16.6 The qualities needed for a good text book: opinions of teachers and

curriculum specialists

For teachers "relating to local community" and "suiting the ability level of the pupils"

107

ROLE OF TEXTBOOKS

101

were the two most important criteria. Accuracy was seen as being relatively unimportant,
being ranked only seventh in order of priority and "good clear illustrations and digrams"
was rated as the least important.

The views of the thirty-four curriculum specialists in the curriculum directorate were
also obtained. Their opinions correlated only moderately (rank order p » 0.56) with those
of the teachers. Not surprisingly the curriculum workers placed greater emphasis on the
textbook meeting "curriculum objectives" Relating to the local community was ranked
only sixth. Both groups gave high priority to suiting the pupils ability level and to
presenting ideas and concepts in logical sequence.

But what was noticeable was that like the teachers the curriculum specialists gave a low
priority to "technical" aspects; accuracy was ranked as being the least important of the
ten, clear printing was ninth and clear illustrations and diagrams eighth in order of
importance.

The high priority given to curriculum objectives, suiting ability level, sequencing and
presentation (contrasted with the low priority given to the technical criteria) suggests
that the textbook is still seen as being a content or sequencing guide and the chief source
of information for teachers.

Problems with the technical criteria are only important when the pupils use the books
since they can be mediated if the materials are teacher-directed. Similarly the low ratings
given to activities and different teaching approaches suggests either that these are seen
as having a low priority in teaching or that they are the domain of the teacher not the
textbook.

From the results we suggest that for these teachers and curriculum specialists the
textbooks have a very specific function. They are guides to the curriculum which the
teachers can feel safe in using. The "good" textbook provides the teacher with the
information he or she needs to teach, assured that it suits the level of the students, is
relevant to the local community and is clearly presented and logically sequenced. Setting
out the teaching methods and the activities to be used are not seen as being a major part
of a text book's functions. Activities are to be determined and organized by the teacher.

The question is, is this view of the role of the textbook consistent with the aims of the
Jordanian reform? Will we ever achieve pupil -centred, activity-based learning which
develops critical thinking and problem-solving skills if the curriculum is still being
mediated through the textbooks which in turn are used almost exclusively by teachers?

References

Abdalla A M (1990) An Evaluation of First Year Practical Chemistry in Jordanian
Universities Unpublished PhD Thesis University of East Anglia

Abu Sardaneh H (1983) Surveying the State of Biology Laboratories and laboratory
Work in Government Secondary Schools in Jordan Unpublished MA Thesis University
of Jordan

General Directorate of Planning, Research and Development (1989) Progress of
Education in the Hashemite Kingdom of Jordan. Report Submitted to the 41st Session
of the International Conference of Education Geneva

The Guardian 6/2/92 Mistakes Turn American History Books into Bunk

KTiai. H A (1981) Using Locally Available Materials for School Chemistry in Jordan
Unpublished MSc Thesis University of East Anglia

17 Developing quality in education: a way
forward

R J D Rutherford, School of Education, The University of Birmingham, B15 2TT t
UK

INTRODUCTION

Quality has now become a major concern in all sectors of education. But what is meant
by quality and, assuming we can agree on a definition, where and how can quality be
developed? This paper will address these three key questions, explore the relationship
with total quality management, and describe a course for teachers which not only
attempted to exemplify the principles of total quality management but also to prepare the
course members to begin the work of quality development in their own schools.

There are a number of overlapping perspectives - drawn from industry and commerce -
which seek to define the meaning of quality. Each perspective has a rather different
emphasis. Quality may be about perfection, inspection, conformance to requirements,
fitness for purpose, improving customer/client satisfaction, or empowering the
customer/client. These perspectives raise interesting implications when applied to
education. Does inspection - the ritual pouring on of quality at the end of a process -
mean that 20% of students must fail in order to "preserve standards"? Does conformance
to requirements - which emphasises "quality systems" and "getting it right first time",
prevention rather than inspection - reduce a teacher's freedom to abandon a well-planned
class and attend to a particular concern or difficulty? Does fitness for purpose beg the
question of whose purpose should have priority: teacher's or student's? Does
empowering the customer/client - giving power to the student - mean enhancing their
ability to exact sanctions if they think that the education they have received is
unsatisfactory?

In the opinion of the author, quality is about improving customer/client satisfaction or,
in educational terms, improving the quality of learning for students. In practice, this
means engaging in a systematic and continuous process of development and incremental
change that is heavily dependent for its success on all the "partners" having meaning,
purpose and satisfaction in their work.

A very simple model of a school can be constructed which identifies the major processes
in the system (Hall 1 989): inputs; transformations; and outcomes. Inputs include policies,
staff, students, resources, and so forth. Transformations include teaching, tutoring,
learning, assessing, as well as staff training and development. Outcomes include the
academic, personal and social development of students and staff, and examination results.
The satisfaction of students, their parents, their future employers, and so forth, with the
education that has been provided should also be considered. It is clear that to develop
quality in all these aspects of a school's work would be a formidable and in some
respects perhaps an unnecessary task: priorities for development must be established.
This issue has been well recognised in schools with the introduction of school

WHAT IS MEANT BY QUALITY?

WHERE CAN QUALITY BE DEVELOPED?

DEVELOPING QUALITY

103

development plans (Hall 1990; Hargreaves and Hopkins 1991). Such plans identify
specific objectives to be achieved during the forthcoming year and include detailed action
plans to show how these are to be achieved, and longer term objectives for the following
two or three years. Staff appraisal schemes are now being introduced in schools and will
help to ensure that the needs and aspirations of individual teachers are in harmony with
the needs and aspirations of the school, as articulated in the school development plan.

Whale and Ribbins (1990) have led a major initiative in quality development in the
Birmingham LEA. They have brought together the experience schools have acquired in
development planning with a systematic approach to monitoring and evaluating. Building
on their work, the author has emphasised the importance of considering the present and
future needs of the stakeholders or partners in the school, of a systematic strategy for
implementing planned change, and of leadership and team work. Thus developing quality
in education is, in the author* s opinion, a systematic and continuous process that involves
four core elements: strategic and action planning; implementing, monitoring and
evaluating planned change; responsive and flexible leadership; and co-operative and
supportive team work. These four core elements are clearly related to Oakland's (1989)
four major components of total quality management: a quality system; statistical process
control; management commitment; and team work. The author has found Oakland's ideas
and analysis invaluable. The following quotation, in particular, has proved most useful
when talking to colleagues about total quality management.

"The author and his colleagues have heard the excuse that 'our industry (or organization)
is different to any other industry (or organization)* in almost every industry or
organization with which they have been involved Clearly, there are technological
differences between all industries and nearly all organizations, but in terms of managing
total quality there are hardly any at all" (Oakland p 159).

The course was a three-term, ninety -hour course for experienced teachers in primary and
secondary schools leading to an Advanced Certificate in Education or. with further study,
an MEd. The first two terms* work consisted of lectures, discussions and workshops, and
tutorial support for the assignments. The third term's work was devoted to action learning
projects which focused on quality development in the teachers' own schools. The course
was held not at the University but, on alternate weeks, at two of the teachers' schools -
outreach centres, fhe author's aim statement for the course was:

"To design and deliver a course - based on the four core elements: planning; feedback;
leadership; and team work - that will enable the participants to develop the knowledge,
skill, experience, confidence and enthusiasm to begin and to continue the work of quality
development in their own schools."

The course was built around ten key questions. The first four of these focused on
strategic planning: What are we here for? Where are we now? Where do we want to get
to? What do we need to focus on? Thinking about the second and third of these
questions naturally leads to the fourth: the "quality gap". The fifth question was about
action planning: How do we get there?

HOW CAN QUALITY BE DEVELOPED?

A COURSE FOR QUALITY DEVELOPMENT

104

QUALITY IN COURSES

A systematic strategy for implementing planned change was described in response to
the sixth question: How do we effect change? This strategy involved identifying the
various stakeholders who were effecting or were affected by the change, and then
analysing each group in terms of six decisive factors. These are: linkage; openness;
gain/loss; ownership; leadership; and power (Lindquist 1978; Berg and Ostergren 1979;
Rutherford 1992). The different purposes for monitoring and evaluating and the different
techniques - questionnaires, interviews, and observation schedules - (Hopkins 1989) were
also described to answer the seventh and eighth questions: How are we doing? How have
we done?

PERFORMANCE INDICATORS FOR THE FIRST TERM

1. Target number of students:

2. Number of students at the first session:

3. Number of students who actually enrolled:

4. Percentage of number of students enrolled/target:

5. Percentage attendances for the nine taught sessions
(the tenth session was a "reading week" at half term):
87 88 72 83 94 83 94 94 66

6. Average percentage attendance:

7. Number of visits to see students in their own school:

8. Percentage of visits to see students in their own school:

9. Number of standard end -of -term questionnaires returned:

10. Percentage of questionnaires returned at the final session
(13 students were present):

11. Medians of the students' responses on a 1 to 5 scale (where 5 is
the most favourable response) to the ten statements on the questionnaire:
45^5544544

PERFORMANCE INDICATORS FOR THE SECOND TERM

1. Target number of students: 18

2. Number of students who are continuing with the course: 17

3. Percentage of students continuing: 94

4. Number of first assignments handed in within three weeks of the deadline: 14

5. Percentage of first assignments handed in within three weeks of the

deadline (17 maximum): 82

6. Grades for the assignments:

A A- A- A- B+ B+ B B B- B- C+ C and two returned unmarked

7. Percentage attendances for the nine taught sessions:
82 88 88 88 88 82 88 88 88

8. Average percentage attendance: 87

9. Number of visits to see students in their own school: 9

10. Percentage of visits to see students in their own school: 53

1 1. Number of standard end-of-term questionnaires returned: 11

12. Percentage of questionnaires returned at the final session
(15 students were present): 73

13. Medians of the students ' responses to the ten statements on the questionnaire:
5 545544445

Tabic 17.1 Course performance indicators

DEVELOPING QUALITY

105

However, following Oakland's suggestions, some numerical performance indicators were
also collected for each term's module of the course (Table 17.1). For example, the
average percentage attendance; the percentage of assignments handed in for marking; the
medians of the students* responses to an end-of-term questionnaire; the percentage of
students who continued on to the following term. This data was, arg?;ably, invaluable for
monitoring the quality of the course and to demonstrate that the author "practised what
he preached". It does not explain the reasons for problems but certainly shows if
problems exist. Such data has not been commonly collated in the School of Education
but as an increasing amount of our work takes place in outreach centres or is on contract,
it will be needed to fulfil any requirements for accountability. When similar data
becomes available it will be very interesting to compare the performance of the courses
at all four outreach centres, and indeed with more traditional courses that are held in the
School of Education.

In practice, these eight questions or stages in planning for quality development fuse into
one another. And the process of planning is at least as important as the final document.
However, ensuring that the plan is actually implemented, monitored and evaluated -
which depends on leadership and team work - are the most difficult and challenging parts
of the whole process. Thus the last two questions in the course focused on leadership
and team work: How can we develop our effectiveness as leaders? How can we build
teams?

A number of perspectives on leadership and team building proved useful in the course
(Rutherford 1992). First, there are the personal qualities and associated actions that
followers look for in leaders if they are to give exceptional performance. Kouzes and
Posner (1987) identify four such qualities: honesty; competence; vision; and enthusiasm.
They emphasise that credibility is the foundation of leadership. Second, there is the work
of Adair (1990) which focuses on three complementary functions that the effective leader
must attend to: achieving the task; building the team; and developing the individuals in
the team. Adair (1990) and Belbin (1981) have much to say about building teams that
the course members appreciated. Third, there is the work of Blanchard et al (1987)
which emphasises that followers have different needs, which depend on the particular task
in hand, so that leaders need a variety of styles and the ability to match the appropriate
style to the follower's needs. Fourth, the work of Argyris (1982) and Schon (1987) on
iheory-of-action underlines the difficulty of practising what we preach, of ensuring that
our behaviour really does match our stated values.

CONCLUDING REMARKS

The course is interesting in that it aims both to prepare the participants to engage in
quality development in their own schools, and to be an exemplar of a "quality course".
A great deal of care went into planning and delivering the course, and into providing
tutorial support. Visiting teachers in their own schools proved extremely beneficial in
helping to build and cement good working relationships, and to talk through problems.
"Keeping close to the customer" was one of the author's priorities. A strong sense of
group identity and commitment developed during the course which was reflected in the
performance indicators. Oakland's framework for total quality management has provided
ihe theoretical basis and inspiration for the course and, in the author's opinion and
experience, these principles can be applied very successfully in education. They underpin
a way of working described in this paper that can empower individuals to deliver quality
in education. However the larger task remains: introducing total quality management
throughout the whole school.

112

106 QUALITY IN COURSES

References

Adair J (1990) Not Bosses but Leaders 2nd Edn Kogan Page London

Argyris C (1982) Reasoning, Learning and Action Jossey-Bass San Francisco

Beibin R M (1981) Management Teams Butterworth-Heinemann Oxford

Berg B and Ostergren B (1979) Innovation Processes in Higher Education Studies in
Higher Education 4 pp 261-268

Blanchard FC, Zigarmi P and Zigarmi D (1987) Leadership and the One Minute Manager
Fontana Glasgow

Hall J Ed (1989) Local Education Authorities Project Management in Education BBC
Milton Keynes

Hall J Ed (1990) Local Education Authorities Project Locally Managed Schools BBC
Milton Keynes

Hargrcaves D and Hopkins D (1991) The Empowered School: the management and
practice of development planning Cassell London

Hopkins D (1989) Evaluation for School Development Open University Milton Keynes

Kouzes J M and Posner B Z (1987) The Leadership Challenge Jossey-Bass San
Francisco

Lindquist J (1978) Strategics for Change Pacific Soundings Press Berkeley CA

Oakland J S ( 1 989) Total Quality Management Buttervvoth-Heinemann Oxford

Rutherford D (1992) Appraisal in Action: a case study of innovation and leadership
Studies in Higher Education 17 pp 201-210

Schon D A (1987) Educating the Reflective Practitioner Jossey-Bass San Francisco

Whale E and Ribbins P (1990) Quality Development in Education: a case study of
supported self -evaluation in Birmingham Educational Review 42 pp 167-179

1 1 9

18 Quality in course design: empowering
students through course processes and
structures in a professional development
scheme

Eric Roper, feeds Metropolitan University, Leeds, LS6 3QS, UK
INTRODUCTION

The PGDip/MEd in the Faculty of Cultural and Education Studies at Leeds Metropolitan
University (formerly Leeds Polytechnic) is a professional development scheme for
educators working in all sectors of the education system. It is argued that the quality of
participants' experiences in the scheme depends on a flexible and coherent scheme
structure which is consistent with and responsive to clearly articulated scheme processes
The scheme will be examined in relation to three perspectives on quality: fitness for
purpose, how «t meets participants' needs and how it empowers participants to take
charge of their own professional development.

In 1990 the MEd was re-designed as a linked, modular Postgraduate Diploma/Master
of Education Scheme. What had been a three year, cohort-based, part time MEd in
applied research in education, became a flexible professional development scheme which,
although it retains a commitment to using research findings in exploring educational
practices and to the preparation of participants to undertake naturalistic research in the
MEd dissertation, also offers a wide range of elective modules which enable participants
to examine substantive areas of educational practice.

The PGDip consists of 12 modules, 6 core and 6 elective; satisfactory completion of the
PGDip permits entry to the Master's stage, which is based on a 16,000 wo;i dissertation
(equivalent to 8 modules). As well as Curriculum Lssues and Assessment Issues, which
provide frameworks for conceptualising key elements of the educational process, the core
PGDip modules are Approaches to Research in Education, Collaborative Action Research
and Recording. Reviewing and Planning (double module); for participants intending to
proceed to the MEd. two of the six elective modules must be Case Study Research I and
Case Study Research II. There are currently 32 validated PGDip elective modules.
These include generic, subject specific and sector specific modules, for example Human
Relations, Mathematics Education and Early Years modules, and participants may take
management-focussed electives from the MSc in Education Management.

SCHEME PROCESSES

The PGDip/MEd has been designed as a professional development scheme and, as such,
seeks to empower participants to take charge of their own professional development. The
flexibility stemming from its modular design facilitates this, but flexibility of this kind
has been tempered by a commitment to certain scheme processes which constrain
participants' freedom of choice, at least at the structural level, for example by limiting
the number of PGDip electives to six (four if participants intend to proceed to the MEd),
Because the scheme seeks to promote in participants the ownership of change as a basis
for professional development and professional learning, the methodology of all modules
seeks to enable participants to articulate their professional experiences, re-examine and
make explicit their professional values and goals, reflect on and analyse taken for granted

114

108

QUALITY IN COURSES

arrangements and practices, and identify any felt unease about their day to day
professional activities in terms of mis-match between aspiration and present experience.
Professional development is grounded in such activities, for it is through them that
participants come to understand the nature of and need for chaw "^cognise what it is
in schools, classrooms and themselves that makes change diffcu.c to accomplish, feel
some control over the situation, construct and own agendas for personal action, and
develop a motivation for change that is personally founded (Rudduck 1988).

All scheme modules focus attention on how individuals and groups, including scheme
participants, see their situations, and on how these different perspectives are influenced
by prevalent educational theories, ideologies and personal value systems, and all make
use of active teaching and learning strategies which sharpen participants' skills to act in
these situations. Moreover, ail modules must address the generic objectives of the
scheme: to enhance participants' self- awareness, insight, cepacity for constructive
criticism, capacity for rational action based on reflection, interpersonal skills and active
professionalism.

The double PGDip core module Recording, Reviewing and Planning underpins the
whole scheme and is central to its process. It involves participants in reflecting,
reviewing, recording and target setting, collaboratively as well as individually. These
activities enhance participants* understandings of themselves, and support their search for
identity as professionals. They enable participants to feel some sense of individual power
at the centre of the action, not only in terms of the scheme itself, but also in relation to
their professional lives as the scheme activities impact on their day to day work.

The research modules in the PGDip are derived from the original MEd and are intended
to provide participants with a thorough grounding in the naturalistic research methods
they are expected to use in their dissertations. Furthermore, experience in the original
MEd confirmed the value of a course process grounded in naturalistic methodology for
promoting both the personal and professional development of practitioners: participants
were ao. to identify ways in which their interactions with colleagues had been
transformed by their understanding of, for example, the need to explore the meanings
invested by different individuals in events and practices in the workplace, by their
articulation for themselves of ethical principles of procedure, and by their development
of the interpersonal skills needed by the naturalistic researcher.

The research modules arc also intended to empower participants to make enquiries into
the learning which they facilitate. Priority is given to equipping all participants in the
scheme to approach in a more informed and skilled way the situations and problems
which they encounter, or might expect to encounter, during their daily professional lives.
As a means to this end, it is thought to be of fundamental importance that teachers and
other professionals in education who are responsible for actions within professional
settings should be equipped to research the consequences of such actions and
progressively identify and focus attention on significant and important issues and
problems in the specific settings in which they occur. In this sense the scheme attempts
to bridge the gap between practitioner and researcher as a basis for participants*
professional development.

SCHEME STRUCTURES

Scheme structures should be fit for these purposes and, ideally, take their form from
scheme processes.

The flexibility inherent in a modular scheme, together with the range of electives, has
certainly assisted in the marketing of the scheme which, now in its third semester of

115

QUALITY IN COURSE DESIGN

109

operation, has over 50 participants. Unlike the original MEd, participants may join the
scheme in any semester (starting in October or February), and may take semesters out or
vary the number of modules taken in any semester, according to personal and professional
circumstances. The Scheme Leader manages a staffing budget and, because most elective
modules have been designed to be offered as taught or tutored modules (with contact time
adjusted pro rata according to numbers of participants) or by independent study, it is
possible to offer participants their preferred elective modules by adjusting the mode of
delivery to take account of available resources.

Because most scheme modules generate activities and investigations that participants
complete in the workplace, they are delivered as long, thin modules, one in each half of
the PGDip/MEd evening, using 10 or 12 of the sessions in the 15 week semester. Taught
modules have c 18 hours of contact time out of a total workload of between 45 hours
(minimum) and 75 hours (maximum).

The Recording, Reviewing and Planning double module is completed in action learning
groups (ALGs), consisting of approximately six participants and an ALG adviser, meeting
throughout the period of its members registration for the PGDip. (ALGs are reconstituted
for the Master s stage of the scheme). In the PGDip these groups meet on six occasions
each 15 week semester, for periods of between one and three hours, giving a total of c
10 hours each semester. This arrangement is a deliberate attempt to counteract what
could in a modular scheme become a fragmented experience, by enabling participants to
identify with a cohesive group.

Action learning groups are mutual support groups, which provide help and advice for
members, who are engaged on similar practical tasks. In the PGDip the task is to
complete the Recording, Reviewing and Planning module, while at the MEd stage ALG
members are preparing their dissertations. The ALG adopts an action frame of reference,
and members question each other about progress and blockages in exploring what has
been done and what needs to be done. Each member is challenged to verify facts and
interpretations, and to justify proposals. By reflecting on what they are doing, members
of the action learning group see how to do it better, and consult others in the group in
the act of getting it done. In helping each other, participants see more clearly how to
help themselves. The variety of participant professional backgrounds in the ALG
(typically reflecting all sectors of the school sys f em, further and higher education and
nurse education) is an asset, for example h challenging members" definitions of their
situations.

Scheme tutors act as advisers to action learning groups, and are responsible for
providing participants with both academic and pastoral support. In the ALGs participants
explore their reactions to the scheme, their professional development concerns and goals,
and identify ways in which the scheme might be used to meet individual needs and
achieve personal goals. In the PGDip, the tasks participants complete with the support
of their ALG are in practice:

♦ log/diary/journal keeping, recording their experiences in the scheme and making
links between the scheme and the workplace;

♦ the preparation of programme proposals for the next semester, which state
professional development targets and identify strategies for achieving them, and
which can irclude proposals for Independent Study modules (written by the
participants themselves to meet identified professional development needs) and
requests for advanced standing based on previous study or professional
development experience; and

♦ the preparation of the Statement (the module assignment for Recording, Reviewing
and Planning submitted at the end of the last PGDip semester, and requiring

lie

110

QUALITY IN COURSES

participants to identify their learning and development during the PGDip stage of
he scheme, grounding their claims in data extracted from the log/diary/joumal,
programme proposals and module assignments).

The style of working in the ALG and the production of the Statement also contribute to
participants' understanding of naturalistic research in preparation for the MEd dissertation.
Work in the ALG makes participants aware of the diverse "multiple realities" of ALG
members as they give meaning to their experiences in the scheme, while the Statement
is a kind of case study of each participant's experience of the PGDip stage of the scheme,
and is focussed on issues which are explored and grounded in data. The delivery of the
module Case Study Research II, which is focussed on data analysis and presentation, is
able to draw on participants' experiences of preparing their Statements.

Participants' programme proposals are examined by the scheme's Approvals Board.
The scheme sets out to embody maximum flexibility and deliberately imposes few
restrictions on participants* decisions about the order in which to take modules, because
individual participants will start from different points in terms of previous experience and
initially perceived needs. ALGs help participants to articulate their growing awareness
of their own professional development, and tG develop programme proposals which match
their needs. The scheme process is an enabling one, supporting participants in developing
programmes which have coherence and progression in terms of individual professional
development needs. The headings to be used in writing programme proposals,
independent study module proposals and requests for advanced standing, together with
the criteria to be applied by the Approvals Board in considering them, are stated in the
scheme handbook.

Because the PGDip/MEd was validated as a professional development scheme, rather
than as a course in the traditional sense, an Approvals Board was considered essential for
examining participants* proposed programmes of study; its existence also enables modules
to be revised and new elective modules added to the scheme in response to the emerging
needs of participants. The criteria to be applied by the Approvals Board have an enabling
function; they are written to support participants* in arguing the case for individual
programmes of study which meet articulated professional development needs.

In the scheme documentation, attention has been given to the information needs of
scheme participants and tutors. As well as stating the criteria to be applied by the
Approvals Board, the scheme handbook includes protocols < which set out scheme
requirements) for the log/diary/joumal, action learning groups (identifying the
responsibilities of both advisers and ALG members), the Statement, MEd dissertations,
and procedures for negotiating a research contract in educational settings; guidelines for
module assignments, programme proposals and for independent study module proposals;
glossaries of scheme and research terminology; and module syllabuses.

Within the scheme, module syllabuses (other than those which describe research training
experiences) are seen as frameworks upon which participants and tutors can negotiate
module programmes which meet the professional development needs of participants, as
identified in their programme proposals. The scope of the negotiation involved not only
covers matters of content in relation to objectives, but also of learning and teaching styles
and assessment. The topics, issues or enquiries which participants pursue for assessment
purposes may arise from programme content or contribute to it, and module assignments
may be produced in a form suitable for a specific audience or use in the workplace (as
long as they address the scheme's three assessment domains: literature and theory,
reflection and analysis, and application and rcflexivity).

117

QUALITY IN COURSE D.2SIGN 111

QUALITY IN COURSE DESIGN: SOME EMERGING ISSUES

It can be argued that the quality of course design is best gauged by the quality of
participants' experiences, and that in a professional development scheme the latter
depends on developing a flexible and coherent course/scheme structure which is
consistent with and responsive to clearly articulated course/scheme processes - processes
which empower participants to take control of their own professional development by
identifying and meeting professional development needs. That this has been achieved to
some degree is evidenced by participants' perceptions of the strengths cf the scheme, and
of the opportunities it offers them. Nevertheless, weaknesses and threats as perceived by
participants and tutors have to be addressed.

The commitment of the scheme as a whole to the naturalistic inquiry/research paradigm,
and the implications of this for participants' programmes and activities both within the
scheme and in their own work situation, is made clear to all prospective participants in
advisory seminars and interviews. Nevertheless, participants understand and/or are
committed to the scheme process to different degrees, particularly in the early semesters.
Thus in a SWOT analysis of the scheme after the first semester of operation, groups of
participants identified the following as "threats": "people who are pressuring for an old
style MEd and won't engage in active learning (and the danger of tutors going along with
them even if it goes against the ethos of the course)"; "getting to know you activities"
(presumably the structured trust and climate building activities used in the first sessions
of the semester prior to participants forming themselves into ALGs); programme
proposals; and the "journal - for some".

As the scheme grows, participants who have overcome the "general insecurities which
come with starting a new course" will be able to reassure new entrants, but in a sense
participants will only come to understand and value the scheme processes through
engagement in them. The quality of their early experiences in the scheme will be critical,
and the perceived "disparity between the quality of leadership of the ALGs", and
participants' "not being absolutely clear", and the "the number of new elements" they
have to cope with, are perceived "threats" that ALG advisers will be able to help
participants deal with only if the expanding team of ALG advisers understands and shares
the ethos of the scheme and the way this is translated into processes and structures.

In the first semester of operation, the ALG advisers were those tutors who had
developed the use of ALGs in the original MEd, and so had experience of the ALG
adviser role. As the team has expanded not all ALG advisers have had previous
experience of the role. Even for those tutors who have had this experience, the role has
expanded: as expressed by one group of participants, the "limited amount of time
available presents a threat when considering the complexity of professional issues" to be
addressed; and the range of tasks to be completed with the support of the ALG has
increased, as has the range of skills participants are required to develop.

That some participants have found the log/diary/journal keeping a threat is
understandable, because it involves the identification of feelings (diary) in relation to
events (log) and, standing back somewhat, the acknowledgement of the issues, threads
and themes that they reveal and that are of significance in the participant's professional
understandings and actions (journal) (Holly 1989). In the ALG participants are
encouraged to share from these records, and although they control the release of this data,
this sharing requires the disclosure of personal information. If they are going to reveal
themselves to others in the ALG, trust in other group members is essential. Issues
relating to the timing and duration of ALG meetings are significant here, as participants
have identified: "ALGs - too infrequent to be the driving force of the programme -

IIS

112

QUALITY IN COURSES

sessions too long - more frequent, possibly shorter, sessions (needed)".

In the participants' first semester, much of the time in ALGs is taken up with coming
to understand the scheme requirements, particularly in relation to log/diary/journal
keeping and programme proposals, and although these are intended to be enabling devices
for identifying and working on professional development targets and strategies, and
participants are encouraged to find their own best ways of completing them within the
requirements and guidance provided, not all participants "own" them in this way. It takes
a minority of participants considerable time to see their value for their own purposes
rather than as externally imposed "threats", and either concerns about what is expected
are expressed and have to be dealt with or they are completed in a mechanistic fashion,
with little investment of self in the process and with little evidence that professional
development targets and strategies are grounded in log/diary/joumal work. There is a
tension between using time in the ALG to promote the scheme's agenda in terms of
requirements, and giving participants the opportunity to create their own agendas, which
is perhaps reflected in the view that there is "not enough 'informal' time in the ALG".

Participants are encouraged to use the ALG, log/diary/joumal, independent study-
opportunities, modules and module assignments strategically in order to achieve the
professional development targets identified in their programme proposals. Ideally
participants will take modules in the scheme in order to work towards these targets, and
will know enough of what they want from a particular module to negotiate appropriate
content and delivery strategies. This negotiation may, however, be too early for optimum
decision making and not allow for the clarification and refining of targets or the
identification of more significant targets as the module unfolds. Although most modules
are validated for delivery in different modes in order to make them more available, some
participants prefer taught inodules and are disappointed that modules they wish to take
are not available in this mode of delivery; on the other hand, many participants prefer
modules to be delivered as tutored modules, giving them more time to work
independently or in syndicates and to focus on their own particular issues and concerns.

The documentation provided in the scheme handbook has provoked the following
comment: "scheme outline difficult to understand, jargon laden - needs to be written in
user- friendly English". While not wishing to dismiss this criticism, it reveals the problem
of how to initiate new entrants into the culture (including the language) of the
PGDip/MEd scheme so that, through their own activities within it. they can begin to own
and shape it.

ASSURING QUALITY: WAYS FORWARD

Participants' and tutors* experiences within the scheme are reviewed each semester in
formal scheme monitoring meetings. One source of data for these meetings is the
University's Student Evaluation Questionnaire, designed for use across all courses but
permitting course-specific items to be added. The challenge faced by scheme tutors (and
participants) is to develop items for this instrument which reflect the culture of the
scheme and the specific issu > identified by participants in SWOT analyses of the scheme
as a whole and in module reviews. ALG advisers have regular contact with participants,
and given that one of the tasks of the ALG is to help participants to review their
experiences in the scheme, detailed feedback is available to tutors. The Statements
produced by participants at the end of the PGDip will provide valuable additional data.

Given the commitment of the scheme to naturalistic methods of inquiry and research,
it is intended to apply these in evaluating the scheme itself. Currently the scheme leader
is using action research methods to investigate his own role as an ALG adviser, and this

QUALITY IN COURSE DESIGN

113

will produce materials for use in ALGs and in staff development activities with tutors
(Roper 1992). There is scope for ALG advisers to undertake collaborative action research
into this centrally important tutor role within the scheme, and for ALGs themselves to
explore their different cultures in inter-group meetings and activities.

Team building and staff development activities are a priority, now that the scheme is
entering its third semester and the tutor team is expanding. Some of these activities will
be grounded in data and documents produced within the scheme: for example, reports of
action research undertaken by tutors into key roles within the scheme, participants*
programme proposals and Statements. Time has been found to enable potential ALG
advisers to observe established advisers in ALG meetings, and for ALG advisers to meet
together to share experiences and provide mutual support. Module leaders need to share
their experiences of getting modules off the ground, of negotiating module content and
delivery with participants and of delivering modules in different modes.

It is also intended to return to basics and for tutors to identify and debate issues and
approaches in teacher professional development, drawing on the growing body of teacher
development literature (Hargreaves and FuIIan 1992), which may offer conceptual
frameworks for appraising the PGDip/MEd as a professional development scheme and
suggest possible activities which can be developed for use within the scheme (e.g. the
writing of collaborative autobiography by members of ALGs). Similarly, literature on
collaborative enquiry may help tutors to appraise the delivery of scheme modules and the
operation of ALGs - posing questions, for example, about die extent to which the latter
have transcended contrived collegiality and developed genuinely collaborative cultures
(Little 1990; Hargreaves 1992). An important reason for the success of the scheme in
attracting participants is its ethos and vision, and its commitment to process, and these
need to be reviewed and reconfirmed as the scheme expands.

References

Hargreaves A (1992) Cultures of Teaching: a focus for change in Hargreaves and
FuIIan (Eds.) 1992

Hargreaves A and Fullan M G (Eds.) (1992) Understanding Teacher Development
Cassell London

Holly M L (1989) Writing to Grow: keeping a personal - professional journal
Hcinemann Portsmouth. NH

Little J W (1990) The Persistence of Privacy: autonomy and initiative in teachers'
professional relations Teachers College Record 91 pp 509-36

Roper E H (1992) How to Intervene? - Facilitating action learning groups in Bruce C
S and Russell A L (Eds.) Transforming Tomorrow Today (Proceedings of the Second
World Congress on Action Learning) Action Learning, Action Research and Process
Management Association Incorporated Brisbane

Rudduck J (1988) The Ownership of Change as a Basis for Teachers' Professional
Learning in Caldcrhead J (Ed.) Teachers ' Professional Learning Falmer Lewes

120

19 Quality horses for quality courses:
matching students with courses in music

Louise Gibbs, Goldsmiths' College, University of London, London, SE14 6NW, UK
INTRODUCTION

Selecting suitable candidates for courses in further and higher education is undoubtedly
crucial to the quality of course outcomes and thus the perceived quality of a course. Poor
or unsuitable students are as likely as poor teaching provision to undermine the evaluation
of a course's quality. While we may have little control over the quality, academic or
otherwise, of those who present themselves as prospective students, nonetheless, the
responsibility for deciding upon what are quality students and selecting them remains
ours. In the cycle of educational activities and thus of quality procedures, student
recruitment and interview marks a decisive point. Misjudged selection or mismatching
of courses and students may make any further evaluations of course quality redundant.
In this paper I wish to argue that:

1 the perception of course quality is individual as well as general; students'
involvement with quality assurance procedures starts with course advising
which should be seen as a case of matching rather than selecting

2 the conditions of adult and continuing education offer examples of client-
sensitive course advising, the analysis of which can offer insight into how
student-based quality procedures may work and thus be improved

3 like any other aspect of a service, course advising needs clearly defined
procedures but also requires the application of "intuition" to be sensitive to
the interpretation and judgement of individuals and their circumstances; a
quality assurance system requires much the same approach if it is to be useful
analysing and recommending on the particular as well as the general.

ASSESSING STUDENT SUITABILITY

In the cycle of course design, planning, teaching and assessment we, as providers of
courses, have some direct control and thus responsibility for ensuring quality in terms of
"fitness for purpose". If the fitness for purpose of a course can be me;isured by its ability
to graduate students who, as a consequence of the course:

• maintain attendance and do not "drop out"

• achieve high examination marks

• perform well

• find employment quickly and maintain it satisfactorily

• obtain personal satisfaction

• acquire deeper knowledge and appreciation of subject

• are motivated and equipped to pursue further study

then we attempt to choose those students who we feel have the potential to last the course
and achieve well in relation to these outcomes. However, we often have no more
sophisticated an instrument for assessing student suitability to study than the absorption
of a body of knowledge and the ability to pass examinations; we often judge and admit
students to further and higher education on the basis of GCSE or A level examination

MATCHING STUDENTS WITH COURSES

115

results even without seeing or interviewing them (Ramsden 1986).

When a course is designed to provide for a clearly specified need, in the way of specific
training rather than a general educational aim, it is not difficult to formulate a model of
the kind of person suitable for such a course and potentially able to fulfil its originating
purpose. If you want to train an orchestral musician, for instance, you select a competent
player, with good sightreading skills, who can blend musically, work well with other
players and take direction. As the selector you are the "expert" (Williams 1986). You
know the standards required. You have formulated some picture of the ideal orchestral
musician, the skills, experiences, knowledge and attitudes which are likely to make not
only a potentially good player but also a good student who would benefit from the kind
of training that you offer, and you select accordingly. The prospective student's role in
this is to accept an offer or reject it - not usually to negotiate the basis upon which it is
made. Entry to most further and higher education and training is controlled in this
apparently autocratic way. The institution sets the entry standards, the curriculum, and
the methods of assessment. However, in the end it is student performance which
vindicates or dwnns institutional provision.

ISSUES OF QUALITY IN MATCHING STUDENTS WITH
COURSES

But what is a "quality" student? Are not those who genuinely wish to study, potentially
"quality" students? And if there can be unsuitable students for courses, can there not also
be unsuitable courses for students? Lack of fit must also be an aspect of lack of fitness
for purpose. And while fitness for purpose can address itself to the outcomes of
educational provision does it not also incorporate the idea of quality of the means in
achieving the ends? Effective initial course advising and consequently better fitted course
placement. I argue, is the first step in the student- institution relationship of a quality
assurance kind. But only if the student is involved and takes responsibility for quality
assurance, as does the institution itself by example. Because the relationship between
institution and student is essentially unequal, it requires the institution to invite the
student to participate. The recent "market-speak" description of our students as "clients"
may indicate not only that institutions see themselves as offering a service, but also that
the relationship between student ana institution is less paternalistic, and more negotiable
in the way of a partnership (Jarvis 1983). The relationship between student and
institution as far as quality assurance is concerned must also surely be one of partnership.

The lessons now being learned in the new generation, market-led higher education have
probably been encountered by adult continuing education some time ago. Higher and
further education could have something to learn from the adult continuing education ethos
if not its method. Adult and continuing education (ACE) has traditionally attempted to
anticipate or respond to the perceived usually nonvocational needs of the local community
based around a concept of personal enrichment or liberal education (Jennings 1981).
Students are involved by choice and usually towards no particular extrinsic goal other
than the personal satisfaction of involvement, study or learning. However, with ACE
moving increasingly into accredited and award-bearing courses preparatory to further and
higher education or continuing professional development, teaching and learning is
becoming as goal-oriented as that within further and higher education (FHE).

The greatest contrasts between liberal ACE and qualification seeking FHE-type courses
would be in the more obvious autonomy and agency in their education that students have
in ACE (see Jarvis 1985). ACE students are used to paying directly out of their own
pockets for their courses and thus they often feel as if they have greater control over how

122

116

QUALITY IN COURSES

they are taught. They may also have a hand in proposing or designing what they learn
along community education lines (Allen et al 1987). And in my own experience of AE
teaching I have more than once felt the need to alter my course plan if it has seemed
inappropriate to the requirements of the majority of students or lose the course altogether.
These students support what they feel to be good courses by "voting with their feet 1 '.
They can walk right out again if it is not to their liking, and as students used to making
moves for themselves they are less hesitant about demanding their money's worth in the
way of adequate resources and relevant and effective teaching. The point is that adult
education students tend primarily to select their courses rather than being selected for
them, and because of their relative autonomy, their support or lack of it is a potent
indicator of the quality of a course certainly from the student' perspective.

The apparent autonomy of the ACE student makes for a dangerous life in adult
continuing education planning but sharpens the mind to quality issues. When students
abandon ACE courses in large numbers the course closes but this loss alerts you to a
possible quality problem. When you lose a few individuals you may never see them
again. The trouble is that issues of quality or lack of it often surface when it is too late.
The usual student contribution to course quality assurance is the course evaluation
questionnaire handed out towards the end of a course. While they can provide very
useful suggestions to the fine tuning of quality course provision, the answers are fairly
predictable because favourable evaluations tend to come from those who are satisfied
enough to stay the course. The most telling evaluations I would suggest come from those
who leave before the end, as this brief report may illustrate.

To give background to a quality evaluation exercise within the music programme in the
Department of Continuing and Community Education, Goldsmiths* College, an enquiry
was made into the reasons why students dropped out of four music classes. Tutor
permission is needed to enrol for all these classes and course advice is available. Some
figures briefly describe the results:

Course: (A) (B) (C) (D) Total

Enroled numbers: 15 18 12 12 57

Left early: 4 4 10 9

Reasons for leaving course:

-domestic or employment commitments: 3

-travel problems: 1

-course level too demanding: 2 (1 found another DCCE course)

-course level not demanding enough: 2 (2 found another DCCE course)
-course not quite what was expected: 1 v' 1 realised course would need some

commitment")

These figures in themselves are not significant and certainly on the basis of the sample
not general is able to the whole music programme which caters for approximately 800
students. What is important to me at least is the fact that out of nine ex-students five left
because of issues to do with the quality of the course. Although three subsequently
received course advice and found another DCCE course, two out of the five slipped
through, did not take advantage of the course advice service at all and left the DCCE
music programme altogether. While student perceptions are important indicators of the
quality in terms of teaching effectiveness and communication, institutions and the teachers
and "experts" that constitute them have the ultimate responsibility for determining the
relevance and content of what is taught and thus the quality of overall provision.

For students to assume some of the responsibility for the quality of their education they
need to be involved on a personal level and they also need to feel that they arc receiving

MATCHING STUDENTS WITH COURSES

117

personal attention. Students feel more inclined to ask your advice if they know your
name and face, though from my own point of view this kind of input to quality is a
double-edged sword. More informative prospectuses and precise course descriptions
help, but experience shows that face-to-face course advice is what prospective students
are after. Many of you will recognise the situation where the answer you are giving to
a telephone enquiry is exactly that given by the prospectus held in the hand of the
enquirer at the other end of line!

ENHANCING THE QUALITY OF COURSE ADVISING

There's a saying that if you wish to leam about something you should teach it. In
teaching some classes on course advising on the newly-launched Certificate in Music
Teaching to Adults course, I became aware of the value not only of having clear course
advising procedures but also of the use of "intuition" in applying them. I became more
explicitly aware of this in having to engage my students in simulated student-course
advising activities in the way of role play. Here is an example of a role play question:

"You are about to retire and you will now have time to devote to learning music. You
own a number of chord organs and electronic keyboards and have taughi yourself to
play by 'fiddling about*. You now feel that it's time to learn to play 'properly'."

One student takes the role of the prospective student and the other the role of the course
adviser.

While it was not too difficult to coach or point out to my students the general lines of
enquiry to be made, or even in what manner questions could be fielded, unanticipated
answers were received which they were nevertheless able to respond to when prompted
or led. We also framed the same role play game into the form of questions submitted to
an adviser and requiring a written reply. My 14 students gave some very individual
answers to the same question, despite making a generally similar analysis in class after
the activity had been done of the information needed by the counsellor and prospective
student. The class answers were analysed against my own answer given as the "expert".

At the start of this particular class, the quality of the advising done by the students was
surprisingly variable. However, once presented with a system in the way of a checklist
of questions, there was noticeable improvement all round with the questioning being less
haphazard and the process taking a shorter time to reach some conclusion. Here is a
somewhat abbreviated form of the procedures used in the course advising training
sessions described above:

1 Find out what and why the student wants to study and preparedness for study

• by direct questioning

• by indirect questioning

• by audition

• by examining portfolio

• by diagnostic test

What you need to know about the prospective student

• Musical interests

• Musical experience, if any

• Music educational background, if any

• Reasons for study

• Level of seriousness, commitment to study

• Appropriate home study or practice support/instrument

124

118

QUALITY IN COURSES

• Other commitments - family/employment

2 Suggest suitable course(s) or programme of study

• programme prospectus

• course descriptions

What you yourself need to know

• Scope of music programme on offer

• Entry requirements, content and level of individual courses

• Other courses which your own programme accesses

• Musical activities and experiences to which your courses can contribute

• Further study in music

• Employment opportunities in music
What in general you must have

• A wide knowledge of musical genres, styles and practices and the kinds
of skills, knowledge and attitudes that are part of these musical traditions

• An ability to listen carefully

• A genuine interest in placing prospective students within suitable courses

• An ability to analyse suitable levels of musical knowledge and experience
allied with a scheme of progressed instruction in music

3a) If suitable course(s) found

• outline course commitments and expectations, agree them, and ask if there are
any further queries

3b) If no suitable course(s) found

• suggest further lines of enquiry or other advisers

While it was not to difficult to spell out the general kinds of questions an adviser could
field, the particularity of responses require a flexible approach. There seemed to be times
when the checklist had to be abandoned or not followed in an evidently systematic way.

THE PLACE OF INTUITION

Earlier I proposed the notion of the use of "intuition" in advising prospective students.
The way that it is used here is to do with openmindedly interpreting and making a
"creative" guess or suggestion about a particular situation or set of circumstances you
may not have encountered before. I would suggest that what I call "intuition" in this
context is not just a flash of insight coming from nowhere and without antecedent or
preparation. The basis of Intuition is often unfathomable at the time but an aspect of what
has been called "tacit knowledge" or of knowing more than you can tell (Polanyi 1966).

You apply intuition when you creatively and imaginatively manipulate your own
knowledge and experience to solve a problem that is new to you or has unique
circumstances, often when formal systems or procedures cannot give guidance. Intuition
may have an unarticulated basis, it may also be "untidy" in the way that it operates, but
is flexible, sympathetic to the particular, and thus personal. Because of this, I would
advocate that intuition is an important part of human interaction. If we are evaluating
other human beings and their experience and potential as we inevitably do as teachers and
educators then to be fair we need to remember that we apply both systematic procedures
and intuition.

MATCHING STUDENTS WITH COURSES 119

References

Allen G et al (eds) (1987) Community Education: an agenda for educational reform
Open University Press Milton Keynes

Jarvis P (1983) Professional Education Croom Helm London

Jarvis P (1985) The Sociology of Adult and Continuing Education Croom Helm London

Jennings B (1981) Adult Education in the United Kingdom Newland Paper No 3
University of Hull

Polanyi M (1966) The Tacit Dimension Routledge & Kegan Paul London

Ramsden P (1986) Students and Quality In Moodie G C (ed) Standards and Criteria
in Higher Education SRHE & NFER-Nelson Guildford

Williams G (1986) The Missing Bottom Line In Moodie G C (ed) Standards and
Criteria in Higher Education SRHE & NFER-Nelson Guildford

20 Recognising quality in engineering
education

D C Hughes and R G S Matthew, Dept of Civil Engineering, University of
Bradford, Bradford, BD7 1DP, UK

INTRODUCTION

Quality can only be considered when both the purpose of education and the course design
features associated with its attainment have been absorbed. This paper considers both
elements with regard to engineering education at degree bvel. The course features are
considered from the perspective of what the students should be experiencing rather than
how the academic is performing.

Quality of higher education is commonly expressed in terms of the teacher's
performance. End of course questionnaires handed out to students often require comment
upon the clarity of overhead projection transparencies, quality of handouts and whether
lectures started and finished on time. This seems to miss the point somewhat; after all,
learning is something which learners do. rather than something which is done to them.
It would be more productive, if more difficult, to assess how well our efforts helped the
students to learn.

We shall take the definition of "quality" to be fitness for purpose. This begs the
question as to what is the purpose of an engineering education? A universal answer is
that we aim to improve the quality of thinking of our students. This has been elaborated
(Hawkins et al 1965), within an engineering context, by the American Society for
Engineering Education (ASEE) to be:

• mastery of the fundamental scientific principles and a command of the basic
knowledge underlying a branch of engineering, and

♦ thorough understanding of the engineering method and elementary
competence in its application.

This paper explores two questions:

1 What are the components of the "engineering method"?

2 How would students recognise that they were on a quality course which
allowed them to develop skills in using the engineering method?

It is not our intention to discuss different teaching methodologies but rather their
characteristics by which quality may be judged.

WHAT ARE THE COMPONENTS OF THE ENGINEERING
METHOD?

In engineering we wish students to recognise that there is not a single correct solution to
most problems. Hence, participants at the workshop defined the engineering method as
a process of design to produce a workable solution within constraints imposed by society.
Although intellectual skills are involved it is important to recognise that the products of
engineering are implemented through people. Hence, two sets of skills are required.
Intellectual skills of creativity, analysis, synthesis and evaluation allow understanding

QUALITY IN ENGINEERING EDUCATION

121

and manipulation of knowledge. Problems may thus be solved and effective decisions
made. In doing so, deficiencies of knowledge and understanding will be recognised and
strategies to overcome them implemented.

Engineers work in teams and require an awareness of group processes and a repertoire
of appropriate skills. Harrisberger (1984) claims the working day comprises 80%" -ings",
namely planning, communicating, listening, persuading, discussing, informing, leading,
delegating, managing etc. The various intellectual skills may also be included in this
grouping. It is also important that young engineers develop a self- awareness of their
strengths and weaknesses.

The remaining part of the day is taken up with the "-ics" - physics, hydraulics,
electronics, mathematics etc. These form the first of the two ASEE goals. Whilst higher
education may not feel that an 80/20 split is appropriate for degree courses, it is
undeniable that the current balance is heavily weighted the other way.

What is immediately apparent is that the passive receipt of information from a lecture
is inadequate to develop skills in the engineering method. The University of Bradford's
mission statement is "Making Knowledge Work". This implies movement and effort.
Thus, students must be actively involved in their own education. It is also apparent that
a reductionist view of engineering is inappropriate. Engineering is a complex whole with
which students must be confronted, not only within a single topic but also across the
complete discipline.

HOW WOULD STUDENTS RECOGNISE QUALITY?

If we wish to improve students* thinking it is first necessary to ask how they approach
their learning. Research suggests that students adopt either a surface or a deep approach
to learning. This is largely a conscious intention dictated by circumstances. The surface
approach is characterised by focussing on the details and information in a learning
environment. Learning is by memorising facts in a linear manner with no interpretation.
In contrast, students adopting a deep approach focus on the meaning of the information
presented to them. They will relate facts, possibly to their own experiences, to develop
a coherent picture. It is obviously the latter approach which we should encourage and
which is implicit in our definition of the engineering method. It is unfortunate to note
the evidence suggests that it is the surface approach which is all too common in HE
(Gibbs 1992).

Four characteristics of course design and delivery have been identified which arc
associated with the deep approach to learning:

• learner activity

• student interaction

• a well structured knowledge base

• motivational context

We have already identified that the engineering method requires active involvement in
the learning process. Activity is not enough; it must be planned and time allowed for
reflection upon the learning so that it may be integrated with existing concepts,
knowledge and experience. One of the key features which drives students to the surface
approach is a heavy workload which does not allow the time for reflection. This must
be resisted, whatever the demands from professional institutions for more and more
content. Proficiency in engineering comes from method or process, not facts.
Not only is the active involvement of other students necessary for the "-ings" but also

128

122

QUALITY IN COURSES

for clarifying concepts. Learning is more profound if the subject matter is discussed,
chewed over or presented. After all, the best way to learn something is to have to teach
it. There are profound implications for the architecture of HE institutions. With the
dominant delivery mechanism in engineering degrees being the lecture we are reasonably
equipped with raked lecture theatres. These are of little use for small group discussions
etc. Flat floored rooms with moveable furniture, socialising areas and the like will
promote human interaction. Arrangements for laboratory classes need to flexible, and the
technical support staff kept informed, so that the students have access when required, not
when timetabled. Library staff need to be kept informed and involved in developments
so that they can ensure that the necessary support is available. Similarly, support for
computer hardware and software needs to be available to support the learning.

The engineering method requires an holistic approach to design which also aids the deep
approach to learning. However, the struclure of knowledge and its inter-relationship with
other knowledge and concepts must be clear to students. We have found that problems
directly imported from industry, whilst true to life, can contain ambiguities or avenues
which detract from the required learning experience. The use of such problems provides
the context in which learning is motivated. Students recognise a need to know for
themselves and experience ownership of the learning. It is incumbent on the academics
to provide an open and supportive environment for the students to experiment, question,
make mistakes and develop.

High level skills take time to develop. Thus, high class contact and excessive course
material should be avoided to create the appropriate learning environment for the
engineering method to be refined. Additionally, a lack of choice of subjects learned and
little opportunity to study a subject in depth reinforce the tendency to adopt the surface
rather than deep approach.

These features should be associated with the complete course from start to finish. The
temptation to say "Let's get the basics into them in the first two years and then ease off
in the final year and do all this experimental stuff" has to be avoided. Students have an
open mind as to what HE is when they first walk through the door. That is the time to
set the academic agenda, which, once set, will become the accepted norm. Confusion and
anxiety will be the result of changing for the last year and a surface approach will be the
inevitable result.

We have identified several fe^ures of degree courses which should be in place in order
for students to adopt both a deep approach to learning and follow the engineering method.
Indeed, the two are inextricably linked. A quality course should be judged against these
criteria rather than simplistic measures of lecturing performance.

References

Gibbs G (1992) Improving the Quality of Student Learning Technical and Educational
Services Ltd Bristol

Harrisberger L (1984) Curricula and Teaching Methods in Engineering Education
Education for the Professions Nelson Walton-on-Thames

Hawkins G A, Pettit J M and Walker E A (1965) Goals of Engineering Education - A
preliminary report American Society for Engineering Education Washington DC

CONCLUSIONS

Section 4: Quality in Learning and
Assessment

The theme that runs through all the papers in this section is that the quality of learning
and assessment is enhanced when learners themselves are taken seriously: when they are
helped to understand how they learn, when their preferences for and reactions to different
learning strategies are acknowledged, when they are empowered to take ownership of
their own learning and assessment, when they are seen as partners in evaluating the
quality of learning outcomes. Among the papers in this section are two workshop reports
which describe the activities completed, activities which were designed to draw on the
collective experience of participants in generating useful insights and products.

Dave Muller and Peter Funnell argue in their keynote paper, "Exploring learners*
perceptions of quality", that quality requires the learner to be at the centre of both the
learning process and the process by which such learning is evaluated. They make a
compelling case for obtaining regular qualitative feedback from learners, indicate how this
process is closer to the tenets of total quality management and can have an immediate
impact on the quality of individual learning, and identify implications for teaching and
learning strategies. Although the context is vocational education and training, their
insights have general applicability in the search for quality in learning and assessment.

"Concept mapping, post-questioning and feedback in distance education", by Robert M
Bernard, Som Naidu and Karin Lundgrcn-Cayrol, presents the findings of a field
experiment to compare the strengths and weaknesses of mathemagenic (eg. post-
questioning) versus generative (eg. concept mapping) instructional strategies. A
significant difference in cognitive gains by nursing students who completed at least 11
of 1 2 concept mapping exercises was found. It was also found that feedback is a key
element for instructional gain in both strategies. The paper also discusses students'
preferences for and reactions to different learning strategies.

The uext two papers address different but complementary themes in the context of
workplace-based learning. Caroline Bucklow in "Comparing chalk and cheese - quality
in assessing work-based learning", outlines a workshop designed to enable participants
to identify priority issues in, and build quality into, the process of assessing students'
workplace-based learning. The workshop is written up in a form that enable? it to be
replicated as a staff development activity for course teams. "Using learning contracts to
enhance the quality of work-based learning", by Iain Marshall and Margaret Mill,
describes a strategy for involving learners in negotiating learning plans and goals that are
responsive to their needs, and also in negotiating what is to count as evidence that
learning outcomes have been achieved. They also show that the accreditation of
workplace-based learning can be cost effective in terms of staff time.

In "Quality of assessment", Phil Race identifies four main processes of learning and
shows how traditional assessment methods are at cross purposes with them. He discusses
ten "worries" about assessment, and makes suggestions for avoiding the problems
associated with each of them. Practical ideas for improving assessment are illustrated,
with emphasis on self- and peer- assessment strategies which enable learners to take
ownership of and learn through the process. In "BS5750 for assessment" he presents the
products of the workshop associated with his paper, in which participants consider the
fitness for purpose of different forms of assessment, review their own experiences of
being assessed, and "lay down the law" by identifying criteria for quality in assessment.

130

21 Exploring learners' perceptions of
quality

Dave Mutter and Peter Funnell, Suffolk College, Ipswich, IP4 lLT t UK
INTRODUCTION

In response to the needs of external funding agencies and in order to provide information
for managers of education and training institutions, considerable emphisis has been
placed in recent years on the collection of quantitative learner feedback m the form of
questionnaires. This data has often been used to support course evaluation processes and
contribute to the creation of institutional performance indicators. It would not be
unreasonable to suggest that those of us with responsibility for the management and
delivery of vocational education and training have been subject to much tighter control
than those engaged in academic education and research or the delivery of the traditional
sixth form work. The Department of Employment, in particular, has placed emphasis
upon data driven planning and the evaluation of the effectiveness of vocational education
and training through the use of employer and learner feedback using questionnaire
methods.

However, in exploring "quality" in the context of vocational education and training, we
have found ourselves focusing on qualitative issues. In our first work in this field
(Muller and Funnell 199la) we explored quality from the perspective of learners,
employers and as a managerial issue. We became committed to "total quality
management" as referred to by leaders in the field (such as Oakland 1989a, 1989b) both
as a concept and as a form of operational practice. In subsequent papers (Muller and
Funnell 1991b, 1992a 1992b) a total quality approach to managing vocational education
and training was put forward emphasising the centrality of the learner in the learning
process.

Nevertheless, it has become clear to us that a managerial commitment, however
important it is, is not the sole prerequisite to a move towards total quality management.
Indeed we now believe that to be faithful to the major tenets of the total quality approach
we need to consider the extent to which we are really close to our customers. Put simply
it now seems to us that in following the total quality approach we need not only to
commit ourselves as educational and training managers to a customer orientated approach
but also recognise that we need also to explore the perceptions of our customers through
their own eyes. This paper sets a framework from within which we seek to achieve this.

WHAT IS QUALITY?

There are clearly multiple and varied definitions of quality as applied to vocational
education and training and multiple definers of the concept. We have sought to argue
against a competence based approach which suggests that quality is delivering a product
which is "fit for purpose". Rather, we have explored quality in terms of a wider range
of factors leading to a notion of "value-addedness". We consider it important that the
notion of quality be defined in terms of the longer term impact vocational education and
training will have on the learner. In exploring quality from this perspective we have
argued that the role of educational providers is to ensure:

"that learners fully participate in, and contribute to, the learning process in such a way that
they become responsible for creating, delivering and evaluating the product." (Muller and
Funnell 1991a, p 175)

131

126

QUALITY IN LEARNING AND ASSESSMENT

Following a detailed review of the literature and the exploration of a number of case
studies, we have proposed five defining criteria which contribute to the delivery of quality
vocational education (Miiller and Funnell 1991a). All of these focus specifically on the
learner and the role which he or she plays in determining the product:

a) quality recognises the central ity of the learner in obtaining successful learning
outcomes;

b) to some extent the learner must take ownership of the learning process;

c) the learner has some responsibility for determining the style and mode of delivery of
learning;

d) learners participate outside the formal learning context to the process of learning;

e) learners innovate, experiment and learn from failure as well as success, a notion
which challenges the concept of "zero defect" prevalent in total quality
management.

The conclusions we have come to have been derived from a managerial focused approach
to investigating quality. The case studies we have employed in our own work: have
tended to reflect different approaches to managing the delivery of education and training
and highlight good educational practice. However it was left to Barnett (1991) to
articulate more precisely the limitations of numerical performance indicators and the need
to place the learner at the centre of the educational institution as a whole. He proposed:

"What we need, in short, are indicators that a concern for the quality of student 'learning
is built into the internal life of the institution. In the previous section, I gave some
examples of what these might be. If we find an institution which has iastituted learning
weeks, has a staff development unit concerned about student learning and the student
experience, gives tangible rewards for high quality teaching, clearly looks to evidence of
the commitment to teaching in its promotions procedures, picks out examples of good
teaching in its newsletter, and encourages staff to explore teaching and learning intentions
within its course review procedures, then we have grounds for believing that here is an
institution in which the character of the student experience really matters and is not just
a piece of rhetoric" (p 50).

There remains, however, a need to go beyond this and to recognise that organisations
through managerial approaches alone cannot place the learner at the heart of the process
without ensuring that the tutor has fundamental devolved responsibility for creating the
product in partnership with the learner. We now believe that it is essential to place the
learner at the centre of the evaluation of learning processes and outcomes by focusing on
qualitative approaches to exploring learners* perceptions of quality. To achieve this the
responsibility for evaluation must be built in to the learning process itself and be owned,
in partnership, by learners and tutors. The rest of this paper will explore some first steps
in fulfilling this commitment to identify and act upon learners * perceptions of quality.

To date the major rationale for exploring and evaluating quality from the perspective of
learners has been to provide information for curriculum managers and those with broader
institutional responsibilities. This has also included external validating bodies and in
particular the Council for National Academic Awards (CNAA) which has placed
particular emphasis on the course review process. In further education, those external
agencies with Responsibility for funding have required the collation of student feedback

EVALUATING QUALITY

LEARNERS* PERCEPTIONS OF QUALITY

127

data, which impacted more upon the political context within which vocational education
was funded rather than the actual experience of the learner. Nevertheless, thete have
been a number of well tried and tested approaches to evaluating quality all of which have
been primarily concerned with providing information to be used by others in designing
the curriculum and in the planning of vocational education. It is notable that a great deal
of the data collected has been retrospective and has been of greater benefit to future
generations of learners rather than current learners.

Quantitative Approaches

The questionnaire approach has long been dominant in vocational education and training
as the means of obtaining feedback from learners. There are many quite sophisticated
questionnaires developed both in the UK and elsewhere (Bedford LEA 1989; County of
Avon 1989; and Thomson 1988). The basis for quantifying learners* perceptions of
quality is usually through some means of scaling. For example, a paper by Okun et al
(1986) reported on the development of a psychometric approach to investigating USA
College students' affective reactions to their academic lives. Students were asked how
they feel about major areas such as the classes they are taking, what they are learning,
their instructors and what is available to them in the way of services, guidance and
assistance. The response options were "delighted, pleased, mostly satisfied, mixed (both
equally satisfied and dissatisfied), mostly dissatisfied, unhappy and terrible." Data of this
kind can be valuable in that it offers quantitative feedback to tutors whilst collectively
providing an overall evaluation of the learning environment.

The need to collate feedback gathered through questionnaires has led to a number of
highly sophisticated commercial systems coming on the market. ScanTech Systems Ltd,
Dunstable, UK, are promoting a means of collecting and collating questionnaire responses
which captures data and processes it for presentation in graphical and tabular formats.
The information provided is of high standard and is readily available to those with front-
line curriculum responsibility as well as to institutional managers. The kinds of areas
explored are not dissimilar to other questionnaire approaches. For example, students are
asked for their views on college facilities such as buildings and the refectory and on
course/programme content and teaching, and response options are very good, good, below
standard and poor. The use of such systems can be invaluable if the major emphasis is
to be placed upon generalised data and a broader approach to evaluating learners '
perceptions of quality. However, it is important to recognise that learners are not in
ownership of this information nor is it built in as an integrated part of the learning
process.

Qualitative Approaches

Simultaneously with the prepaiation of this paper, the proceedings of a recently convened
CNAA conference on "Evaluating the Quality of the Student Experience" were published
(Burrows 1992). A major theme emerging from the conference has been the need to
place the student more at the centre of the evaluation process. A number of the papers
argued that giving students more autonomy and greater responsibility within a quality
assurance system must be seen as part of the educational process itself, and that greater
emphasis should be placed on the need to develop a dialogue between students and tutors.
Clearly in this context the quantitative, questionnaire derived, approach fails to place
ownership with the learner or create a dialogue which is in itself customer orientated.
Interestingly whilst a number of the papers presented explored qualitative rather than

133

128

QUALITY IN LEARNING AND ASSESSMENT

quantitative approaches to gaining feedback (eg. Baldwin and Carter 1992) it was a
student contribution which argued the case for building self and course assessment into
the programme of study itself (Rostron 1992).

In our view qualitative approaches to learners' perceptions of quality are closer to the
basic principles of total quality management and should have a greater impact upon the
teaching and learning process. From a managerial perspective the total quality movement
places particular emphasis on getting as close as possible to the customer and on building
quality in to the production process rather than excluding deficient products at the end.
If we apply this to vocational education and training, and recognise that the learner is
both customer and product, the need to find means of building quality into the learning
process becomes self evident.

Those closest to learners are either fellow learners (eg. those following the same
programme of study) or those with teaching responsibilities. In our view qualitative
approaches need to bring together learners and tutors in such a way that their explorations
of the learning process can have an immediate impact upon the curriculum. At the heart
of vocational education and training lies individuals with shared responsibility for
delivering a quality product in terms of a qualified learner outcome and a well developed
curriculum process. The encouragement of learners to share with tutors the ways in
which the process is developing can be seen to be of greater value than the retrospective
recording of quantitative data which may or may not be acted on by managers.
believe that it is necessary to give greater currency to the constructive dialogues which
learners and tutors have and to build this into the process of learning through devolving
responsibility to those who have the knowledge to transform the product at its formative
stage.

IMPLICATIONS FOR TEACHING AND LEARNING
STRATEGIES

In placing greater emphasis on regular qualitative feedback the process of teaching and
learning is likely to become influenced in different ways. There are immediate
implications both for learners and their tutors in coping with the responsibility of
managing their own curriculum. Many of these issues need greater exploration and it is
our intention to confront some of these during the next phase of our own work, which
will be an empirical investigation of learners' perceptions of quality. However, it is
important to identify some of these issues and challenges at this stage:

a) A major challenge will be the devising of means to support and teach learners how
to evaluate quality. This will require both objective explorations of learners*
perceptions of quality as well as an analytical approach to curriculum design by
tutors.

b) Learners will need support in understanding their own role in constructing the
products of vocational education and training. This is a particularly difficult concept
to grasp in the context of a customer purchasing a product from a provider. Rarely,
as purchasers, do customers have to participate in creating the product they are
buying.

c) Evaluation will need to become part of the teaching and learning programme. By this
means qualitative evaluation will become integral to the delivery of the curriculum
and will not be simply an adjunct to it. This too will challenge many existing
assumptions about the nature of curriculum design and the resource allocated to
traditional teaching.

LEARNERS' PERCEPTIONS OF QUALITY

129

d) There will be a need to formalise in a constructive, and non- threatening and non-
bureaucratic, manner the ways in which learners and tutors talk about the curriculum
and learning processes. It is a common assumption that good interpersonal
relationships between tutors and learners is an insufficient condition for quality
assurance procedures. In our view this assumption needs to be challenged. Indeed
in the process of challenging this assumption we may find that for too long we have
ignored good practice in this area.

e) There may emerge a need to give greater importance to peer review. Learners are
often the best judge of the curriculum and its delivery, and if engaged not only in the
process of comment but in taking action, are likely to recognise issues and to respond
positively to them.

f) Finally, qualitative evaluation without action is of no value. We will need to explore
and encourage learners and tutors to evolve informed strategies and to take actions.
Responsibility through constructive conversations will be shared and should lead to
a more sophisticated approach to the delivery of quality in vocational education and
training.

THE CHALLENGE - REALISM OR RHETORIC?

Concerns have been expressed that the "language" of qualitative approaches to evaluation
is both derived from, and only appropriate to, higher education. We feel uncomfortable
with this argument. Clearly it is important to recognise the different needs, and skill and
experience bases, of the consumers of vocational education and training. However it is
our view that all learners can be empowered to take responsibility for their own learning.
Indeed there are excellent examples of this operating in the compulsory education system
where one of the major strengths of the educational process is the partnership which has
emerged between parents, pupils and their teachers.

Clearly there is an idealism here. However at a practical level few of us with
ixperience of working in vocational education and training are unable to engage young
trainees or returning adults in conversations about their hopes, plans and current learning
experiences. Increasingly, the narrow views of teaching specific skills which lead to
competence without capability, is coming under critical attack (Marshall 1991). Few of
us would disagree that we want learners who are both competent and capable and,
whatever the teaching methods employed, that we need to develop entrepreneurial and
leadership skills. All work situations in the end require the individual to make decisions.
The starting point is placing some ownership for learning with all of those fully-
integrated and engaged with the educational system. What can be forgotten is that the
young trainee of today is the mature learner of tomorrow, complete with family and
community responsibilities. Life teaches us how to make decisions and to own
challenges, opportunities and problems. We must be wary as educators of failing to
recognise the power of the education process itself in supporting this.

THE NEXT STAGE

As part of our own process of learning and teaching about quality we are now planning
to mount a series of empirical studies into qualitative approaches to investigating learners*
perceptions of quality. It is our intention to collect data and to work with colleagues in
the more traditional areas of further and vocational education. We have plans to try and
compare both craft and technician courses as well as mature and younger learners. As
part of this empirical exercise we wish to bring together the learner and the tutor with a

135

130

QUALITY IN LEARNING AND ASSESSMENT

view to empowering both partners not only to evaluate quality but to improve it. We also
hope to demonstrate that the emphasis we have placed on learners owning the product is
appropriate and in doing so to be able to support the tenets of the total quality approach
to vocational education and training. In summary, we believe that:

a) it is important for tutors to get close to the learner and hence the product;

b) for quality to be built in to this process; arid

c) to encourage the learner and the tutor to take responsibility for constant
improvement.

This will legitimise the place of the learner at the centre of the evaluation process, and
is in line with the theoretical principles derived from the total quality approach. As such
it places responsibility on learners and their tutors for identifying the factors which
determine quality and for acting upon them.

References

Baldwin J and Carter B (1992) Workshops and Direct Comment, an Alternative to Staff
Questionnaires in Burrows A (ed) Evaluating the Quality of the Student Experience
CNAA London

Barnett R (1991) Delivering Quality in Student Learning in Miiller D J and Funnell P
Delivering Quality in Vocational Education Kogan Page London

Bedford LEA (1989) The Bedfordshire Quality Project An Evaluation Pack for Use by
Course Teams Bedford LEA Bedford

Burrows A (1992) (ed) Evaluating the Quality of the Student Experience CNAA
London

County of Avon (1989) County of Avon: Ensuring Quality Course Review Profile
County of Avon Avon

Marshall K (1991) NVQ's: An Assessment of the "Outcomes" Approach to Education
and Training Journal of Further and Higher Education 15 pp 56-64

Miiller D J and Funnell P (1991a) Delivering Quality in Vocational Education Kogan
Page London

Miiller D J and Funnell P (1991b) Realising Potential Through Quality Vocational
Education in Winterburn R (ed) Realising Human Potential Kogan Page London

Miiller D J and Funnell P (1992a) Initiating Change in Further and Vocational
Education: the quality approach Journal of Further and Higher Education

Miiller D J and Funnell P (1992b) An Exploration of the Concept of Quality in
Vocational Education and Training Education and Training Technology International

Oakland D N (1989a) (ed) Total Quality Management IFS Bedford

Oakland D N (1989b) Total Quality Management Heinemann Oxford

13G

LEARNERS 1 PERCEPTIONS OF QUALITY

131

Okun M A, Kardash C A, Stock W A, Sandler I N and Bauman D J (1986) Measuring
Perceptions of the Quality of Academic Life Among College Students Journal of College
Student Peisonnel 27 pp 447-451

Rostron J (1992) "Consumer Opinion" or "Action Research"? An evaluation student
speaks out in Burrows A (Ed) Evaluating the Quality of the Student Experience CNAA
London

Thomson C (1988) (Ed) Client Satisfaction: Monitoring Quality Further Education Staff
College Bristol

137

22 Concept mapping, post-questioning and
feedback in distance education

Robert M Bernard, Som Naidu and Karin Lundgren-Cayrol, Department of
Education, Concordia University, W Montreal, PQ, Canada

The aim of this paper is to disseminate findings from a field experiment in a Distance
Education setting, where the main goal was to compare mathemagenic and generative
instructional strategies for cognitive gain on both a short term and a long term basis. A
secondary goal was to look at students' preferences of learning style, and willingness to
acquire a new strategy. Mathemagenic and generative learning strategies are here defined
according to Rothkop (1970) and Jonassen (1985). Mathemagenic strategies refer to
instructor imposed activities which direct the learner to the basic content of the text (ie.
inserted or post-questioning techniques, behavioral objectives, advance organizers etc.)
as opposed to generative learning strategies, where the learner creates his/her own
representation of the content (eg. concept-mapping, note-taking, elaborations etc.).

It has been argued that mathemagenic devices are efficient time-wise in guiding the
student to specified content (Hamaker 1986). However, it has also been revealed that
students in distance education settings appear to skip these devices, unless they see an
immediate benefit. Therefore, this method might be less effective in terms of recall on
a long term basis (Clyde et al 1983; Marland et 'A 1990).

On the other hand, generative learning strateg es, such as concept mapping and note-
taking, are aimed at activating students to build on their previous knowledge and
construct a meaningful graphical and/or conceptual representation of the material to be
learned. Essentially, generative methods activate intellectual capabilities, such as the
ability to make observations through the use of inference (ie. to synthesize, predict and
explain events), thus moving beyond rote-learning and hopefully promoting meaningful
and reflective learning (Ausubel, discussed in Novak and Gowin 1984).

A survey of the literature provided the following points as guidelines for the hypotheses
tested in this study:

• although attrition is usually high, some students tend to leam generative strategies
with more ease than others; they either do not have a workable strategy (Novak
and Gowin 1984; Clark 1990; Dean and Kulhavey 1981) or they have a preference
for visual aids in learning (Feldsine 1987; Okebukola and Jegede 1988);

• students being presented with mathemagenic strategies tend to leam basic content
as well as students being presented with generative strategies (Novak and Gowin
1984; Miccinati 1988);

• students using generative learning strategies tend to do better on conceptually
hierarchical content (eg. biology) once the strategy is well practised, than students
being presented with mathemagenic learning devices (Moreira 1977; Dean and
Kulhavey 1981; Novak 1990; Feldsine 1987; Schmid & Telaro 1990).

INTRODUCTION

CONCEPT MAPPING

133

METHOD

Design, Subjects and Procedure

This study was carried out through the Department of Nursing at St Xavier University
in Antigonish, Nova Scotia, giving Nursing 200: Community Mental Health as a distance
education course in a four year programme. The participants were 141 female nursing
students in an obligatory 20 week distance education course, consisting of weekly
assignments, a midterm and a final exam.

The midterm and final exams were the dependent measures (DV). Both dependent
measures were separated into a set of multiple choice questions, and a set of essay
questions broken down into key-idea and related-idea units, both including mapping
versus non -mapping items. This technique produced five midterm and five final measures
- 10 dependent measures in all.

Treatment groups were assigned to geographical clusters, keeping the largest group for
the concept-mapping condition, because of the expected attrition from this condition.
Distribution of subjects was as follows: the control group having post-questions without
feedback (n = 22); the post-questions with feedback condition (n = 36); the combined
group (questions and mapping exercises) (n = 32); and the concept mapping with
feedback condition (n = 50). The concept-mapping group naturally separated itself into
high persistent mappers (n - 22; M=l 1 maps; SD=1.5) and low persistent mappers (n =
28; M=3.5 maps, SD=1.6). This bifurcation was used for statistical analyses and is
discussed in depth elsewhere (Bernard and Naidu 1992).

Materials

All students received the same instructional package, with an additional study guide that
differed according to the instructional strategy employed for that treatment condition.

Control Group

The control group received self-assessed tests with no feedback at the end of each
module, which was the normal procedure.

Post-Questions with Feedback

This group received the same self-assessed tests as the control group, but got feedback
on 12 of the 20 occasions.

The Combined Group

This group received the same training guide as the concept mapping group and the same
self-assessed tests as the two other groups. They were recommended to do 6 maps with
feedback and 6 self- assessed tests with feedback.

Concept Mapping

A training guide on concept mapping was provided. Students were encouraged to

139

134

QUALITY IN LEARNING AND ASSESSMENT

construct 12 maps during the course of the treatment, and received feedback on each map
completed.

Attitude Questionnaires

All students received a questionnaire evoking their views of the learning experience, as
well as a 5-point Likert type questionnaire consisting of 15 statements eliciting
information about their predominant style of studying.

RESULTS

Group Equivalence and Covariate

Group equivalence, including the two concept-mapping groups, was established through
an analysis of variance using a combined score from four previous exams (F(4,135)=.89,
p >.05). The covariate was found significant and was used to adjust the means for the
10 cognitive measures, both at the midterm and in the final exam. All scores were
converted into z-scores.

Cognitive Measures: Midterm and Final Exams

The omnibus MANCOVA produced a significant main effect for treatment,
F(40,494)=2.13, p<0.001. The Roy-Bargmann stepdown approach was chosen, since the
sequence of the tests were of importance (Bernard and Naidu 1992). This approach
produced four significant variables. At midterm, multiple choice items F(4, 134) =2.90,
p<.05) and key non-mapping items, F(4 ,131)=2.82, p=.G05 were significant. In the final
exam, key-mapping items F(4,128)=3.97, p<,01 and key non-mapping items
F(4,126)=2.98, p=.02 were significant.

Post- hoc analyses were conducted to determine which groups differed and these results
are shown in Figure 22.1. Interestingly, the high persistent concept-mapping group
performed significantly lower than all other groups on multiple choice items at midterm.
However, in the final exam this difference disappeared. This phenomenon is supported
in the literature and could partly be explained by the fact that students find it hard to
learn a new strategy, and therefore struggle with the strategy more than the content.

It is worth noting that the post-questions with feedback group performed better than the
control group, suggesting that feedback is a beneficial element with both mathemagenic
and generative strategies.

Furthermore, a significant difference was revealed in favour of the high persistent
concept mappers at both midterm and final exams on key non-mapping items. This
suggests that even for non-mapping items these students were enabled to relate more
information correctly than students in the other groups. For the key-mapping idea units,
the high persistent mappers produced better results than all other groups both at midterm
and in the final exam. This implies that recall of main ideas is learned more thoroughly
with concept mapping, maybe because students elaborate and spend more time on the
material, than when using mathemagenic strategies. Moreover, the combined group
outperformed the control group on key mapping items in the final exam, further
confirming the idea that concept mapping nelps recall for these types of cognitive
measures.

140

CONCEPT MAPPING

135

Measures

Kay-Non (F)
Qr3 > Grl/445

Qr4 > Grl
KtyMap (F)

Gr3> Gr4* Gr1

Below the mean

Key NonWap(M)

Gr3> Gr4,5 A Grl

Mult. Ch.(M)
Gr3< ali groups

Above the mean

-1 OA -0.6 -0.4 -0.2

FINAL

CoflM Group 1
UwCU Group 2
HghCU Qrou P 3
****** Group 4
GrOcp5

MIDTERM

0.2 0.4 0.6 0.8 1

Z-Scor»s (M410; SD-+-1 .0

Figure 22.1 Significant results from the cognitive measures at midterm and final
Attitudes after Experience

The factor analysis on the study method questionnaire included six factors that together
explained 64.8% of the variance. The variance accounted for by each of the six factors
is shown in Figure 22.2.

Factor scores (regression approach) were saved and a follow-up analysis revealed a
significant main effect for factors, Hotellings F(20,454)=2.02, p<.01 . Univariate contrasts
indicated that this difference resides in attitudes between the low and high persister
mapping groups (t=-2.79, p<.01) on Factor 2. None of the other factors or groups
produced any significant results. Figure 22.3 demonstrates these findings.

Six questions were constructed to elicit information from all students about their
attitudes towards self-assessed tests, since this is a mathemagenic device often used in DE
as well as traditional settings. No significant differences were found among groups.
However, it is interesting to note that 94% reported to have done most or all of these
exercises. Most (83%) verified all the answers, however only 63% reported that they
always reviewed incorrect answers, suggesting that post-questions are not always used as
intended by the instructional designer. Moreover. 48% of students found them difficult
in the beginning compared to 25% at the end. Only 36% found them time-consuming
and 67% found them useful to help them study.

136

QUALITY IN LEARNING AND ASSESSMENT

<**)

Pi*)

Fwtarl:

■

□

F**ort:

Factors (**•»%>

DWculty ramembertno; facts and (totalis
Preference for visual repraseotaioo of content

Ukos to understand content thourougty

Preference for k»y concepts

Ukas no la taking and memorization
Preference tor existing order of presontafon.

Figure 22.2 Variance explained by attitude factors about study methods

Factor s cor 9 (M-0;3d

07 -j

Facti

Fact.2

Fact.3

FacU

Fact.5

* significant differences

Groups
Control
H Low CM
| | High CM

Factors

Fifture 22.3 Significant difference between high and low mappers: preference for
visual aids in education (factor 2)

142

CONCEPT MAPPING

137

High and Low Mappers

As shown in Figure 22.3 low persistent mappers appear more negatively inclined towards
visual and graphical aids in learning new materials.

A closer look at these responses revealed that 65% of the high persisters found maps
provided by the instructor useful whereas only 45% of the low persisters thought so.
When asked how difficult concept-mapping was at the beginning of the experiment 62%
of the low persisters thought it was very difficult compared to 35% of the high persisters.
By the end of the experiment only 20% of the high persisters still reported it difficult,
with an encouraging 45% reporting concept mapping as quite easy.

The questionnaire also indicated that most low persisters found the exercises "very
difficult to do" (62%), "very time-consuming" (83%) and "not very useful"(75%),
whereas the high persisters still found them "difficult" (35%) and "time-consuming"
(45%). However, 50% of the high persisters reported that they thought of them as a
useful aid in learning new material compared to only 25% of the low persisters. When
asked whether they mostly preferred well tried out approaches to learning, 46% of the
low persisters agreed to this compared with 30% of the high persisters, indicating a
higher probability of willingness to acquire a new learning strategy.

Concept Mapping Group: High Low

Questions

Maps by instructor are helpful 65% 45%

Mapping very difficult in beginning of experiment 15% 50%

Mapping easy in end of experiment 45% -

Mapping is time-consuming 45% 83%

Mapping is useful when learning new material 50% 25%

I mostly prefer well tried out approaches to learning 30% 46%

Table 22.1 Summary of attitude questions

DISCUSSION

Perhaps the most important finding in this field experiment is that it is possible to teach
distant learners generative learning strategies that are beneficial for cognitive outcomes
on a long term basis. It also showed that mathemagenic and generative strategies are
beneficial for different types of outcomes, since high concept-mappers performed better
at key-idea items, whether they mapped them or not, than all the other groups. No
differences were found for multiple choice items at the final exam, suggesting that factual
or declarative knowledge can be gained by any strategy, but conceptual and procedural
knowledge such as that found in the sciences might be enhanced by concept-mapping as
a visual aid in learning (Moreira 1977; Feldsine 1987; Novak 1990; Schmid & Telaro
1990). In this study the mapping exercises were matched to elements in the course that
lend themselves to concept-mapping (ie. conceptually hierarchical knowledge as opposed
to declarative or fact remembering knowledge, Beukhof 1990). It could be speculated
that course content in general consists of both types of knowledge. Its implication for
material design might be to analyze content in order to pinpoint these differences.

143

138

QUALITY IN LEARNING AND ASSESSMENT

The other outcome confirms what the literature says (Clark 1990), that all learners do
not want to or can not adopt a self- activating, generative strategy. This conclusion is
supported by the high attrition rate of 56% in the concept-mapping condition as compared
with the mathemagenic conditions (post-questions .with feedback and combined group)
where no attrition at all was noticed. The results from the attitudinal instruments further
demonstrate that the difference for preferences of study methods seem to lie in the fact
that some students prefer visual and graphical aids (the high persisters), and a tendency
by the low mappers to prefer well tried out approaches. All other factors and groups did
not differ.

Since the aim was to examine the distance students* reactions to mathemagenic versus
generative learning strategies, it could be concluded that mathemagenic learning methods
can adopted by anyone, but that they do not necessarily optimize learning. Beudoin
(1990) expresses concern over the dependence that mathemagenic strategies create and
contends that distance learners must be helped to develop learner controlled learning
strategies. By contrast, a generative learning strategy such as concept-mapping appears
desirable in this respect, but has drawbacks when the fact that only some learners were
willing to acquire it, however successfully, is considered.

F jth in traditional and dis:ance education there is a tendency to view the learner as a
reactive rather than a proactive participant in the educational process. Expository
learning, mathemagenics (*othkop 1970), teaching machines/programmed learning
(Pressey 1927; Skinner 1958 . to mention only a few, are orientations that place emphasis
on the constructive- interpre* alive role of the teacher-designer. There is little doubt that
these perspectives are valuable, but they represent only one side of the educational
equation. The other side argues that empowering students with a set of learning tools (ie.
study skills and strategies) can provide both short term learning gain and long term
learning potential (Bruner 1966; Rigney 1978; Weinstein and Meyer 1986). This study
contributes to the idea that distance educators can design materials that will improve
learners study skills and strategies.

References

Bernard R M and Naidu S (1992) Post-Questioning, Concept Mapping and Feedback:
a distance education field experiment British Journal of Educational Technology 23(1)
pp 48-60

Beudoin M (1990) The Instructor's Changing Role in Distance Education The
American Journal of Distance Education 4(2) pp 21-29

Beukhof G (1990) Learner and Program Controlled Text Processing in H Mandle, E
De Corte, S N Gennett and H F Friedrich Learning and Instruction European Research
in an International Context (Vol 2:2) Analysis of Complex Skills and Complex
Knowledge Domains New York Pergamon Press

Bruner J S (1966) Towards a Theory of Instruction Norton New York

Clark R E (1990) When Teaching Kills Learning: research on matheinaihantics in H
Mandle, E De Corte, S N Gennett and H F Friedrich Learning and Instruction European
Research in an International Context (Vol 2:2) Analysis of Complex Skills and
Complex Knowledge Domains Pergamon Press New York

CONCEPT MAPPING 139

Clyde A , Crother H, Patching W, Putt I and Store R (1983) How Students Use
Distance Teaching Materials: an institutional study Distance Education 4(1) pp 4-26

Dean R S and Kulhavey R W (1981) Influence of Spatial Organization in Prose
Journal of Educational Psychology 73(1) pp 57-64

Feldsine J E Jr (1987) The Construction of Concepts Maps Facilitates the Learning of
General College Chemistry: a case study Unpublished doctoral dissertation Department
of Education Cornell University Ithaca NY

Hamaker C (1986) The Effects of Adjunct Questions on Prose Learning Review of
Educational Research 56 pp 212-242

Jonassen D H (1985) Generative Learning Versus Mathemagenic Control of Text
Processing in D H Jonassen (Ed.) The Technology of Text Educational Technology
Englewood Cliffs NJ

Marland P W, Patching W, Putt I and Putt R (1990) Distance Learners' Interactions
with Text while Studying Distance Education 11(1) pp 71-91

Miccinati J L (1988) Mapping the Terrain: connecting reading with academic writing
Journal of Reading 31(6) pp 542-552

Moreira M A (1977) An Ausubelian Approach to Physics Instruction: an experiment in
an introductory college course in electromagnetism Unpublished doctoral Dissertation
Department of Education Cornell University Ithace NY

Novak J D (1990) Concept Maps and Vee Diagrams: two meta-cognitive tools to
facilitate meaningful learning Instructional Science 19 pp 29-52

Novak J D. and Gowin D B (1984) Learning How to Uarn CUP Cambridge

Okebukola and Jegede (1988) Cognitive Preference and Learning Mode as Determinants
of Meaningful Learning through Conceptmapping Science Education 72(4) pp 489-500

Pressey S L (1927) A Machine for Automatic Teaching of Drill Material School and
Society 25 pp 549-552

Rigney J W (1978) Learning Strategics: a theoretical perspective in H F O'Neil Jr (Ed.)
Learning strategies (pp 165-205) Academic Press New York

RothkopEZ (1970) The Concept of Mathemagenic Activities Review of Educational
Research 40 pp 325-336

Schmid R F and Telaro G (1990) Concept Mapping as an Instructional Strategy for
High School Biology Journal of Educational Research 84(2) pp 78-85

Skinner B F (1958) Teaching Machines Science 128 pp 969-977

Wcinstein C E and Mayer R E (1986) The Teaching of Learning Strategies in M C
Wittrock (Ed) Handbook of Research on Teaching pp 315-327 MacMillan New York

145

23 Comparing chalk and cheese - quality in
assessing work-based learning

Caroline Bucklow, Coventry School of Art and Design, Coventry University, CV1

Assessing industrial placements as an integral part of an undergraduate degree is a
contentious issue. It is clear that industrial placements provide students with valuable
experiential learning which complements academic study, but the issues relating to
assessment of this type of learning are not easy to resolve. What aspects of students'
learning can we reasonably assess? Who should perform the assessment? What type of
assessment is appropriate? How do you define quality in assessment and how do you
ensure it? These are vital issues which affect the significance accorded to assessed
workplace-based learning, by students and employers alike.

The process of assessment can be critical in shaping a student's perception of the value
of workplace-based learning in relation to academic study. Assessment can increase the
credibility the student accords a range of skills developed through workplace-based
learning and this can deepen the student's involvement with their own learning process.
However, the beneficial results are only achievable if all the parties involved are
committed to maintaining quality, both in the learning experiences offered and in the
assessment process.

The workshop described in this paper provides an opportunity to explore issues of
quality in assessing workplace based learning through the medium of case study. The
cases are based on our experience in assessing workplace based learning for students on
the BA (Hons) Technical Communication course at Coventty University. The as*.essment
scheme used on the course has been developed in the light of research sponsored by the
Training Agency (as it was) and the CNAA. Participants in the workshop undertake two
tasks to help develop their own responses to the issues involved in maintaining quality
when assessing contrasting industrial placements.

The idea for the workshop grew out of discussions among members of the course team
for Coventry University's BA (Hons) Technical Communication degree, as they
developed the assessment scheme for an industrial placement programme. Although the
team was agreed on the need for objective assessment based on measurable outcomes, it
was only by engaging in a series of discussions about issues of quality that we were able
to agree on an assessment process.

During the discussions it became clear that the assessment process would have to
accommodate huge differences in the workplace context of individual industrial
placements and in the expectations different organisations had of the students.
Additionally, the team recognised that students, academic staff and employers might be
looking for different outcomes from the placement and felt it was important that the
assessment process would be sensitive to different viewpoints.

The process the team went through is represented, in a concentrated form, in the
workshop tasks. The role of the workshop facilitator is similar to that of the Training
Agency funded project officer who put together the assessment scheme currently in use
on the course. The workshop involves participants in two main tasks. The first task is

5FB, UK

INTRODUCTION

WORKSHOP

ASSESSING WORK-BASED LEARNING

141

to agree on the key quality criteria appropriate to assessing workplace-based learning.
The second task is to apply these criteria in developing an assessment scheme based on
case studies of student industrial placements. These activities offer an opportunity to
explore different aspects of quality in assessment and provide an opportunity to devise
different assessment schemes and to see how well they stand up to selected quality
criteria.

Task 1 - Identifying Priority Issues in Assessing Workplace-based
Learning (30 minutes)

For Task I the facilitator is probably best employed helping ail the participants to identify
with their roles and helping the groups to reach decisions.

Materials

Pieces of card (in three colours) , one for each participant
Writing materials

Objectives

By the end of the task participants should have:

• formed syndicate groups for task two

• discussed possible interpretations of quality from the viewpoint of student,
industrial mentor/supervisor and academic supervisor

• negotiated a set of quality criteria on which participants can draw for the main

The success of the task depends on participants' willingness to engage in role play.
Stage 1 (15 minutes)

Participants are each given a piece of coloured card. The colour of the card indicates the
role the participant takes in the discussion. There are three roles; student, academic
supervisor, and workplace supervisor/mentor. As far as possible participants should adopt
a role to which they are unaccustomed.
Participants form a group with others in the same role.

Individually, the participants list the key features they (as members of their role group)
would wish to include in an assessment scheme embodying the principles of quality.

The group members then exchange ideas and decide on a list of 'quality 01161™' they
wish to include in the assessment process.

Stage 2 (15 minutes)

The participants form new groups. This time each group includes participants
representing the three different roles. The task now is to negotiate a set of quality criteria
that takes into account the varying concerns and perspectives of the different role groups.
The intention here is to highlight the differences of opinion which may exist between the
roles and to explore ways of reconciling the different perspectives.

task.

142

QUALITY IN LEARNING AND ASSESSMENT

At the end of the allotted time the group must have a set of agreed criteria. Any feature
which is unacceptable to one or more of the roles must be modified to meet the original
objections or be discarded. These criteria can be displayed on flipchart paper so that the
groups can compare their own ideas with those of other groups.

Task 2 - Building Quality into the Assessment Process (45 minutes)

For this task the facilitator will probably need to be available to help the groups focus
on the task but should not be tempted to supply solutions to the issues raised.

Materials

Case study material describing three industrial placement situations with contrasting
characteristics (for further information about this material, contact the author).

Source material on good practice in assessing workplace learning (see References)

Writing materials

Flipchart paper and pens

Objectives

By the end of the task each syndicate group should have:

♦ identified key issues to be addressed when applying their agreed quality criteria
to diverse workplace learning situations

♦ decided on some approaches to assessment of workplace learning that meet their
own criteria for quality

♦ summarised their findings on a flip chart in poster format ready for presentation
to other syndicate groups

Introduction (15 minutes)

The facilitator introduces the individual cases, drawing attention to particularly
idiosyncratic features which might influence the assessment process. The facilitator
suggests features that might be brought out in the assessment scheme. These should
relate to the quality criteria identified in the first task. For example they might include:
setting objective success criteria, encouraging students to develop transferable skills,
emphasizing deep learning. The facilitator might provide examples of some approaches
used in existing workplace learning programmes. This task includes opportunities for
participants to raise questions about the case study material.

Main Activity (30 minutes)

Participants remain in the syndicate groups formed in stage 2 of the first task.

Each group reviews its quality criteria from Task 1 in the light of the need to devise
an assessment scheme that will work for each industrial placement described in the case
study material.

It is up to the group to decide how to approach the task. Some groups may wish to
concentrate on looking at the nature of the assessment process, some may wish to define
the appropriate outcomes of the process, some may wish to consider how to record the
results of the assessment to implement the quality criteria.

ASSESSING WORK-BASED LEARNING

143

It is not necessary for each group to develop a complete assessment scheme for this task
to be successful. Some groups may only be able to reach agreement on the outline
principles. Similarly, the case study material should be seen as suggestive of the range
of problems that may be encountered by people attempting to devise or implement such
schemes. It may be. appropriate for workshop participants to draw on examples from
their own experiences, either to supplement case study material or in the place of the
material.

The group can choose which pans of its deliberations it wishes to share with the other
groups.

Syndicate Groups Report Back (20 minutes)

Each group in turn shares its findings with the other groups. The time available will
typically be between five and ten minutes for each group, depending on the number of
syndicate groups.

Workshop Conclusions (15 minutes)

Each participant in turn is given the opportunity to comment on the success of the
workshop from their own perspective. The facilitator may offer general concluding
remarks if this is appropriate.

References

Coates H and Wright J (1991) The Integration of Work Based Learning with Academic
Assessment, Guidelines for good practice Coventry Polytechnic Coventry

24 Using learning contracts to enhance
the quality of work-based learning

Iain S Marshall and Margaret L Mill, Napier University, Edinburgh, EH14 1DJ,
UK

INTRODUCTION

Involving learners in the development of their learning contracts is proving a powerful
means of enhancing the quality of learning for undergraduates on sandwich col. es, and
for employees in small businesses. A learning contract is a formal agreement between
the learner, assessor and mentor, about what will be learned, and how that learning will
be achieved and assessed.

Quality learning which takes place in sandwich placement, or work experience, is
gaining full academic recognition at Napier University (formerly Napier Polytechnic) in
Edinburgh, where undergraduates are collaborating with teaching staff and employers to
develop learning contracts which earn academic credit towards their degree. The
Department of Employment art funding the refining of a stude-.t-driven model, which can
be used in a variety of institutional contexts (Figure 24.1).

Outcomes

Agreed Expectations

Agreed Criteria

Agreed Learning Outcomes

Negotiated Assessments

Credit for Learning Achieved

Figure 24.1 Model of student driven three-way learning contract

By encouraging the students to take a large measure of responsibility for identifying what
they want to learn from the experience of sandwich placements, the motivation of the
learner is harnessed to achieving agreed learning outcomes. The model encourages
ongoing re-negotiation by all three partners. Students are keen to turn learning which
they value into academic credit towards their honours degree.

Academic staff tend to find the focus on the outcomes of learning an alien concept, and
this identifies a need for appropriate staff development.

The Napier project involves students in each of the stages identified in Figure 24.2.

USING LEARNING CONTRACTS

145

Workshop meetings of students
with staff to prepare individual
Learning Plans

Student negotiates
Learning Plan
with Polytechnic Assessor

Students agree
Learning Plan
with Placement employer

Student in placement
organisation working
towards Learning Outcomes
Review and update Learning
Contract

Student returns to Polytechnic

with evidence of
Learning Outcomes achieved

Polytechnic assessment of
evidence of learning
presented by student/employer

Figure 24.2 Model of work-based learning process for students

Employers of our sandwich students, especially small and medium sized companies,
greatly appreciate students negotiating what they hope to learn from placement
experience, in advance. It saves time and money by avoiding inappropriate matching of
student and placement. Students are glad to have the chance to turn relevant learning
from experience, into real academic credit towards their degree.

Of course, this approach raises a whole range of staff development issues for the
university, as it shifts the role relationship between staff and student, towards greater
learner autonomy. However, a possible bonus to the academic institution is to
legitimately shorten the overall length of a course, by offering academic credit for a
previously "dead" period.

There are three good reasons why systems and procedures must be developed, which
allow students to ea' > credit which counts towards their degree, from work-based
learning:

146

QUALITY IN LEARNING AND ASSESSMENT

• In academic institutions there is a distinct tendency for resources to follow the
award of credit.

• Many of our student have an instrumental approach to course work. The award
of marks or credit influences the effort they are prepared to put into an
assignment.

• Where employers are participating actively in the process they express the view
that students should be rewarded for achieving relevant learning outcomes.

ASSESSMENT OF WORK-BASED LEARNING

Assessment of the evidence submitted by the student is central to the award of credible
academic credit.

In the learning contract model being developed in Napier University, the following
principles and conditions are emerging.

Some general principles:

1 That assessment be collaborative rather than unilateral.

2 That assessment be constructive and responsive to learners' needs.

3 That the conditions necessary for good assessment must be given a priority in the
allocation of time and other resources.

Some necessary conditions:

1 That there be clear learning outcomes which are agreed between learner and
assessor to be relevant to the qualifications sought and valued by the learner.

2 That the credit awarded for evidence of learning should fairly reflect the quality
of that learning.

3 That there be a policy regarding the nature of evidence of learning and a
readiness to explore different instruments of assessment.

4 That there should be mechanisms which take account of the views of the learner
and the employer in arriving at a collaborative assessment.

5 That there should be a policy for the professional development of teaching staff
and of employer 'mentors', in support of the woik-tased learner and in the
assessment of the evidence of learning which is submitted.

6 That there be a structure within an institution where the academic Board of
Studies can evaluate its success in achieving its own stated aims, and adjust its
practices appropriately.

The Learning Outcome

The fundamental element of the Napier model of work-based learning is the learning
outcome. By encouraging the learner to reflect on learning which they already have
achieved, or identify learning they will attempt to achieve, attention is paid to specifying
learning which is supported by adequate evidence.

Students are encouraged to group learning outcomes they plan to achieve from
placement under three headings:

Job Related Outcomes

Personal Development Outcomes

Course Related Outcomes

USING LEARNING CONTRACTS 147

During the planning phase prior to going on placement, the developing list of learning
outcomes is negotiated with staff in the university. The first major consideration is that
the learning outcomes should be judged to be meeting the aims of the degree. The
second is that the learning outcome is capable of being assessed.

Each student negotiates with a member of the university staff who will be the university
assessor of the evidence which the student will offer in support of the agreed learning
outcomes. The list of learning outcomes proposed by a student is used as part of the
agenda during placement interviews with potential employers. Invariably, further
negotiations occur which have to satisfy all three partners - the student, the company
mentor and the university assessor.

Throughout the placement experience, the list of learning outcomes is regularly
modified and renegotiated between all three partners to the learning contract. Before the
student is due to finish the placement, they confirm with the university assessor the list
of learning outcomes they wish to be assessed against.

The company mentor receives a print out of this list in the form of an assessment grid,
which asks the mentor to confirm whether or not the student in each case:

a Has indeed accomplished the learning outcome.

b Has provided evidence of understanding against a scale 1-4.

c Has provided evidence of adding value.

This assessment by the mentor is invariably discussed with the student, and sent to the
student's university assessor, who has his/her own assessment grid which is asking
similar questions of the portfolio of evidence the student submits in support of their
claimed learning outcomes, and arrives at a considered judgement as to the credit to be
awarded.

By developing professional judgement through experience gained with colleague
assessors and students, the question of quality and standards are continually reviewed and
refined.

New Pilot Project

A spin off from this work has been a new development of the model, which is being
supported through funding of another project by the Scottish Enterprise Company and
Lothian and Edinburgh Enterprise Limited (LEEL). The idea is to enable the Work-
Based Learning Unit to target a number of small business enterprises. The intention is
to identify training needs within the small businesses, negotiate those needs into an
agreed learning contract, and to use a variety of appropriate resources to meet these
needs, through flexible delivery mechanisms.

One of the innovative aspects of this new pilot project, is the involvement of small
businesses, in partnership with a major company, which has these smaller companies as
suppliers. A major benefit to the large company is the interest they have in the continued
well-being of their supplier companies, and this approach to total quality management is
enhanced by the customer/supplier partnership in identifying areas for improved
efficiency. It offers an opportunity for the sub-contractor to participate in an analysts of
their own training needs and to improve their efficiency by developing appropriate
management skills. .

All of this experience leads us to see the learning contract as a powerful mechanism tor
holding together: what has to be learned/demonstrated, how this will be achieved, and
how it can be assessed.

153

148

QUALITY IN LEARNING AND ASSESSMENT

ACCREDITATION OF WORK-BASED LEARNING

Napier University is moving towards a credit-based system, which will incorporate work-
based learning, but at present undergraduates still have to achieve a number of marks
from a variety of instruments of assessment, which determines their honours degree
classification.

A danger in the move toward a credit-based scheme, is that where units of credit are
based on units of time, ie 1 credit = 8 hours of appropriate activity, then the
discriminatory power of the model is lost. To be regained, it seems to us that we must
move on from the simplistic time-based unit of credit, toward an acknowledgement of the
relevant learning outcomes which can be realistically achieved and demonstrated by a
learner. This would allow students to earn more or fewer credits towards their degree,
by the way of work-based learning. Where courses are explicit about the learning
outcomes which are at present implicit within the syllabus, the issue of credit transfer,
based as it would be on credit for learning outcomes achieved, is greatly facilitated.

Moreover, the Napier experience is that this need be no more expensive than delivering
university based C.A.T.S. credit - and at a stroke, takes away the widely held view that
work-based learning has to be so resource intensive that it must be 'too expensive*.

By calculating the total number of units of C.A.T.S. credit available in an academic
year, and dividing by the total number of available academic staff hours, we arrive at an
efficiency ratio which shows that on average each available staff hour can generate 1.2
units of credit for students.

Total F.T.E's x 120 Units of Credit = 1 .2 Units of Credit per

Total number of available academic staff hours available staff hour

When we look at the units of credit available from 'sandwich' placement, and use the
figures from our project to estimate staff hours necessary to support and assess this
number of credits, the resulting efficiency ratio is 1.9 units of credit per staff hour.

Units of available credit x student numbers = 1.9 Units of Credit per

Total staff hours in napier work-based learning model available staff hour

Finally, the conclusion reached by the staff at Napier University is that the award of
academic credit for evidence of relevant learning is the engine which drives the process
for undergraduates. For people in companies, it is the recognition that what they are
learning is of benefit to them in the work that they do. The challenge confronting us is
to continue to refine appropriate systems and procedures for supporting relevant learning.
Anyone interested in learning more about this approach is encouraged to contact the
authors at the Work-Based Learning Unit, Napier University, 219 Colinton Road,
Edinburgh EH14 1DJ, Telephone: 031 455-4390.

References

Boak G (1990) Developing Management Competences Pitman London

Carr and Kcmmis (1983) Becoming Critical - Knowing Through Action Research
Deaking University Press Geelong

Knowles M S (1986) Using Learning Contracts Jossey-Bass San Francisco

154

25 Quality of assessment

Phil Race, University of Glamorgan, Pontypridd, CF37 1DL, UK
INTRODUCTION

Much of this volume is about the assessment of quality - this paper is about the quality
of assessment. What our students really pay for (or what society pays us for ) are their
degrees or their diplomas or their certificates. Yet we put much more emphasis in trying
to teach our students than we normally put into designing assessment - particularly the
"formal assessment" that counts towards final qualifications. Assessment is where
learners often get a raw deal.

The 1991 AETT Conference had the theme "Developing And Measuring Competence".
As one of the editors of the proceedings of that conference (Saunders and Race 1992),
I became well acquainted with the many views on competence which were aired by the
various contributors. It was heartening to see that many of the approaches to the
development of competence focused on defining the evidence which would be sufficient
to demonstrate competence.

However, I was left with some concerns about whether competence would often end up
as minimum competence rather than high competence. Figures 25.1 and 25.2 pose some
questions about "shades" of competence, and about a range of descriptors which may be
needed to provide more information than simply "can do".

Figure 25.1 Shades of competence

150

QUALITY IN LEARNING AND ASSESSMENT

excellently

distinguishedly

innovatively

Figure 25.2 Some competence descriptors

Higher education in the United Kingdom is being moved steadily towards greater
participation rates. This move includes not only rapidly increasing numbers of school
leavers, but also many more non-traditional entrants. Everywhere, lecturers and tutors
are seeking help in handling large classes. It is clear that the workload of staff in higher
education is increasing, as they become responsible for more and more learners.

In some ways, teaching can be "done to" large numbers almost as easily as to small
numbers - it's almost as easy to lecture to 300 as to lecture to 100. However, teaching
greater numbers may have little connection with learning by greater numbers. And when
it comes to assessment, there are no short cuts. It usually takes three times the amount
of time to assess 300 learners as it would to assess 100 learners. The main challenge we
face as we move towards a mass higher education system, is merely maintaining the
quality of assessment we presently have - not to mention improving it.

HOW MOST PEOPLE LEARN

First, consider how people learn - real people, that is. not educational psychologists or
their learners! During the last couple of years, I've given thousands of people post-its,
and asked them to write down the answers to the following questions on them.

I. Think of something you're good at - something you do well. Write down a few
words about how you became good at it.

155

QUALITY OF ASSESSMENT

151

2. Think of something ywfeel good about - something that gives you a "glow". Write
down a few words explaining your evidence for this good feeling.

3. Think of a learning experience that was not successful. Write down a few words
explaining what went wrong.

The answers to "1" are almost always along the following lines:

♦ doing it

♦ practice

♦ trial and error

♦ learning from mistakes.

In other words, most people learn by doing rather than by sitting at the feet of masters
or mistresses.

The answers to "2" are usually along the following lines:

♦ feedback from others

♦ other people's reactions

♦ because I can see the effect.

In other words, to feel good about something (including learning) , people need feedback.

The answers to "3" are more complex. Sometimes they reflect things that went wrong
with the "learning by doing" stage - for example lack of opportunity to practice.
Sometimes they reflect things that went wrong with the feedback stage - in other words
lack of the opportunity to develop good feelings about what had been learned. Two more
problems often emerge:

♦ lack of motivation in the first place

♦ lack of opportunity to make sense of the learning experience.
LINKING TRADITIONAL ASSESSMENT TO LEARNING

So how does assessment usually relate to the four fundamental steps in the learning
process? Let's take the most common form of assessment - the written examination.

♦ Wanting to learn (motivation)

Not many people like exams! The fact that there is an exam coming along at the end
of the road is not the strongest motivator for most people. The inevitability of
traditional forms of assessment is a key factor in preventing many people from
participating in learning.

♦ Learning by doing (practice, learning from mistakes, and so on)

Learning by doing can indeed happen during assessments, including written exams.
Mistakes are indeed made during assessment - plenty of them! But learning by doing
while being assessed is hardly the best way of using experiential learning. Besides,
it is usually presumed that the learning should have taken place before the assessment
event.

♦ Learning through feedback (to develop positive feelings about the learning)

All assessment results in some sort of feedback. However, it's often the absolute
minimum of feedback - for example a mere score - and that's weeks or months after
the event! There is little or no real feedback, and chances to learn from the feedback
are minimal.

157

152

QUALITY IN LEARNING AND ASSESSMENT

• Digesting (taking stock, making sense of the experience, and of the feedback)

Exams are better known for producing indigestion than for allowing people the chance
to consolidate their learning. As I've said, the feedback is usually very limited in
scope (and often delayed in time) and is not a useful means towards "digesting".
Such feedback as there is, tends to be one-way feedback. There's or no little chance
to discuss the details or negotiate what best to do next.

At each step, assessment processes are at cross purposes with learning processes. Perhaps
traditional forms of assessment have only one real contribution towards learning- people
are frightened (shamed) into doing some learning so that they may minimise their chances
of being shown to be "lacking". Much intensive learning is done just before exams - but
most of it is of a superficial nature, and soon forgotten again.

TEN WORRIES - AND SOME SUGGESTIONS

I list below ten worries I have about assessment, outline my concerns, and offer some
suggestions about how the problems may be minimised. Many of my suggestions point
towards involving learners in their own assessment. This can be achieved by letting
learners get their hands on the assessment criteria. It can be achieved even better by
letting learners apply the assessment criteria, in self -assessment and peer- assessment. It
is achieved best by helping learners to formulate the criteria, then apply them. An
illustrated discussion of uses of self- and peer-assessment is included later in this paper.

1 . Assessment is often done in a rush, to meet exam board deadlines. It's rarely done
under the best of conditions!

This is because assessment tends to be done to learners - not by them. Assessment tends
to be done at the end of learning something, rather than as a means to help the learning
processes. In public exams, examiners often face piles of some hundreds of scripts, all
of which need to be finished within only a week or two. Some suggestions for avoiding
the problem are as follows:

• allow much more time for self-assessment, so that it can be done well

• allow learners to use self- and peer- assessment, so they can learn by assessing.

2. Assessment is often done by bored people, tired of reading the same answers to the
same questions (and seeing the same mistakes).

Examiners get thoroughly fed-up as they wade through hundreds of scripts. They get
discouraged when they see things they hoped their learners would have mastered, only
to find thai messages have not got across. Any tedious or repetitive task causes people
to change their mood. If assessors' moods plunge, the objectivity of assessment is likely
to be affected accordingly.

• decrease the emphasis on traditional written exams altogether

• allow learners to learn by reading their own mistakes, and those of their peers.

3. Assessment tends to be governed by "what is easy to 3ssess u . Therefore, traditional
written exams (relatively straightforward to assess) are used. These measure learners'
skills at tackling traditional written exams.

QUALITY OF ASSESSMENT

153

How much you know?

How much, you don't know?

How fast you can write?

How good your memory is?

How much work you did the night before?

How well you keep your cool?

How competent you are?

How well you can read the questions?

How good you are at answering exam
questions?

How practised you are at answering exam
questions?

How you perform under pressure?

How good you are at time management?

How well you can keep addressing the
question?

How often you've practised on similar I
questions? I

How well you read your own answers? |
Figure 25.3 What exams measure?

There is still not enough attention being paid to what should constitute the evidence upon
which to base awards. Many important competences are simply not assessable by
traditional methods. While it is perfectly possible to use traditional methods to measure
recall of facts and information, it is not-at-all easy to use such methods to measure
innovation, judgement, or personality. Figure 25.3 shows an overhead transparency that
I use to alert students to the various agendas that may be served by traditional exams.

• look carefully at what is being measured by each form of assessment

• refrain from measuring the same things all the time - especially recall (people who
can find and apply information are usually more valuable than people who simply
happen to remember a lot of it).

4. Learners rarely know the intimate details of the assessment criteria, and how we
interpret them.

There really is no excuse for this. The reason may be sinister - that those who design
the assessment criteria are not sufficiently confident about them to show them to the
learners! Assessors often fear that learners may demand to know "why did I get 65% for
this, when my friend got 75%?" Surely, they have every right to ask this sort of question
- and to learn from the feedback they should be given by way of a response to the
question.

• give learners the opportunity not only to see the intimate details of assessment
criteria, but also to use the criteria.

153

154

QUALITY IN LEARNING AND ASSESSMENT

5. How should we develop learners' unassessable qualities? Should we refrain from
developing them because we can't measure them?

"Don't bother to learn anything, when you can't see how they can ask you about it at the
end of the day": this is a perfectly rational view taken by learners, deciding what to learn
and what not to learn.

• bring the unassessable qualities firmly onto the agenda; explain to learners why
they're important, and work out with learners what kinds of evidence can be
linked to these qualities, and how the demonstration of that evidence can be built
in to assessment procedures.

6. Almost all assessment processes in common use foster learner competition rather than
collaboration. No wonder our educated people are so bad at working in teams.

Learners preparing for exams are often quite secretive about the work they do. No-one
likes to be thought of as a "swot"! However, it's more sinister than this: we actually
compound the competition by using norm-referenced assessment far too much. In other
words, only a certain proportion of learners are allowed to receive "A" grades, or 1st-
class honours degree classifications. Therefore, learners are in competition.

• use criterion -referenced assessment only - abolish the use of norm-referencing

• help learners to feel that they can help each other prepare to demonstrate their
competence, without disadvantaging one another.

7. What competences are measured by assessment anyway? Are they "can do"
competences? Or are they simply "did do, once" ghosts?

Exams tend to measure "did once" competences. At their worst, they still remember
"knew once" competences rather than "did once"!

• increase the proportion of assessment schedules allocated to continuous assessment
- which measures "is doing" competences rather than "did once" ones.

8. If we were to introduce "Quality in Assessment BS5750A": what should the criteria
look like? What evidence of competence should assessors demonstrate?

At present, it is automatically assumed that anyone appointed to a post involving teaching
or lecturing is blessed with all the skills needed to design assessment schemes and
implement fair assessment. People are appointed to teaching (and assessing) posts not
on the basis of how well they can do either task, but often on the record of their own
academic performance.

• assess the assessors - have a system of "licences" to assess, and police the system
thoroughly

• increase the uses of self-assessment and peer-assessment, which depend far less
on subjectivity of assessors, and allow far greater amounts of feedback to
contribute towards successful learning experiences.

9. "If you can't measure it, it doesn't exist. If you can measure it, it isn't it". What
should we be trying to measure?

It has been said that one of the main faults of our education and training systems is that
we tend to teach people things that are already understood, instead of equipping them to
understand new things. Assessment reflects this.

QUALITY OF ASSESSMENT

155

• use self- and peer-assessment as an inherent part of learning processes, with the
emphasis on learning rather than assessment outcomes.

lO.Where stops the buck? Whose fault is it that assessment is so artificial? HoD?
Employers? Assessors? Validators? The Government? Yours? Mine?

If you imply that there is something suspect about people's abilities to assess, it is badly
received! Assessment is something that is usually done privately rather than publicly, and
people go to great lengths to ensure that they retain privacy. Is such privacy really
needed mainly because of the "put down the number you first thought of" syndrome?

LEARNING THROUGH ASSESSING

In much of my discussion so far, I've been focusing on the dangers when assessment is
"done to" people, and hinting at the benefits which can result when learners themselves
are allowed to be intimately involved in their own - and each other's - assessment. Self-
assessment, and peer-assessment may lack some of the precision of the best of "formal"
assessment - where (some) assessors have a great deal of experience, and (sometimes)
assess fairly and conscientiously. However, what may be lacked in terms of precision
is more than compensated for by the benefits of deeper learning, which go hand in hand
with the act of learners themselves assessing.

Close Encounters with Assessment Criteria

This is the crucial difference between formal assessment, self -assessment and peer-
assessment. Learners find out a lot about any subject simply by applying assessment
criteria to examples of work in that subject (whether the examples are self-generated,
made by other learners or issued by a teacher). Previously, assessment criteria have
seemed to learners to be the property of examiners. There has been a tendency for
teachers to regard assessment criteria as quite private. For many years, I marked "O u -
level and "A" -level scripts for two of England's public examination boards, and the
marking schemes were invariably labelled "secret". Even where model answers and
marking schemes have been required to be sent to external examiners or moderators, the
vital information in such schemes has seldom been shared with learners, and until
recently hardly ever applied by learners themselves. Yet when learners get the chance
to get their hands on assessment criteria, they seem to develop a thirst for the information
they can derive from them - leading to much deeper learning.

Self-assessment and Peer-assessment Are Not Just Self-testing

These forms of assessment when well -developed involve two processes:

• involving learners in identifying standards or criteria to apply to their work

• allowing learners to make judgements about the extent to which they have met
these standards and criteria.

Assessment Criteria: Black and White or Shades of Grey?

In subjects like maths, science and engineering, things are often either right or wrong -
and it is relatively easy to devise assessment criteria for tests and exercises. However,

161

156

QUALITY IN LEARNING AND ASSESSMENT

even in subjects such as law or social studies, there are identifiable hallmarks of a good
answer or an unsatisfactory answer to a question. Such hallmarks can be turned into
checklists of a flexible kind, which enable the characteristics of good and less-good
answer to be compared and contrasted. Students can benefit by learning in the act of
applying assessment criteria to their own work, and to each others* work.

Benefits to Learners of Close Encounters with Assessment Criteria

Learners can quickly find out about incorrect assumptions they have been making. They
are able to find out the answer to the question: "What am I expected to become able to
do?" There are of course many more benefits, depending on how we involve learners in
using assessment criteria - including helping learners themselves to formulate the criteria
(when this is possible or appropriate) - leading to the most obvious form of ownership
of assessment.

Some Examples of Self- and Peer-assessment Mechanisms

Self assessment is not confined to the variety that is widely used in open and distance
learning (though of course that is one powerful form of it). Self assessment processes
can include any of the following:

• providing learners with assessment criteria and a marking scheme and allowing
them to mark their own work

• as above, but then allowing learners the chance to compare their mark with that
of a "professional" marker

• as above, but also giving learners feedback about the quality of their self
assessment

• enabling individual learners to generate assessment criteria, and use them to assess
their own work

• enabling a group of learners to generate assessment criteria, and so on

• allowing learners to use core criteria generated by a group, plus additional criteria
specific to their own pieces of work, with an agreed weighting

• groups of learners can be issued with criteria to apply to each others* work

• groups of learners could produce criteria and apply them to each others' work.

There are further combinations of these. There is also the additional matter of whether
the grades or scores contribute in a formal way to the performance records of learners.

Eliciting Assessment Criteria from a Group

I have found that the following approach gives useful results for groups of 10-20.

• I ask each learner to privately list (eg.) six things you would expect of a good V
(where V could be an essay or presentation or handout and so on).

• I ask learners to go to groups of 3 or 4, and discuss criteria.

• I ask the groups to make a list of criteria, and to prioritise them.

• I flipchart the most important criteria from each group, then the next most
important, etc.

• I ask the whole group whether anything important is missing from the flipchart
list.

• I tidy up the flipcharted items if necessary, and number them eg. 1-8.

• I ask each learner to privately distribute (eg. 20) "marks" among the criteria.

162

QUALITY OF ASSESSMENT

157

• I write each learner's "mark" alongside each criterion on the flipchart. Then I
either average them out, or allow each learner to apply his/her own weightings in
the peer assessments to follow.

• Learners then go off to perform the task (individuals or groups).

• I then produce a grid with their criteria and weightings, ready for peer assessment.

An Example

I helped a group of students from a foundation course in Science, Engineering and
Technology to generate and prioritise some simple criteria for a short presentation that
each student was to give to the group. (The presentations were optional; the assessment
for the "Learning Strategies" module of the course contained six equal elements, and
students could choose which five they tried - in other words they could choose not to
give the presentation if they really wished to). We agreed that anyone who had the
courage to get up and start immediately deserved to "pass" - in other words they got the
first 40 marks there and then. Seven criteria were then devised by the students to account
for the remaining 60 marks, and the peer-assessment grids became as shown in Figure
25.4.

\ Criteria

Weight

1 confidence

^ staying on
topic

3 well

illustrated

4 well

researched

^ handouts
overheads

1 0

® dealing with
questions

^ timing

1 0

starting

^^Total

100

Figure 25.4 Peer-assessment grid for presentations

163

158

QUALITY IN LEARNING AND ASSESSMENT

Each student filled in a grid for each of the presentations he or she witnessed (and
students additionally self-assessed their own using the same criteria). For various
reasons, some students were only able to be present for half of the time involved, and one
or two students participated as assessors, but did not wish to give presentations
themselves. I myself worked as one of the assessors, and was relieved to find that in
general, the marks I awarded each of the presentations were close to the average mark
for that presentation.

Learners' Ownership of Assessment Criteria

The sort of peer-assessment described above is suitable for tasks such as presentations,
where many people can assess the same piece of evidence, and where scores can then be
compared and discussed by the group. For individual tasks such as essays, reports,
projects, dissertations and so on, it is likely that each piece of work will reflect slightly
different criteria (or even very different criteria), and then it is often best to allow for
some "agreed" criteria, and some "idiosyncratic" criteria so that each learner can exercise
more ownership of the assessment criteria.

The most important outcome of involving learners in the formulation of self -assessment
or peer- assessment criteria, is that learners address the task with criteria in their minds,
and the quality of their work seems to be much higher than it may otherwise have been.

Throw Away the Numbers or Grades?

I've often suggested to learners after a peer assessment exercise that the numbers or
grades awarded (if not contributing to their overall assessment) were only a vehicle to
help them do learning of a higher productivity. However, I've found they usually want
to hang on to the numbers - good or bad. Perhaps this is evidence of the sort of
ownership we* re aiming for?

How Well Can Students Assess Themselves, and Each Other?

In general, students are quite accurate in their assessing. I have found that when students
are asked to "guess" their own performance scores just after completing an exam, around
90% of students "guess" within 5% of their actual scores. It is useful to identify the 10%
who had an inaccurate perception of how they had done - they usually benefit from a
discussion to probe the causes. Those 10% may be over anxious, and underestimate their
achievements, or over-confident and over-estimate their achievements. When
discrepancies in self- assessment occur, they are usually due to one of the following
causes:

• there is some tendency for learners lo over- rate themselves in areas to which they
are new - this tends to happen with the weaker members of the group

• there is a tendency for some learners to under-rate themselves in areas in which
they are experienced - this tends to happen with the more-skilled members of the
group.

Peer-assessment and self -assessment can be usefully combined. Peer-assessment can be
conducted "blind" so that "arranged" scoring is avoided. If the peer- assessment mark or
grade is equal to the corresponding self- assessment mark (e.g. within 5%) then the self-
assessment marks go forward into the assessment system - possibly with a staff "scan"
to ensure that fair play is in operation. Ot is far quicker to scan a piece of work to check

QUALITY OF ASSESSMENT

159

whether the assessment is fair, than it is to mark the work from scratch). When self- and
peer-scores differ, negotiation or staff intervention may be necessary (but this happens
surprisingly rarely in practice),

SELF-ASSESSMENT, PEER-ASSESSMENT AND LEARNING

Throughout this paper, I've advocated the benefits of helping learners become intimately
involved in processes of assessing, and I've pointed to the hazards of traditional
assessment procedures. To conclude, lets look once again at the four main processes of
learning, and see how self- and peer- assessment link to them.

• Wanting to learn (motivation)

Motivation can be improved by early success. Self-assessment in particular can be
used with the comfort of privacy, and learners can gain confidence by finding that
they are "doing alright" long before they need to prove so publicly or formally.

• Learning by doing (practice, learning from mistakes, and so on)

There's no better way to find out about one's successes and failures than by finding
them out for oneself - or having a peer help one do this, rather than an "authority
figure" like a tutor or examiner. The very act of assessing is intrinsically "learning
by doing" - it involves the application of criteria, decision-making, judgement, and
reflection.

• Learning through feedback (to develop positive feelings about the learning)
The showstopper of formal assessment has to be the dreadfully limited feedback that
is the norm. Peer-assessment can allow for a great deal of feedback - far more than
could ever be given by a tutor or assessor. In addition, the feedback gained in peer-
assessment is usually far less threatening than that from "professionar 1 assessors.
Indeed, peers will often argue and debate issues, further deepening the usefulness of
the feedback exchanges they receive.

• Digesting (taking stock, making sense of the experience, and of the feedback)
Both self- assessment and peer-assessment can help learners make sense of their
learning experiences - and of the feedback they gain. Furthermore, the time lag
between the learning and the feedback can be much less than with traditional methods
of assessment. Therefore, the feedback is much more actively received and the
learning thereby enhanced.

Reflecting on Assessment

It can be argued that people who need a "tester" are inadequately prepared to be sent out
into the world outside. Self-assessment and peer-assessment can both be an important
part of the learning process. The learning experience resulting from such forms of
assessment is more important than the result of the assessment. SeJf-assessment or peer-
assessment do not necessarily have to lead to any "formal" (recorded) assessment. The
aim can be purely as a learning experience, with the "marks" simply part of the process
through which that experience is facilitated.

Self- assessment and peer- assessment arc skills, and become more reliable with practice.
Receiving feedback on the quality of these forms of assessment is vital if learners are to
derive the maximum benefit from engaging in them.

Self -assessment and peer-assessment should be introduced early - for example during
the first term rather than being left until the final year. Late in a course, students may
see little point in embracing new ways of learning.

165

160

QUALITY IN LEARNING AND ASSESSMENT

"Ownership" is the most crucial aspect of successful learning, and both self-assessment
and peer-assessment are closely connected to the development of ownership of learning.

Not all students warm to the "exposure" of self- or peer-assessment. They may begin
their studies with expectations that they will be assessed by professionals. "What's in it
for me?" they naturally may ask. They need to be convinced that self- and peer-
assessment have direct benefits for themselves, and do not represent an abdication from
duties on the part of tutors. Some tutors however feel it is dangerous to "lose control"
of assessment. If such tutors try to employ self- or peer-assessment, but constantly
safeguard their right to step in "should things go wrong", the whole concept of such
forms of assessment is undermined.

So what about teaching? Admittedly, our students learn from us. But they probably
learn more on their own, and they probably learn even more from each other. Much of
their learning occurs in the immediate run-up to assessment of one kind or another - so
the role of assessment is an important factor in the circumstances which accelerate
learning. Therefore, perhaps our biggest contribution to our students' learning is directly
associated with the quality of the assessment our students encounter - and has less than
we'd like to think to do with our teaching activities. I believe there is a strong case for
using self- and peer-assessment not primarily to assess, but as processes to enhance
learning.

Finally, if and when we must resort to traditional, formal assessment, I believe that
there is a great deal of room for improvement. The "ten worries" I expressed earlier may
help to set an agenda for improving the quality of assessment. I end this paper with 6
recommendations.

WHAT CAN WE DO TO IMPROVE ASSESSMENT?

• Extend and develop the competence-assessment approach, where people
accumulate evidence of the things they do, and when they're ready, bring the
evidence forward for assessment.

• Use assess ment as a learning opportunity. Give people feedback on their
performance, and help to improve it. Make the primary purpose of assessment
that of helping people find out how their learning is going, so that they can adjust
their strategies accordingly.

• Use assessing as a learning process - help learners to self- assess their own work,
and peer- assess their colleagues' work (whether or not the marks or grades are to
"count"). Let learners gain familiarity with the nature of the assessment criteria
which they need to live up to. Help them find out what sort of performance they
need to be able to give.

• Change the culture where any professional can sit and scribble down an exam
qu* ion in a few minutes, to a culture where all important exams are made up
entirely of questions which have been piloted, tested, evaluated, and proved to
measure desirable things.

• Develop team approaches to question setting, and marking. Introduce at the very
least double-marking as a standard for exams - and when possible multiple
m.vking.

• Pay people well for assessing, instead of just paying them for teaching. Teachers
often earn "holiday money" by doing extra exam-marking. It's all the wrong way
round! Assessment is so important that perhaps it should be that assessors can
earn "holiday money" by doing a little teaching! Then require them to
continuously demonstrate their competence at the job of assessing.

QUALITY OF ASSESSMENT

161

References

Saunders D and Race P (Eds) (1992) Developing and Measuring Competence Aspects
of Educational and Training Technology XXV Kogan Page London

26 BS5750 for assessment

Workshop led by Phil Race, University of Glamorgan, Pontypridd, CF37 1DL, UK
INTRODUCTION

The following abstract had been published in the Conference documentation: "The
workshop will be a participative event, sharing the good (and bad) experiences of
participants - both of assessing - and of being assessed. The workshop will begin by
extending the 'ten worries about assessment' presented in the associated paper (chapter
25 in this volume). Syndicates will examine ways and means for overcoming the
problems. The primary aim of the workshop is to generate a set of 'quality criteria
which can be applied to assessment processes, practices, and instruments, to guarantee
learners a fair and just assessment".

The aims and objectives of the workshop were:

• to pinpoint the weaknesses in assessment practices and devices

• to explore alternative ways of assessing

• to enhance the learning which can be derived from assessment

• to produce "quality criteria" for assessment.

PARTICIPANTS' EXPECTATIONS

Participants were given a small piece of acetate and asked to write on it their answers to
the question: "what do you personally hope to gain from this workshop?' . This aUowed
the workshop to address particular issues which participants had brought to the woii^hop.
The acetates were displayed to the group, and clarification encouraged where needed.
Their expectations were as follows:

• Where assessment fits into BS5750 (Nick Owen)

• To get an idea of how BS5750 compares with other possible models (Chanid)

• To use BS5750 creatively (Jill)

• To highlight quality criteria (Alec Bickerton)

• Relating BS5750 to assesn-nent, specifically quality manual documentation

(Michael) .

• How to improve assessment - ideas, experiences of others (David)

• When and how it can go wrong (David)

• Further ammunition and ideas for reforming assessment (Bob Sawyer)

• Rationale for justifying democratic versus traditional assessment for use in the
context of a traditional organisation (Jeremy Arter) m

• Identifying appropriate means of assessing (non-teaching background) (fatncia)

• Criteria for designing quality assessment; ideas on using computers for continuous
assessment (Chin-Poon) .

• Ideas to help colleagues to more effective assessment with less quantity ot work
(Exam board activity can surely be just as effective with fewer numbers to
crunch!) (Julie Clarke, a registrar) m

• Why BS5750 is in the title but not mentioned in the abstract (Chris U Haganj me

found out!) i e % * (

• Clear ideas about quality assessment; how to avoid unhelpful assessment ot

students' work (Christine)

• New insights into the use of assessment (Paul Ellis)

16C

BS5750 FOR ASSESSMENT

163

♦ Changing students' expectations to accept student-centred assessment (Larry
Roberts)

♦ How far should courses be "assessment driven"? The relationship between
assessment and outcomes. How far should the teaching and learning strategy
reflect the assessment strategy? (Steve Leary)

♦ Ideas (Andrew)

♦ Ideas to use in a course I am designing where I might be able to include diagnosis
as a form of student learning (Rachel)

♦ Some ideas on BS5750 criteria for assessment (Jill)

♦ Ideas, concepts to be of use with FE colleagues wrestling with problems of
assessment (Jack Oakley)

♦ Find out a little about how FE/HE have initiated changes/improvements in
assessment procedures, and how this is linked to 5750 (Peter Leckstein)

These expectations are of considerable interest in their own right. They illustrate the
many different ways that people view assessment. Several expectations mentioned
BS5750 in particular. The facilitator "came clean" at once on this, and indicated that the
purpose of the workshop was to generate information which would lead to criteria for
high quality in assessment, rather than look at how existing BS5750 documentation may
lend itself to applications to assessment.

Many participants had already been present when Phil Race ran his earlier Conference
session "Quality of assessment", and he issued further copies of his paper with that title.
He summarised quickly how learning is dependent on the four processes of "wanting",
"doing", "feedback", and "digesting" (as discussed in detail in the paper), and reminded
participants that one of the intentions of the workshop was to explore how best
assessment could be designed and implemented to assist these processes of learning.

FITNESS FOR PURPOSE

Participants were divided into syndicates, and briefed as follows to consider particular
types of assessment, and prepare an overhead transparency addressing each of the
following issues:

SYNDICATE TASK I

Choose a type of assessment

♦ decide exactly what it actually assesses best

♦ discuss how it helps the processes of learning

♦ list some advantages and disadvantages of the type of assessment.

Figure 26.1 consists of transcripts of the acetates produced by the syndicates, which add
up to an interesting exploration of a number of different possible processes of assessment.

INDIVIDUAL RECOLLECTIONS OF ASSESSMENT

Participants were issued with post-it slips, and asked several questions about the last
assessment they themselves had experienced as learners.

Their replies are transcribed in Table 26.1. They show a fascinating range of feelings
about assessment, attitudes towards the results of assessment, and views about what (if
anything) was learned from the various experiences of being assessed.

163

164

QUALITY IN LEARNING AND ASSESSMENT

1 ROLE PLAYING

What? attitudes

procedural behaviour

skills (Interpersonal)

(problem solving) (self awareness)

applying theory

HOW it help$ • confidence boosting

• practice

• experiential learning (remembered)

• time effective

MU V a N Id y t?£>

Prnhlomc

• build a profile

• student reluctance

" yUlUc IU \AJ\ I IJjyiol IOC

cyu iiiyj tut ouiiic

• nper learnino

UVwl IVUl 1 III IU

• not taken seriously

• offopfivo Ipprninn

CIICUUVC ICCUIllliy

ui ii ecu

• instant feedback

• traumatic, subjective

2 ORAL PRESENTATION

mm %^ 1 1 * \ mm I II IhWmI V 1 # 1 I 1 I ^

- on bphalf of s nrniin

\j i J i^v* Mull \j\ ci yi ^Um

What it assesses

Group Work

Individual

• content against objectives

• presentation style

• design and organisation

• voice

• time management

• use of audio-visual aids

• visual aids

• use of sources

• ability to answer questions

How it helps learning

• immediate feedback

• group learning

• peer tutoring

• peer assessment

Advantages

Problems

• validity

• time consuming

• good learning environment

• free riders

• design consistency

3 SELF-ASSESSMENT

What it assesses

An example: bricklayers and total quality management programme

How it helps learning

ownership

involvement

Advantages Disadvantages

• involvement leads to motivation

how will they judge for themselves

• realisation of responsibility

their ability to implement their learning?

• focuses responsibility

cost of establishing competence criteria

• another ethic throughout

being adhered to?

the organisation

policing outcomes

abdication of responsibility?

honesty? cover up mistakes?

BS5750 FOR ASSESSMENT

165

4 STUDENT SELF-ASSESSMENT

Examples: portfolio of work, exhibitions,

microteaching, teaching a lesson

What it assesses

• students' perceptions of their own learning

• ail areas (Bloom)

How it helps learning

• leads to learner autonomy

Advantages Disadvantages

• reflective • abuse?

• diagnostic • subjectivity

• developmental • false humility

• load taken from tutor • subversive

5 PERFORMANCE ASSESSMENT

What is assessed?

conformity to norms: job-related criteria

How it helps learning

it can help learning, but it aims:

• to qualify,

• to certify competence,

• to verify.

Advantages Disadvantages

• clarity • time to establish

• a working tool

6 SKILLS ASSESSMENT

• skills being assessed

• conditions specified

• standards of achievement identified

What is assessed - ability to carry out a task or perform a skill

How it helps learning

• sequences / prioritises the learning process

• identifies difficulties

Advantages Disadvantages

• Indicates skills • resource demands heavy

• "1 can do" / "feel good" factor • resource driven

Figure 26.1 Fitness for purpose of different types of assessment

It should be remembered, however, that the participants themselves represented learners
who had succeeded. Their views would therefore be expected to be considerably more
positive than an average group of learners, whose future careers may still depend upon
what happens next time they are assessed.

ERJC

171

QUALITY IN LEARNING AND ASSESSMENT

oo o>

fc oo

•55 po

8:2

0)
0)

i§.I

oo - £

Q) oo t

oo to oo

oo gj oo

CO O Q>
CO

o ^

0)
o 5

co o
cp

> CO CO

III

^ QQ O Q UJ

5
1

o
c

CO u.

"S o

° E
I J!

> ed

C O

I a,

Hffi

5 2

M 1

3 «U

a. 8

■8

3 S!

TO 3

■a 8

l a

3-S

lit

1 1

> _

o
Z-S

BS5750 FOR ASSESSMENT

s
I

q>
£

Q S

o
c

S
fi

«)

0) £

JJ GO

PS25

> C3 4J

.a 8 °
is

II §

e
8.

o >>
2 E

-a

2*53
2.S

ill

cq5

Iff s

fill

3 » o §

■gall

168

QUALITY IN LEARNING AND ASSESSMENT

CtJ

CCi

tfi
Q>
V)
V)
CCi

</>

<)
£

OUL

S
O

CJ>

V)
CD
V>
V)

o ^
o c

. . <-t-l

4=1 o .

gx:
o

w c

i o

o "o
«S £

ST"

Is £
§ « 2

«3 § H

"Bio
ex 0 «2

JC > 3
O O ~

■5

^ £ SS

on w C

| 3

~ 3

O o

f =
II

c
o

B.-8

C/} (/!

OS -5

< O

- -a

{'§>

w> S3
2

2^
-5 o

ex.

•8
"S

o .
i— > <u

5 *Sb
0£

•2 g 8

2 *o 8

3 32
o <5

c/J

JO 01

BS5750 FOR ASSESSMENT

169

5
S

a
3

5c
o
c

5
S

&
"5

CO
OQ U.

O 2

■5
o

£
•5
o

l™ 1

■fl

=a-s
3 "2

"S'S

■8

CX. '

5 3
as

q o

c c

c
a>

co
x:
o

3
CO

CO
CD

-J)
CO

cO
c
o

CD
JZ

cn
c

E
l_

C
3

X>
CO
X)
O

CD
i_

x:
"O

> CO
CD C

co —

3 C
CO CO

o _

CD CO

■? co

>s CD

CO ©
CD

.52 o

. CD
CD .-
>

O CP

■D 0)
CD Q_

CD
X:

CD O

cn*—

"O CD

s_ E "Z

iS c ^ to
« O w P

CD **
3 £

X5
CD
O
O

£ O Q-

CD
CO

CD
>
CD

CO
CD
CO
CO

O CD

tj to

o S>

x: g

to E

5 c

o ©

CO o
CD

S2f

to CO tO

E S

CO »_
tO m
CD =

o x:
to o

CD ^

a. E
*co

c >>
0) CO
J-
CD

CD
CD •

£ 8
E 8
a>.°
•|T5

ai'g §
g> go
a) c

" £ "co
« S x:

r E I

co £ •

a 2 to
I? 8

C c O
CD CD X

o-r 3

o i2

S. 5

c E^

> £ CD

> o c
£ to

III

CD o

e £

- CD

co -Q to

\_ Q) C

CD 2s to

5>5 S-

CD CD

CD ^2
CO 3

"co .E
c c

% CO

CD "
X)

8 S «

w .£ ® o

.2, E oj -

2, Q) CD

S?— >- T3

.£ oj « r

I* « S

a o •«

O « (0

« 8 «

§ ° 3
o to <5

sli

H to •

CO ^ £

C 5 -
0) CO -

2 tD

«fc x:
I c-

O tD

o ~
o 7*

O) tD
c >

.= CD

co £.

3 ^

3 CD

CO kl

CO CO

£ CD

^ c

to .

CD CO

x: x

to $ w

CT CD

£ H co O co

CD
O to

i to cS

- CD CO

o co

C CO CO

o c >

J- Q.£

(D O m
t: tO

P CL>

m «>

T3 CD
CD C

Q. CO

E ©

o .t;
E §

CD <P

■o o

CO Q.

Z q-

H °

• O)

CD •-
CO c

W CO

CO ^
CD *

CO _
CO T3

O) CO

c to

'tp p

170

QUALITY IN LEARNING AND ASSESSMENT

LAYING DOWN THE LAW

Participants were again divided into syndicates, and asked to produce and overhead
transparency giving their recommendations for "criteria for quality in assessment". Each
syndicate presented its acetate briefly to the whole group. The following are transcripts
of the acetates produced.

1 Recommendations fur Assessment

1 Of both process and product

2 Emphasis on verification/certification of product

It must:

• be standardised - as a process

• give practical feedback

• be standardised against established criteria/objectives

• be valid/content/predictive

• be reliable

• be appropriate to the task

• be easy and practical to use

2 Laying Down the Laws

a) Never use just one method to assess anything - use as many as possible.

b) Avoid numbers (or letters) as "marks" where possible. Use words.

c) Be diagnostic where possible, not normative. (Act on the diagnosis).

d) Be student-centred where possible (Profiles, not single shots).

e) Explain all criteria used beforehand to the students. If possible, negotiate them
with the students.

f) Spread the lead on both students and staff.

g) If judgemental, be gentle.

h) Failure -> diagnosis counselling -> opportunity -> success.

3 Criteria for High Quality Assessment

• Exams scrapped!

• Honest

• Fast feedback

• Concise, no jargon

• Fair

• Encouraging, confidence-building

• Achievable

• Continuous

• Not threatening

• Learning experience

• Flexible

4 Criteria for Quality Assessment

1 Link assessment to outcomes of teaching - ie. what did you plan for students to

ISO

BS5750 FOR ASSESSMENT

171

know, understand and become able to do?

2 Use assessment to:

• identify what students may need to learn

• celebrate what students have achieved

• diagnose students' problems

• suggest how students can improve (formative)

• summarise what students know at the end of a course

• give information to future employers.

3 Use different assessment techniques to match different course approaches. Don't
rely on one method of assessment.

valid

Aims of
rse

ermine

When?

During?
After?
Mix?

Why?
(see
Phil's
paper)

not necessarily
the easy option

knowledge

Itt/tSdes Product?
attituaes p rocesS es-a

student input
involvement

Contract

students
understand

<D Rachel, Julio, Andrew
and David

Figure 26.2 Rules for assessment

172

QUALITY IN LEARNING AND ASSESSMENT

5 Laying Down the Law: Asking the Right Questions about Assessment?

• Is it necessary?

• Is it relevant to the course objectives?

• Is it valid?

• Is it accurate and reliable?

• How does it build confidence?

• How does it lead to learning?

• How does it reinforce learning?

• Does the assessment drive the curriculum, or the curriculum drive the assessment?

• Is it fair to different students?

• Can the students practice?

• Can students know the criteria?

• Can students influence the criteria?

• Can quality feedback be given to students?

• Can the results of the assessment be used to redesign the assessment strategy and
the teaching/learning strategy?

6 A Visual Approach

Figure 26.2 shows one syndicate's visual approach to "laying down the laws".

CONCLUSIONS

I had fun running this workshop, and I think participants rather enjoyed it as well! It was
a "busy" workshop; participants produced a great deal of material in the 90 minutes
involved. Although delegates to an international conference on education and training
are somewhat self-selecting in that they are people whose education and training
experiences lend to have been highly successful, I hope that this collection of their
wisdom and experience may be useful in helping to make assessment processes in general
better-suited to their purposes. I hope that in particular, we may move further towards
using assessment as a valuable part of the learning process, rather than a way of
measuring how far the horse may have bolted after leaving the stable door unlocked!

Acknowledgements

I am grateful to all the participants who took part in this workshop:

J Arter, A F Bickerton, Brian Canniford, Chu-Poon Yap Ching, Julie Clerk, Paul
Ellis, Chahid Fourali, J Greenacre, Rachel Hudson, Steve Leary, Peter Leckstein, Jill
Lloyd, Patricia McCarron, Chris O'Hagan, Jack Oakley, Michael Owen, Larry
Roberts, Bob Sayer, Michael Shoolbred, Prof A R Sykes, Christine Tan.

1 fie

Section 5: Quality in Teaching and Training

The papers in this section fall into two groups: three address the quality of teaching in
higher education, and two of these consider the role of teacher appraisal in the process
of developing quality. The clear message conveyed is that quality will be enhanced only
if those seeking to bring about change understand and acknowledge the meanings teachers
give to their professional activities and experiences. The remaining three papers describe
applications of technology to training and total quality approaches to managing the design
and development of CBT.

In "Eating frogs and bridging gaps - post-Warnock conditions for teaching quality", Clive
Colling explores the implications of the five conditions for teaching quality in higher
education identified in the report of the Committee of Inquiry set up by the Polytechnics
and Colleges Funding Council. His observations are important and timely, given the
context of recent increases in participation rates in this sector.

David Jones and Janet Hanson take up the challenge of the Warnock Report and
demonstrate in their paper, "Teaching quality - 'I know it when I see it!'", how one
higher education institution has set about identifying, supporting and developing good
teaching practice. They outline a scheme for the appraisal of teaching practice, report the
opinions of participants, identify proposed changes to the scheme in the light of the pilot
and address wider concerns expressed about teaching appraisal in the education literature.
In "Upward appraisal and its implications for higher education", Gerard McElwee and
colleagues report the use of a questionnaire based on a ten-dimensional model of quality,
developed for the service sector, in appraising lecturer performance. Although they focus
mainly on the reactions of a sample of lecturers in a UK Business School to the use of
students as appraisers, they also address the consequences of upward appraisal from the
student point of view and indicate how upward appraisal data can be used to enhance the
quality of teaching and learning.

In "A total quality approach to managing CBT development", Steven Shaw and David
Shaw describe techniques and approaches for reducing cost, improving quality and
assessing quality in CBT. They provide a useful and detailed guide to quality
management methodology that could be implemented at the level of a training division
or department, even if a commitment to TQM has not been embraced at a higher,
strategic level by the organisation as a whole. Meurig Williams and Graham Carr
show in their paper, "Towards quality management in training design", how an integrated
quality management policy is applied throughout the training development cycle, and
across all training media, by the Training Development Group at Lloyds Bank. They also
present a case study which shows how quality management tools are used in CBT
courseware production.

Feter Willis and John Eary, in "Achieving quality in networked interactive video",
describe an award-winning multimedia system for training police supervisory officers in
crowd control procedures at major spectator events such as football matches. They
outline the quality approach in the design of the system, describe the equipment,
functional specifications and the system -in -use. and identify its potential for management
training.

1*3

27 Eating frogs and bridging gaps - post-
Warnock conditions for teaching
quality

Clive Colling, University of Northumbria, Newcastle-upon-Tyne, NE1 8ST, UK
THE QUALITY DEBATE

People have very different views about what constitutes "quality" in higher education.
Some believe the provision of material resources/equipment, buildings, etc to be a major
criterion; others highlight the qualifications of staff and the examination results of
students' Evidence of innovation, staff development, employer liaison and the pastoral
care of students are also considered to be important indicators of quality.

But at the point of delivery of any higher education system must be high quality
curriculum content, effective and meaningful assessment procedures, and high quality
teaching. Good teaching promotes good student learning, and exceptionally good
teaching encourages students to be independent learners who know how to apply their
knowledge and skills in practice, and consequently how to leam for themselves.
Promoting learning by enabling students to "learn how to learn" will be the major
criterion for successful professional teaching activity in higher education in the 90's. But
achieving efficiency of teaching with effectiveness - for example, coping with high
numbers of students for less time in restricted space - will be a major dilemma. Rarely
has a decade started with so many challenging prospects for managers and teachers m
higher education in the UK.

Eighty-seven thousand more students, 41.560 of them full-time, are projected over the
next three years: a remarkable scenario, exceeding even the Government's expectations
on greater access. Annual rates of growth of up to 33% are predicted by 24 former
Polytechnic and Colleges Funding Council sector institutions, including five former
polytechnics. Against this backdrop, doing nothing about teaching quality is not an
option. There is an imperative for higher education institutions to accommodate students
with a wider range of academic and practical experience than before, many of whom will
not have the traditional qualifications for entry. Not only will entry requirements and
procedures have to change, but teaching methods and design of courses will have to meet
the needs of the new tvpes of student. The Government believes that increased
participation in higher education need not be at the expense of academic excellence; but
it will be for the HEI's themselves to find ways of meeting the difficult challenge of
maintaining (and enhancing?) quality standards in the 1990s.

Universities and Colleges therefore face a fundamental change, in the next decade, if
the Government's aim of increasing access to higher education is to be achieved. All
institutions need as a matter of urgency to find ways by which they can ensure that an
influx of more students will not adversely affect the quality of the education that is
offered. "Teaching quality" thus takes on a new significance.

FOCUS ON TEACHING

In 1988, the then Secretary of State, in a letter to the Chairman of PCFC, confirmed the
Government's concern that teaching quality should be taken into consideration by tin*
Council. He wrote "I look to the Council to develop further indicators of both the quality
and quantity of institutions' teaching and should be grateful if it would ... consider how

18*

176

QUALITY IN TEACHING AND TRAINING

these might be used as an input to its funding policies and decisions". While research
has for generations been evaluated in higher education funding decisions, and its quality
and efficiency both subject to review, teaching had not been subject to this kind of
scrutiny. Now it was quite clear that the quality of teaching was to be one of the criteria
employed in the allocation of funds.

The Committee of Enquiry into Teaching Quality, chaired by Baroness Warnock, was
set up by the Polytechnic and Colleges' Funding Council in January 1989. The terms of
reference were:

a) to identify characteristics of effective and efficient teaching;

b) to identify which of these characteristics can be developed as indicators of
teaching quality;

c) to suggest the means by which institutions could demonstrate the effectiveness and
efficiency of their teaching and the promotion of students' learning;

d) to advise on possible strategies for the Council which would serve to raise the
quality of teaching. These might include procedures for monitoring and evaluating
outcomes, and funding mechanisms (PCFC 1990).

The Warnock Committee met in April 1989, and held nine further meetings. In May
1989, 84 PCFC institutions were invited to respond to three questions:

a) how does your institution identify efficient and effective teaching?

b) what is your institution doing to preserve high quality work once it has been
identified?

c) what is it doing to improve quality in those areas which have not been so

Responses were received from 57 institutions. The Committee analysed the responses,
and followed them up with a programme of visits, covering modular course arrangements,
mature students, the teaching of engineering and modem languages. Enterprise Initiative
developments, initial teacher training and the teaching of humanities. Institutions visited
included a specialist arts institution, an institution that works with more than one
validating body, and several which had educational methods or development units.

A review of relevant teaching and learning literature was commissioned. A series of
consultations and discussions took place with representatives from industry, commerce
and the professional bodies. The key points in the Committee's report (PCFC 1990) are:

a) that teaching must be interpreted broadly, as the initiation and management of
student learning by a teacher; that it must be responsive to student needs; and that
the conditions necessary for good teaching must be a priority at every level of the
institution;

b) teaching must be judged "good" by whether it contributes to the purposes of
higher education - the life-chances of the students; excellence in teaching could
be traced to the ethos of the whole institution;

c) five necessary but inter-related conditions must be fulfilled before teaching can
be judged good:

• clarity of aims and objectives related specifically to teaching, and confidence
that they are worthwhile and appropriate to students' needs;

• a policy regarding curriculum organisation and delivery, including a readiness
to consider different methods of promoting learning;

• a policy for professional development of teaching staff, including appointment.

identified?

185

CONDITIONS FOR TEACHING QUALITY

177

induction, appraisal and development;

♦ means by which the views of students and employers can be used in
judgement of the curriculum, its delivery and outcomes;

♦ an identifiable framework within which an institution can evaluate its own
success in meeting its objectives and adjust its practice accordingly.

John F Kennedy said, "Change is a way of life. Those who look to the past or present
will miss the future". Translating the recommendations of the W a mock Committee into
practical action in classrooms, labs and studios across the former PCFC higher education
sector will require change. Change is constant in many enterprises but, as somebody
once said, in education, interpreting change is rather like trying to grasp fog. A head
teacher friend once likened managing educational change to trying to build a ship - at sea,
from the keel up!

It is important that any change to professional teaching practices in higher education is
seen to be constructive and positive; supporting and encouraging existing good teaching
practice, rather than trying to revolutionise processes that have been going on for a long
time. Certainly, some old assumptions may need to be tossed out - does one hour's
lecturing automatically produce one hour's learning? - but there should be no pig-headed
insistence on change for "flavour of the month" purposes. A lot of excellent educational
practice exists across student learning outcomes. It is a matter of:

♦ examining and knowing what we actually do, as teachers and managers of learning
resources;

♦ learning from the past - both the good and the bad;

♦ working together to promote growth (do we really collaborate effectively, as
professionals?);

♦ knowing and using our limited resources effectively;

♦ assessing our own performance;

♦ using the corporate aim to put students first, and keep our thinking focused on the
institutions' performance.

A first scan of Warnock's five conditions may produce the response "We've been doing
this for years!" from some teaching colleagues. But "doing this" sometimes equates to
symbolic action, undertaken and written about in order to satisfy a plethora of
committees. Literally interpreting the tasks encompassed in the five conditions requires
a professional engagement with difficult tasks, and the raising of hard questions. A brief
"Cooks Tour" may illustrate my point.

Teaching aims and objectives should be clear, worthwhile and appropriate to
students* needs.

What do "clear", "worthwhile" and "appropriate to students' needs" mea.i for individual
teachers, their courses, their students and their particular employers? One key
recognisable characteristic of quality must be relevance. A key quality process must be
to do with listening. Listening is strategic! The student perception is a reality. We need
deliberate self-critical analysis of teaching practices and procedures in order to continually

RHETORIC INTO REALITY

CONDITION 1

178

QUALITY IN TEACHING AND TRAINING

seek to improve what we provide for students. Does our teaching promote learning as
part of a high quality student experience? Do we listen to what students say? Do we hear
what they say. when they say it? Do we do anything as a result? There will need to be
a partnership in deciding what is "clear", "worthwhile" and "appropriate".

Curriculum policy should encourage new methods of promoting learning.

There are extraordinarily innovative educational practices going on across the former
PCFC sector. The problem in the present competitive climate is finding ways for
professional teachers to share good practical ideas. The point is, many institutions are
hotbeds of experimental activity in teaching and learning, and the Wamock
recommendations are quite clearly supporting and encouraging constructive change and
investigative educational development. To stand any chance of claiming extra cash for
teaching quality, institutions will need to be able to demonstrate that the education they
offer will enhance students' understanding and imagination, and "excite their intellectual
curiosity and expand their knowledge and skills". Simply floating the usual papers at
review and validation time won't be enough. It's not just a matter of what course teams
feel, think and know about the curriculum, it's what they do. Are they promoting active
learning? Live projects? Independent, autonomous learning? Peer assessment?
Collaborative group work? etc, etc.

Professional development policy should include appointment, induction, appraisal
and development.

For many of us, tiiis is the really good stuff. The Wamock Report recommends that
induction courses for teaching staff should be compulsory for all teachers in their first
year and that induction should continue beyond an initial course. Teaching methods,
strategies and tactics should be all built in here. The Report is giving a hefty shove to
the whole idea that teaching should be way up on institutional agendas. It is saying that
there should be visible reward systems for good teaching and that senior management
should support strategies aimed at raising the general level of teaching. The message is
that full-time and part-time teaching staff should have access to support for their own
professional development. And we are not talking about some weekly "Mary Poppins
good time hour" here, but properly structured, organised, funded and delivered staff
development programmes. (For the checklist of important criteria, for interpreting
condition 3, take a look at page 49 of the Wamock Report.)

Students and employers should contribute to the judgement of curriculum design,
delivery and outcomes.

Many courses in many institutions have for years actively involved students and potential
employers in educational development decision-making. Perhaps others have simply
talked or written about involving them. The message from Wamock is quite clear:
institutions should commit resources to actively seeking the views of employers,

CONDITION 2

CONDITION 3

CONDITION 4

CONDITIONS FOR TEACHING QUALITY

179

professional bodies, staff and students about the needs, requirements "and imperatives"
for each group. (Again a checklist of possible criteria can be found on page 50 of the
Report )

No one would claim that there is a "handy dandy" quick-fit way of involving students
and employers in judgements about curricula, delivery and outcomes. We really do need
innovations and intelligent thinking in this area. It just isn't feasible to produce
thousands of feedback questionnaires for example, which nobody ever analyses or
interprets, and which have no status beyond being "draft" excluders in small overcrowded
staff rooms. The views of students and employers need to be used to "actively improve
the planning and delivery of academic programmes.

CONDITION 5

Institutions should evaluate their attainment of their objectives, and adjust their
practice accordingly.

Throughout the years of CNAA rule, courses, departments and institutions have stated
objectives. The "objectives" business has become a fine art. Those colleagues who can
string the right fine words together for course documents are in much demand at review
and validation times. "Look at this Linda! 'Collaborative, activity-based, student-centred,
formative assessment procedures will underpin course^ principles for high quality
pedagogic action'. It's a wrap, they've got the lot there!"

But promoting and supporting institutional self -evaluation is about making objectives
operational, and negotiating what they mean in action. What are the unintended outcomes
of stated objectives? How should they be reviewed and redefined? What are the key
criteria and indicators of true progress and improvement? What sort of evaluation
evidence will serve as a basis for claims of "outstanding" quality m the next funding

round? . ~ f

HEIs have fulfilled their obligations for stringent quality assurance to the Council tor
National Academic Awards for years. Peer validation review procedures have done an
excellent job in sustaining and monitoring quality for students across the former PCFC
sector Wamock's fifth condition is, however, seeking to build on existing review
practice, by recommending to the Funding Council that it looks for specific evidence of
commitment to teaching quality enhancement not just at course and departmental level,
and not just on paper, but throughout the management and practice of the institution.
This is a harder edged perspective, and is underpinned by the principle stated in the
Report that the conditions necessary for good teaching must be given priority at every
level of the institution.

References

Polytechnic and Colleges Funding Council
committee of enquiry PCFC Bristol

(1990) Teaching Quality: report of the

EJJC

183

28 Teaching quality - "I know it when I see
it!"

David Jones and Janet Hanson, Bournemouth University, Bournemouth, BH125BB,
UK

INTRODUCTION

In October 1990 the Wamock Report on teaching quality threw down the gauntlet,
sheathed in a velvet glove, challenging higher education institutions to demonstrate how
they could identify, support and develop good teaching practice. This contribution
demonstrates how one institution, Bournemouth Polytechnic (now Bournemouth
University), took up this challenge. An Appraisal of Teaching Practice scheme was
developed and piloted in two departments in Autumn 1991.

The scheme devised is both developmental and evaluative. It aims both to enhance
teaching quality by identifying good practice and facilitating the development of
appropriate staff development, and to allow the institution's Academic Board to obtain
an aggregate view of the teaching quality of each department. It involves the judgement
of a peer and the individual lecturer's head of department. Both observe two teaching
sessions and give verbal and written feedback to the lecturer concerned.

This case study provides an insight into the main issues involved in the observation of
classroom practice in a higher education environment and discusses the practicalities
surrounding its implementation.

DESIGN OF THE PILOT

At the start of the pilot we suggested three aims for the exercise:

♦ all panics to emerge from the process with their dignity and integrity intact;

♦ that the experience should lead to a continuing improvement in teaching;

♦ that we should be in a better position to advise on the design and implementation
of a future system which would achieve the first two aims.

The overall response emerging from the review of the pilot suggested that we were not
far away from achieving the first two goals and we also think that we have found ways
to improve on the initial appraisal of teaching quality pilot.

The pilot involved staff from the departments of Business Information Systems and
Product Design and Manufacture and took place over a period of three weeks between
7 and 25 October 1991. Involvement was voluntary and 27 out of 29 staff from the two
departments took part as appraisers.

The appraisers comprised the five heads/associate heads of the two departments
(A/HODs) and 13 experienced members of the Course Evaluation Group (CEG). This
group acts as a peer review body for course approval and monitoring and was therefore
the natural place to look for peer appraisers of teaching quality. Indeed, members are
frequently appointed to this group on the basis of recognition of their teaching expertise.
The head of department also seemed a natural choice for appraiser. As the lecturer's line
manager, he has responsibility for assisting the individual to make use of staff
development opportunities.

Two observation sessions per member of staff were planned, both to be observed by the
appraisce's A/HOD and a member of the CEG. One session was selected by the

TEACHING QUALITY

181

appraisee and the other by the A/HOD. We decided that, if possible, each appraisee
should have the same CEG appraiser observe both sessions.

Following early discussion with the departments involved, we varied the pilot to include
a group who would be appraised for one of the sessions by a subject specialist instead
of the A/HOD This group was selected on a random basis and those selected were asked
to nominate a subject specialist appraiser. Four of these appraisers were eventually
nominated. In total, 54 sessions were arranged and the responses from the evaluation
questionnaire covered 35 sessions.

The observation criteria agreed by Academic Board were:

1 Organisational and structure of session

2 Clarity of presentation and use of resources

3 Presentational style and classroom management

4 Monitoring of effectiveness

5 Whether the aims of the session have been met

Three staff development workshops were held for the appraisces and two for the
appraisers. These were designed and managed by the authors.

The appraisal of teaching process involved four key stages for each session appraised:

1 Briefing meeting

2 Observed session

3 Verbal feedback

4 Written feedback

1 The briefing meeting between appraisers and appraisee which took place before the
observed session was regarded as an integral part of the appraisal process. The
appraisee was able to set the scene in terms of such matters as the stage of
development of the course, the objectives of the session, and the student group. The
appraisee was also able to suggest particular features which he/she would like the
appraisers to focus on during the observation. Guidelines on observing teaching
behaviour and on giving and receiving feedback were provided to all taking part.

2 The observations then took place and the appraisers' comments were recorded on the
observation record designed for the pilot.

3 Verbal feedback and discussion of the session with the appraisee took place as soon
after the observation as possible.

4 The written feedback record provided the formal record of the process of
observation and feedback and included suggestions for future action. It was
designed lo be completed jointly by the appraisers and sometime after the verbal
feedback sessions. A copy of the record was then sent to the appraisee who signed
it and if appropriate added further comments before sending it to their head o f
department.

At the end of the three weeks of observation activity a review of the exercise was carried
out. The review was based on a questionnaire citculated to all participants who provided

APPRAISAL PROCESS AND DOCUMENTATION

REVIEW OF THE PILOT

182

QUALITY IN TEACHING AND TRAINING

individual and anonymous comment on a range of matters. When the results of the
questionnaires had been collated, meetings were held with groups of appraisers and
appraisees to check the findings.

The questionnaire asked for specific information about the details of how many sessions
were observed, the type of sessions and how long the process took. It also sought the
opinions of appraisers and appraisees on several aspects of the process, including their
experience of observing and being observed, the adequacy of the preparation for the
appraisal scheme and any suggestions for changes.

The response rate was high. 21 replies were received from 27 appraisees (78%) and
18 replies were received from 18 appraisers (100%).

Comments About the Briefing and Feedback Sessions

We had stressed that these were an integral part of the appraisal process so we wanted
to find out how well they had been carried out and to what extent the appraisee had been
encouraged to take the initiative in asking the appraisers to focus on specific features of
the session. In the event, the problems of matching the timetable of the appraisees and
the appraisers meant that only 20% (4) of the appraisees had the opportunity of holding
a briefing session prior to both observations though 70% (14) had one briefing session.
66% of the appraisees felt they had been given an opportunity to ask the appraiser to
focus on a specific feature of the teaching session.

Comments About the Immediacy of Feedback and Follow-up
Action

The literature on feedback underlines the value of a quick response and over 95% of
those appraisees who responded said they received verbal feedback quickly after each
session. The written feedback was not so immediate and frequently not provided.

Giving Feedback

Most appraisers appear to have followed the guidelines on giving feedback in that 75%
(15) of the appraisees who were observed reported that they had an opportunity to put
their ow>? views first; 80% (16) said they were given an opportunity to seek clarification
and all those observed felt that their views were valued.

Comments on the Observation Criteria

The observation criteria were found to be helpful by 68% of 1 13/19] appraisees and 71%
[12/17] appraisers in preparing for the session.

Comments on the Validity and Utility of the Appraisers'
Observations and Feedback

During the preparation sessions some concerns were expressed about the relative value
of specialist as opposed to non-specialist feedback. Indeed the pilot design was varied
to allow a small group of observations to be undertaken by subject specialists nominated
by the appraisee.

TEACHING QUALITY

183

The results of the questionnaire showed little variation between the rating of the Course
Evaluation Group members, the HOD/AHOD and the subject specialists. This perception
is confirmed by the appraisers, in 66% (12/18) of the cases appraisers reported that their
comments always matched those of the second appraiser.

Comments on the Overall Quality of the Appraisers

Despite the overall endorsement of the validity and the utility of the appraisers*
observations, some concern was expressed by the appraisees about the quality of the
appraisers. The concern centred on the need for objectivity. Many other characteristics
and qualities emerged in response to the question which was asked of appraisees and
appraisers: "What are the qualities of a good appraiser of teaching practice?"

There was encouragingly a good deal of overlap between the set of responses.

Sample of appraisees* comments: "Willingness to discuss with humility not desire to
impose own style preferences" and "Someone who doesn't try to impose own teaching
methods on a colleague".

Sample of appraisers* comments: "Promote sessions as exchange of methods" and "To
give constructive feedback".

Comments on the Preparation Sessions

The questionnaire data indicated that the training was successful in explaining the concept
of teaching appraisal; 65% (13/20) of appraisees and 83% (15/18) appraisers agreed this
to be the case.

Personal Concerns

The pilot scheme inevitably aroused much uncertainty among lecturers, so both sets of
respondents were asked to say what concerned them most about the process and also to
identify the principal benefits to counteract the feelings of uncertainty.

Some clustering of views emerged from the individual comment of the appraisees (21
respondents). These covered concerns about the increased administrative load connected
with the scheme; the quality of the appraisers; the need for staff development support to
underpin the development of teaching competence; the representativeness of the sample
of observations; and the use to be made of the information collected about an individual's
performance.

Overall the comments of the appraisers (18 respondents) did not group as well as those
of the appraisees, but concern was also expressed about the eventual use of the
information and the additional administrative load. We return to these concerns when we
consider the modifications we propose to make to the pilot scheme.

It was clear that both appraisees and appraisers placed a high value on the outcomes of
the appraisal process and saw its contribution in terms of improving practice, personal
benefits and reflection. In answer to the question: "What do you think might be the
benefit for your personally of the appraisal of teaching practice?" Appraisees* replies
included: "Thinking and talking about my teaching with colleagues" and "Nothing but
improvement in my teaching skills".

It was also clear from individual comments that appraisers felt that the process was
valuable in terms of their own learning: "Exchange of ideas and different perspectives"
and "I leam for my own teaching".

184

QUALITY IN TEACHING AND TRAINING

PROPOSED CHANGES TO THE APPRAISAL OF TEACHING
PRACTICE SCHEME

As a result of the pilot experience and the feedback provided by the review, we decided
to make several changes to the appraisal process and the documentation should we run
it again.

In view of the relative agreement of views between the CEG appraisers and the
A/HOD' s we decided that we could relax the requirement for both sessions to be
observed by the A/HOD. A/HOD will only observe one session.

We also felt that our choice of the CEG members as peer appraisers was justified since
there was little difference in the validity and usefulness of their observations compared
with those of the subject specialist observers.

We felt that the observation criteria had not been sufficiently detailed to provide specific
enough feedback to the appraisees or to be very helpful to the appraisers when planning
the observation. A tool which some had found very useful was the list of "effective
teacher behaviours" developed at Newcastle Polytechnic (Colling 1990). We felt it would
be useful to extend the use of this tool but we did not wish it to be regarded as a detailed
checklist. We decided to remove our existing criteria and invite appraisees and appraisers
to negotiate criteria for each session. We suggest that the list of effective teaching
behaviours can be used as a way of identifying specific teacher activity to be observed,
which may differ in each session and that the list of effective teacher behaviours provides
a common vocabulary during the verbal feedback session.

We also felt that the list as published covers teaching behaviour which can be observed
mainly at lectures and seminars but over a period of time we expect to build up a bank
of additional activities and effective behaviours which apply to the teaching specific to
this university.

Very few appraisees received any written feedback and we realised that our method
was not designed to encourage this, so in future we will advise appraisers to give verbal
and written feedback during the feedback session. We have revised the written feedback
form to encourage appraisees to use it first to reflect on their performance, and their
comments are then followed by appraisers' comments. This means that only one piece
of paper is in existence and it remains with the appraisee to pass it on to their HOD. The
appraisers will also be urged to leave their observations records with the appraisee. We
feel that these procedures should help to give the feeling to the appraisees that they are
responsible for the flow *> c information, and to encourage ownership of the system. It
should also help to prevent breach of confidentiality. Advice on these procedures will
be gathered together to form a code of practice governing appropriate conduct for the
appraisal of teaching practice.

It is useful to draw some conclusions about this pilot in relation to the wider concerns
expressed about teaching appraisal in the education literature.

It is becoming recognised that general criteria for effective teaching can be identified,
(Ramsden 1991) yet it is important to realise that it is not wise to import wholesale
criteria from one institution to another. Each institution has to develop its own specific
criteria and on this matter we agreed wholeheartedly with Seldin's comment that "the
success or failure of the colleague observation programme is contingent on the common
acceptance by the observer and teacher of the appropriateness, reasonableness and fairness
of the rating instrument and its implementation" (Seldin 1980, quoted in Weimeret al 1988).

CONCLUSIONS

ERLC

TEACHING QUALITY

185

We feel that observation can be used as a developmental process in encouraging good
teaching practice but that the observation records should not be the only source of
information about teaching practice considered in the appraisal interview. We urged
appraisees to collect information themselves about their teaching from their students and
colleagues. As with any evaluation system, there should be multiple sources of input.

We gained much good advice and examples of good practice from the literature on
teaching appraisal in schools (Bollington and Bradley 1990), from management literature
on the ways to give and receive feedback (Wood and Scott 1989) and from existing
sources of advice on appraisal in higher education (Gibbs et al 1989). We recognised
that in order to be' valid, comment must be based on observation not inference and
furthermore, based on what was specifically observed during the two sessions. It was
important not to make generalisations about the quality of teaching from these two
sessions.

We also felt that we proved that it is possible and useful to use appraisers outside the
discipline areas of the appraisees and that their observations, although maybe different
to a subject specialist's, were as valid. We felt that our system of monitoring and
evaluation of courses provides us with other ways of checking the subject content of
teaching.

There are also features of the quality of the feedback which we found to be important.
Feedback must be based on behaviour rather than personality, or what the teacher did
rather than what the teacher is. Feedback should also be specific enough to allow the
appraisee to formulate an action plan for self development.

It is suggested that it is difficult to achieve inter-reliability between observers when
many are used but we feel that it is preferable to disseminate good practice by using large
numbers of appraisers rather than restricting the observations to a small group of
appraisers, who may achieve high inter-reliability but at the expense of the feeling of
ownership of the system which we want to encourage.

References

Bollington R and Bradley H (1990) Training for Appraisal: a set of distance learning
materials Cambridge Institute of Education Cambridge

Colling C (1990) Teaching Quality Matters Bulletin of Teaching and Learning Issue
4 July pp 10-17

Gibbs G, Habeshaw S and Habeshaw T (1989) 53 Interesting Ways to Appraise Your
Teaching TES Bristol

Polytechnic and Colleges Funding Council (1990) Teaching Quality: report of the
committee of enquiry PCFC Bristol

Ramsden P (1991) A Performance Indicator of Teaching Quality in Higher Education:
the course experience questionnaire Studies in Higher Education 16(2) pp 129-150

Seldin P (1980) Successful Faculty Evaluation Programs Coventry Press Crugers

Weimer M G. Kems M M and Parreit J (1988) Instructional Observation: caveats,
concerns, and ways to compensate Studies in Higher Education 13(3) pp 285-293

Wood R and Scott A (1989) The Gentle Art of Feedback Personnel Management April

194

29 Upward appraisal and its implications
for higher education

Gerard McElwee and Tom Redman, David Eadson, Adrian Evans, Leeds
Metropolitan University, Leeds, LSI 3HE and Teesside University, Middlesbrough,
TS1 3BN, UK

INTRODUCTION

A number of potential appraisers are available for an organisation to appraise its
professional staff. These include self, peers, managers, internal and external customers,
and subordinates. In this paper we examine the potential for the use of upward appraisal,
that is the use of data generated by and from students, in the assessment of lecturer
performance. It remains an issue for others to decide, posing some interesting questions
for educational philosophy along the way, whether upward appraisal of lecturers should
be classified as appraisal by customers or subordinates!

Upward appraisal is claimed to have many benefits for organisations. These include
improving management style and people management, facilitating personal development,
increasing productivity (Bernardin and Beatty 1987), enhancing the recruitment
attractiveness of the organisation, and facilitating "voice" (Bernardin 1986) within the
organisation. It is also claimed to have many advantages over traditional appraisal
schemes. Firstly, by the very nature of their relationship students are usually in much
greater contact with the lecturer than the line manager. They are thus in a good position
to observe directly a large volume of lecturer behaviours. Secondly, such observations
are from a unique vantage point - "the receiving end". Thirdly, it offers a multiple source
of appraisal data. Lecturers have many students but usually only one direct line manager.

Performance appraisal generally in the education sector has been particularly
underdeveloped compared to industry. However, it seems that upward appraisal is one
area where education has clearly led the field. For example, one study of the appraisal
of lecturers in a UK university found some 29% of lecturers had experienced the use of
information from students as part of an appraisal of their course/teaching (Rutherford
1988). The use of student evaluations of lecturer performance has a long history of
attracting the researcher's attention (eg Breed 1927).

Although many aspects of student appraisals have been researched, there appears to be
a number of "blind spots". Firstly, the studies have neglected the effects of upward
appraisal of lecturing performance on the commitment of the students to their studies,
course, institution etc. Secondly, the effects of upward appraisal on the lecturer's
commitment and morale remain unexamined. Finally, most of the research that has been
conducted is for research purposes only, yet it seems that appraisals differ markedly
between the laboratory and real life (Banks and Murphy 1985).

Our study of student appraisal of lecturer performance within a Business School seeks
iO address some of these issues. We have examined, via the use of semi -structured
interviews with lecturing staff, their experiences and perceptions of the use of appraisal
data generated by students. We seek here to answer a number of questions about the use
of upward appraisal for lecturers. How "acceptable" to lecturers is the use of student
generated data in the appraisal process? For what purposes can student generated
appraisal be used? Who should have access to the data? What are lecturers perceptions
of students as fair and accurate raters of their performance? What are effects of upward
appraisal on lecturer and student morale and commitment?

UPWARD APPRAISAL

METHODOLOGY

187

The QUALED pilot study, described below, identified the existence of gaps between
students' and staff perceptions of attributes of Teesside Business School (TBS), and
between staff and student expectation and perception of the activities of the School. It
is argued that the existence of these gaps is of cause for concern as it is a source of
dissatisfaction with the services provided.

The objectives of the current study were to use the following criteria to measure the
quality of educational provision: the concept of a gap analysis between perception and
expectation of higher educational service; the ten dimensions of reliability,
responsiveness, competence, access, courtesy, communication, credibility, security,
understanding, and tangibles: the concepts of the SERVQUAL model as developed by
Parasuraman and Zeithmal (1985).

SERVQUAL was originally designed to assess the provision of services within the
service sector. In effect it consisted of a 22-item, seven-point Likert scale. It was felt
however that not all of the questions were directly applicable to higher education. A 35-
item, seven-point scale was chosen.

The questionnaire had three sections. The first section contains a series of questions
relating to the normative expectations of quality provision in the TBS. The second
section contains a series of questions relating to the perception of quality of services
provided by TBS. The third section was designed to provide demographic and other data
from the respondents.

Although the items of the first and second sections were similar, as indicated above, the
directions in each of the sections wert slightly different, in that in section A respondents
were asked what they expected in regard to each item, whereas in section B respondents
were asked to report their perception of the quality of service received on each item. To
all intents and purposes this framework is not dissimilar to that chosen by Parasuraman,
or that chosen by Saleh and Ryan (1991) in a subsequent adaptation of the SERVQUAL
model to the hospitality industry.

The sample consisted of 17 TBS staff and a cohort of 120 students on a BA (Hons)
Business Studies degree. The questionnaires were completed by the respondents in the
presence of, in the case of staff, student researchers, and in the case of students, the first
author in a time allocated as a teaching period. The sample of 137 was comparable in
size with other similar studies, Parasuraman and Zeithmal (1985) and Saleh and Ryan
(1991). The survey was carried out over a two week period in February 1992.

For this pilot study, our main concern was to provide a preliminary analysis that would
test the hypothesised ten dimensions of quality derived from the SERVQUAL model.
Factor analysis of the 35 items in the questionnaire (a principal factor analysis using an
oblique (oblimin) rotation) revealed, however, only eight applicable factors for both
expectation and perception of provision. It also indicated that these factors, in both cases,
accounted for only 50% of the variance, and that the items relating to each of the
SERVQUAL dimensions were spread across the factors.

We need to investigate in more detail the measuring instrument. Considering that the
questionnaire items were derived directly from the work of Parasuraman, it may be that
they are not directly relevant to the higher education sector and will need further revision.
Indeed, Parasuraman \s ten dimensional hypothesis of service quality may not be directly
applicable to issues of quality in HE. As regards methodology, when applying factor
analysis, and considering the number of items used, a much larger sample may be
required and a "test/re- test" design may be more effective in distinguishing between
expectation and experience.

188

QUALITY IN TEACHING AND TRAINING

FINDINGS

Firstly, it seems the majority of lecturers are in favour of using student generated data in
appraising their performance. A number of these, perhaps to be expected in a Business
School, expressed their approval couched in a language of customer care. For example,
lecturers felt upward appraisal could measure "customer satisfaction", "client approval of
the service provided", and provided a "key to delivering a quality product". One lecturer
suggested that such a source of aopraisal data was more valuable than that provided by
his line manager because "the customer is always right whereas the management here
very rarely ever are".

Particularly emphasised by those lecturers in favour of upward appraisal was the value
to themselves of student feedback for personal development. A number of those
interviewed stressed the value of student appraisals in providing a source of feedback on
teaching behaviours they were completely unaware of. For example, one lecturer
described how a student evaluation questionnaire had brought to his attention that he
spent much of the time in lectures looking out of the window rather than at the class and
that this was interpreted as a sign of his disinterest by the students. Thus upward
appraisals possess the valuable potential to allow us a view of ourselves as lecturers as
others see us.

A small number of lecturers were on the other hand opposed to any use of such data
in their performance appraisal and indeed to upward appraisals being conducted at all.
A range of reasons were offered for this. Prominent here was a fear of what was
described as the "get even factor". Lecturers were concerned with disaffected students,
for example those who had scored badly in assessments, those who had personal grudges
etc, using the vehicle of upward appraisal to give damaging and unfair ratings of their
performance. One lecturer described how she felt upward appraisal would encourage
students to become "over-litigious against some members of staff" and would thus "spend
their whole time complaining rather than biting the bullet and getting on with it".

A number of lecturers appeared to be "casualties " of the upward appraisal process.
One lecturer described the damage to his morale and commitment to lecturing as a career
that had been caused by previous "maulings" in student evaluations, so much so that a
change in occupation had been seriously considered. Another lecturer described how he
felt student evaluations he had personally experienced reduced to little more than
"popularity contests 11 . They were considered as "providing an ego boost for some but the
death knell to others".

Another concern with using students as appraisers was that students have only a partial
viewpoint and knowledge of the role of the lecturer and that upward appraisals should be
carefully restricted to areas such as delivery, assessment, lecturer availability,
approachability etc. A common view was the. the criteria used in student evaluations
needs careful design, and it rarely receives this. Course tutors, those usually responsible
for the design process, often used questionable evaluation instruments, usually obtained
from prescriptive publications with very little consideration of customisation for the
particular group/course to be surveyed A further worry was that many instruments
measured little more than lecturer "entertainment value".

Lecturer acceptance of the use of upward appraisal declined dramatically for purposes
other than developmental activities, for example reward systems or promotion. When
questioned on the acceptability of its use for forming a data source in formal "evaluation"
type appraisal interviews support was much more limited. However, given such a
position lecturers were generally fairly open about who should have access to upward
appraisal data. Only a minority of lecturers thought it should by restricted to themselves

JL K~' i

UPWARD APPRAISAL

189

and their immediate line manager. Over half of our sample felt such information should
be available to course boards and higher levels of management.

Some concern emerged with how this information should be presented to course boards.
One lecturer described what he felt was a rather insensitive handling of student
evaluations where the results were aggregated and an overall rating was tabled as a
standing item on the course board agenda. This practice caused particular concern as this
very public arena was the first time that lecturers had access to what were occasionally
very poor ratings.

Interestingly from our findings, lecturers appeared to be more willing to share the data
from upward appraisals with managers rather than students. A number of staff felt that
none of the results of upward appraisal should be reported to students. Only half of those
interviewed thought debriefing sessions between lecturers and students post appraisal was
a useful exercise. On probing this issue, a prominent concern here was that staff would
be put into an embarrassing position in such debrief ings if students had rated them badly,
rather than a lack of their value.

The reporting of upward appraisal results back to students was found to be very limited
in practice. Generally it seems that although a substantial amount of upward appraisal
existed in the Business School, on only one course were the results found to be reported
back directly to the students. No examples of systematic briefing sessions with students
using the data generated by upward appraisal were found.

Over 70% of our sample had experienced upward appraisal by students within the last
two years. A number of concerns emerge from our preliminary analysis. Firstly, it
seems little thought is given to the design of the upward appraisal instruments used. A
number of criticisms could be made about the psychometric and other properties of many
of the instruments used for upward appraisal found in this study.

Secondly, it would seem that the education sector could leam from industry's use of
upward appraisal in a number of areas, not least in feedback to those who provide the
data. Here industrial practice often encourages, in a few cases this is compulsory, those
being appraised to conduct face to face debriefing sessions with the appraisers (Bernardin
1986). Such debrief ings have the potential to be a very valuable developmental activity
but need to be conducted with care. Here industrial practice emphasises the need for
skilled facilitators in debriefing sessions. Upward appraisal, as practised in the higher
education sector, seems to generate a rich source of data but organisations do very little
with it. Our argument is that upward appraisal should be integrated into and inform
decision making on the broader learning and teaching practices of the institution via
incorporation into a student Personal Development Portfolio (PDP). Currently two of the
authors are developing a model that will achieve this.

Finally, although space has denied us the opportunity to examine the impact of upward
appraisal on student commitment and morale there is some evidence that this is far from
clear cut. Indeed, similar to the effects of other sophisticated interventions that are
designed to increase commitment, student appraisal of lecturers may paradoxically
actually decrease it (Bushardt et al 1991; Illes et al 1990). For example, one of our
preliminary findings is that students who had been involved in a number of
lecturer/course appraisal activities over the first two years of a three year part-time course
were increasingly reluctant to participate in such activities in their last year and some
refused completely.

Two main reasons were given for this. Firstly, they felt strongly that it reduced to little

CONCLUSIONS

190

QUALITY IN TEACHING AND TRAINING

more than a paper exercise. No feedback to them was given on the results of previous
reviews, nor did any positive action seem to ensue. One of their measures here was that
the lecturers they had complained about and given very poor ratings to were still lecturing
them. What is more, they continued with the same bad practices as before. No
improvements could be discerned. Further, a number of lecturers had, in the words of
one student, "got the hump" about the ratings they had been given and reacted
"petulantly".

A concern of some of the students was that following what were obviously very
negative ratings (they could only assume this as no formal feedback was given to them),
a number of lecturers were perceived as apparently retaliating by setting more work, more
difficult assignments, and giving lower grades. Given that upward appraisal had resulted
in no noticeable improvements and a perceived retaliation by lecturers it would be no
surprise to find a reduced student commitment and morale. Again, the potential to
prevent this negative impact of upward appraisal on students may derive from the use of
a PDP mechanism that integrates them into broader decision making.

Finally, one thing is evident from our preliminary analysis. Although there is a
considerable amount of research on the upward appraisal of lecturers, much of this has
been for research purposes only and concentrated on the more "technical aspects". Clearly
there is a need for a broader based approach that examines the impact of upward
appraisal on practice before whole-heartedly recommending its wider adoption as a
mechanism to enhance quality in higher education.

References

Banks C G and Murphy K R (1985) Towards Narrowing the Research -Practice Gap in
Performance Appraisal Personnel Psychology 35 pp 281-322

Bernardin H J (1986) Subordinate Appraisal: a valuable source of information about
managers Human Resource Management 25 pp 421-439

Bernardin H J and Beatty R W (1987) Can Subordinate Appraisals Enhance Managerial
Productivity? Sloan Management Review 28(4) pp 63-73

Bernardin H J and Klatt L A (1985) Managerial Appraisal Systems: has practice caught
up to the state of the art? Personnel Administrator 30 (ii) pp 79-86

Breed F S (1927) Factors Contributing to Success in College Teaching Journal of
Educational Research 16(4) pp 247-253

Buhalo I H (1991) You Sign My Report Card, I'll Sign Yours Personnel May 23

Bush G W and Stinson J W (1980) A Different Use of Performance Appraisal:
evaluating the boss Management Review November 14-17

Bushardt S C, Duhon D L and Fowler A R Jr (1991) Management Delegation Myths and
the Paradox of Task Assignment Business Horizons March -April 37-43

Crittenden K S and Norr J L (1973) Student Values and Teacher Evaluation: a problem
in person perception Sociometry 36(2) pp 143-151

UPWARD APPRAISAL

191

Feldman K A (1976) Grades and Student's Evaluation of Their Courses and Teachers
Research in Higher Education 5(3) pp 243-288

Feldman K A (1977) Consistency and Variability Among College Students in Rating
Their Teachers and Courses: a review and analysis Research in Higher Education 6(3)
pp 223-274

Feldman K A (1978) Course Characteristics and College Students' Ratings of Their
Teachers: what we know and what we don't Research in Higher Education 9(3) pp
199-242

Feldman K A (1983) Seniority and Experience of College Teachers As Related To
Evaluations They Receive From Students Research in Higher Education 18(1)
pp 3-124

Feldman K A (1984) Class Size and College Students' Evaluations of Teachers and
Courses: a closer look Research in Higher Education 21(1) pp 45-116

Feldman K A (1986) The Perceived Instructional Effectiveness of College Teachers As
Related To Their Personality and Attitudinal Characteristics: a «wiew and synthesis
Research in Higher Education 24(2) pp 139-213

Feldman K A (1989) Instructional Effectiveness of College Teachers as Judged by
Teachers Themselves, Current and Former Students, Colleagues, Administrators and
External (Neutral) Observers Research in Higher Education 30(2) pp 137-194

Illes P, Mabe C and Robertson I (1990) HRM Practices and Employee Commitment:
possibilities, pitfalls and paradoxes British Journal of Management Vol 1 pp 147- 157

Parasuraman A and Zeithmal V (1985) A Conceptual Model of Service Quality and its
Implications for Future Research Journal of Marketing Vol 49 pp 41-50

Pctrini C (1991) Upside-Down Performance Appraisals Training and Development
Journal July pp 15-22

Saleh F and Ryan C (1991) Analysing Service Quality in the Hospitality Industry Using
the SERVQUAL Model The Services Industries Journal 1 1(3)

Stinson J and Stokes J (1980) How to Multi- appraise Management Today June pp 43-53

Rutherford D (1988) Performance Appraisal: a survey of academic staff opinion Studies
in Higher Education 13(1) pp 89-100

Tsui A S and Ohlott P (1988) Multiple Assessment of Managerial Effectiveness: inter-
rater agreement and consensus in effective models Personnel Psychology Vol 41 pp
779-803

200

30 A total quality approach to managing
CBT development

Steven G Shaw and David R Shaw, Concordia University, Montreal and Curvet
Information Systems, Ottawa, Canada

INTRODUCTION

A "total quality" approach to management of instructional development has many
complex elements. Essentially, quality management involves (a) defining a quality-level
bracket, (b) improving the quality of the product within that bracket, and (c) reducing the
cost of production.

Our full model of quality management for an organisation reflects four levels of
information that are integrated horizontally and vertically between and across levels:
strategic (used by senior management), tactical (used by middle management), supervisory
(used by lower management) and functional (used by operational personnel). Originally
developed for large publishing companies, it is now being adapted to CBT in an
Instructional Design Quality Management Manual.

Here we present techniques for quality manrgement that can be implemented at a
tactical level or lower. That is to say, we provide a model and methodology that could
be implemented at the level of a training division or department, even if a commitment
to total quality management has not been embraced at a higher, strategic level by the
organisation as a whole.

The paper discusses techniques and approaches for reducing cost (including scheduling
approaches, estimating, job and version control, developing benchmarks, reducing cycle
time), improving quality (robust design, pre-design, concurrent engineering, design for
production, rapid development) and assessing quality (quality indicators). Specific
illustrations of the application of principles and methods of quality management to
development of computer-based training are provided.

IMPLEMENTATION

There are as many models of total quality management as gurus. However, you can
design your own program. In fact you will have to do this, anyway. Each organisation
requires a unique solution. T e methods and procedures presented below are thus a
shopping list, to help you get a quick start at the departmental level. They may not all
be suitable to your circumstances.

Another important consideration is that implementation may take years (typically two
to five). Progress may wax and wane during this period. Initial results may even be
confusing (Sutton 1991). To see a program through these difficult times, senior
management must have total commitment. Some of the common reasons for failure are
(Bellis-Jones and Hand 1989): inadequate resource levels; poorly designed procedures;
inadequate documentation; lack of training; inadequate supervision; poor documentation;
poorly motivated staff; changing priorities; and poor supporting computer systems.

Other researchers have reported these barriers (Keys 1991): lack of use of Theory Z in
human relations; lack of to'.al employee commitment; attempting to produce at capacity;
excessive emphasis on short-term profits; and outdated cost-accounting systems.

Most of the available models of implementation focus on manufacturing (Albrcgtsc et
al 1991). Applying TQM to services is more difficult, nonetheless it is applied
successfully in projects and service industries, including educational instruction.

ERLC

201

MANAGING CBT DEVELOPMENT

193

Here we look at some methods for local implementation at the departmental level, based
on SWOT analysis, an improvement and an implementation .strategy (Edosomwan and
Savage-Moore 1991).

SWOT Analysis

Develop policy and procedures to prepare an annual departmental marketing audit, using
an analysis of Strengths and Weaknesses, Opportunities and Threats, and Issues. Strengths
and weaknesses are internal factors that may or may not be within your control.
Opportunities are external factors frequently outside your control. Threats and issues are
posed as questions to stimulate discussion.

Improvement Strategy

Develop an improvement strategy for the department based on:

• activity analysis

• customer identification

• process improvement

Do a global optimisation of the department by defining core, support and discretionary
activities. Core activities, the reason for the existence of the department, use group skills
to add value to the business. Support activities clearly support core ones. Discretionary
activities are all of the running around necessary to get things done. Use time cards to
determine where effort is spent. Analyse the results to determine how to reduce time on
discretionary activities. Typically efficiency improvements of 10-20% are possible (Bellis-
Jones and Hand 1989).

Identify the external customers of the department. In the case of a CBT department,
these are likely to be other departments. Find out exactly what these customers want or
expect. Then work backwards within the department, identifying internal customers. Each
worker is a link in a chain of customers that ultimately leads to the external one. Each
link is at once a customer to one other link and a service provider to another. Focus on
the interface between links. Find out what internal customers want by asking them
questions like, "What do you need from me. How do you use my output, How can I
improve my service to you. Am I providing things you don't use" (Lee 1991).

Finally, develop a plan to implement a program of continuous process improvement
(CPI), perhaps using some of the methods discussed in this paper.

Training Needs

Assess training needs. Train supervisors and workers in how to measure quality and
implement quality programs. Focus on creating an environment where change and
innovation come naturally. Encourage adoption of the most current proven tools (Vesey
1991).

Implementation

To implement the CPI plan, set up a departmental steering committee. The purpose of the
committee is to identify quality projects or problems. It uses various analytical techniques

194

QUALITY IN TEACHING AND TRAINING

to decide which project or problems it should undertake. The committee establishes
quality teams to analyse and eliminate the problem (Nandyal and Welch 1991). The
findings of teams are binding on the committee; otherwise the exercise is pointless.
Teams fc~us their discussions in the context of tools like quality function deployment
(QFD) and fishbone diagrams, thus avoiding encounters that turn into gripe sessions or
are otherwise counterproductive.

COST REDUCTION

The procedures \^Iow are presented in a rough order of implementation. Having workers
do run diagrams, for example, while interesting will yield few benefits if there is no
overall quality framework in the department. On the other hand, error review can be
implemented readily without a framework. Similarly, maintenance, safety and security,
job and version control are basic organisational needs, presented here to reinforce the idea
that every aspect of an organisation has cost and quality implications.

Some companies, for example, even stipulate the maximum number of allowable rings
before someone answers the telephone. Again, it must be stressed that these are ideas for
consideration and may or may not suit your organisation.

Maintenance

Develop procedures for engineered maintenance cycles for facilities and equipment. Do
equipment maintenance in off-hours. Keep equipment in perfect condition.

Safety and Security

Develop procedures for safety and security. Include virus protection and backup and
archiving procedures for computer systems.

Job Control

Develop procedures for assigning docket numbers as the primary job control. Use a
mnemonic plus numerics. Use a checksum with numerics. Design information systems
to check mnemonic against customer lookup table, and to check numerics using
checksum.

Version Control

Develop policy and procedures for version control. In most cases a manual or sequential
change model is sufficient. Where multilingual products are developed, a selectable
change model may be required. The sequential model manages files, filenames,
file-version names and the sequence of deltas that make up a file version (Cronk 1992).

Suppliers

Develop procedures to purchase products like diskettes from companies that certify
quality at the source. If not, develop procedure to inspect goods immediately. Report

2<C

MANAGING CBT DEVELOPMENT

195

defects as soon as possible. Develop procedures to use economical order quantities based
on available discounts and inventory costs.

Undercapacity Scheduling

Use undercapacity instead of capacity scheduling. As its name suggests, it schedules
somewhat less than a maximum daily workload. For example, if there are 7.5 hours in
the workday, you might schedule just enough work for 6.75 hours.

The basis of undercapacity scheduling is to give workers enough time to do the job the
right way the first time. This avoids costly re-runs due to errors. Also aim to do the same
amount of work each day.

Undercapacity scheduling also allocates time for training and problem solving, and is
thus key to quality management.

Adopting undercapacity scheduling may be difficult psychologically for supervisors and
managers who feel they must run a tight ship. They keep the pressure on, and have
difficulty understanding that undercapacity scheduling actually increases annual
throughput. A production group producing a consistent, predictable amount each day will
outperform one using capacity scheduling.

To achieve undercapacity scheduling, use a Master Production Schedule having three
divisions: master schedule, actual demand, and available-to-promise. Set up the master
schedule based on forecasts. As orders are booked, enter hours for each job under actual
demand and compute the difference between forecast and demand. The difference is
available-to-promise to user departments. Knowing the available-to-promise obviously
helps the scheduler. It avoids false promises and the inevitable user dissatisfaction they
entail.

Lot Size

With undercapacity scheduling in place, gradually reduce the lot size. Instead of giving
a worker 100 frames to complete by a given deadline, give him/her 20 frames and a
shorter time frame. This increases the required supervision but benefits include less
pressure on workers, hence higher quality, faster feedback, and increased flexibility. If
one station frees up earlier than expected, work can be redirected.

Cross-Training

To enhance flexibility and productivity, cross-train workers in each other's job. For
example, with training, graphic artists instead of the instructional designer can specify
graphic requirements. Cross- training of this type may be increasingly necessary with
multi-media, where Renaissance-type instructional designers may be rare (see Training
and the Development of ISD Expertise, below).

Set-Up Time

Investigate and improve setup time. This is the time necessary to setup for a job or
particular kind of work. The ideal is one-touch setup - one button to push. Reducing
setup time cuts costs through increased productivity and increases flexibility.

204

196 QUALITY IN TEACHING AND TRAINING

Just in Time

As lot sizes are reduced gradually, move to "just in time" (JIT). JIT exposes problems,
and enforces problem solving. Normally managers build extra inventory and put slack
time into schedules to compensate for known problems. Slack time is a common practice
in project management for CBT. Eliminating these buffers reduces overhead and
associated costs. It exposes problems, forcing you to find remedies. This improves
quality, reduces waste and re- runs and lowers work-in-progress inventory carrying costs
(Alonso and Frasier 1991).

For example, when schedules are not being met, the tendency is to release jobs early,
in the hope that somehow deadlines will be met. In practice, the customer will use the
extra time to issue change orders, increasing the uncertainty in the schedule, causing the
system to become chaotic.

JIT principles can also be applied to project management, reducing project life cycles.
Any management practice that shortens the planning horizon improves project profits
(Alonso and Frasier 1991). Planning frequently (i.e., monthly) increases the accuracy of
forecasting and scheduling. In one case, "simultaneously improving all of the delays and
goals under management control [benefited] the client's net cash by over 13% of sales"
(Alonso and Frasier 1991).

Scheduling

There are various approaches to scheduling, but the bottom-line objective of JIT is a zero
deviation from schedule.

Use backward scheduling for periodical work or work by appointment. It starts with the
distribution date then works backwards to arrive at the date when files must be delivered
to the production department.

Use forward scheduling for services on demand, i.e., rush jobs. Calculate forward
schedules from delivery of raw data or the time at which personnel will be available to
work on the job, to arrive at the delivery date/time. Rush jobs may not affect mainstream
CBT activities but may be a factor in related service departments.

Synchronised scheduling is a variation of backward scheduling. The final assembly date
establishes the schedule for subprocesses, using standard offsets for lead time. Base the
off -sets on production benchmarks. When calculating lead-time, consider these elements
in descending order of significance (queue time often accounts for 90%) (Schonberger
1985):

• queue time

• run time

• setup time

• wait (transportation) time

• inspection time

• move time

• other

Whichever technique you use, the resulting schedule must be regular. If it is, you can
then aim to reduce the required time project by project. The scheduler must keep a close
eye on work in progress (WIP), to avoid unnecessary levels or capacity scheduling. This
improves service by reducing queue time, responding more quickly to users, reducing
uncertainty in schedules, and giving users more reliable status information. As lead times

20Z

MANAGING CBT DEVELOPMENT

197

shorten, forecast accuracy improves. There is less need for shop-floor ccmrol by
schedulers and supervisors.

Obviously there are practical limits to reductions in WIP. There should be enough jobs
to keep work centres busy. Loads at work stations will vary. There is pressure to overload
some to avoid underloading others. However, increasing the workload will only increase
queue times, making schedules less realistic, with al! the chaos and inefficiency that
implies. Another difficulty is that it is often hard to predict queue times for any particular
job. It may queue at several workstations before completing its routing.

Like forecasting, scheduling is necessarily inexact. For example, vacation periods are
usually unknown. Personnel book off sick. Customers miss deadlines. Nonetheless,
scheduling is necessary and the key to strategic planning, eg:

• changing procedures

• modifying production methods

• introducing new technologies

• assessing resource requirements

• forecasting budgets

Good practices make it easier to meet schedules. First identify the key success variable
for your operation (eg. on-time delivery) and design a system that produces the required
results. Re- assess this key variable from time to time.

Calculate the daily capacity of your production centre in hours rather than production
units such as frames. Determine the effective capacity using a formula such as:

Effective daily capacity = ((H-L)/H)(HxSxE) where:

H = normal hours in a day excluding lunch, breaks, etc.
S = number of shifts in a day
E = number of employees

L = loading factor for undercapacity scheduling (eg. 0.5 hours/shift)
<(H-L)/H) = undercapacity factor

When drawing up schedules, be aware of pre-established commitments for regular jobs.
Make allowances for rush jobs, using estimates based on historical data. Schedule rush
jobs separately from regular ones. Consider doing rush jobs, altogether, every day at a
certain time. Use simpler control methods for rush jobs and charge them out at cost-plus.
Use a quicker invoicing procedure for rush jobs.

Also estimate expected sickness and absentee levels from historical data and maintain
a list of part-time employees. Encourage employees to post their vacation periods as early
in the year as possible. Schedule regular preventive maintenance, preferably in off-hours.
Schedule to avoid known bottlenecks, levelling peaks.

Understand and exploit the paradoxes. When schedules are irregular, there is a tendency
to rush jobs into production as soon as possible, in the hope that they might be completed
on time. Conversely, make it a rule to release regular jobs into production as late as
possible, ie. just in time. This reduces the chance of late changes. Before releasing a job,
be sure all information is complete and all components are available. If the job is
incomplete, set deadlines for the supply of missing information. Charge for last-minute
changes to jobs based on accurate records of time and materials.

Update your schedule using information from time sheets and observation. Use flexible
production, changing from single to partial or full second shifts as necessary. Determine
economical breakpoints between the various options. For example, use temporary labour

20G

198

QUALITY IN TEACHING AND TRAINING

for the first 5% of extra work, use overtime for the next 5%, then reduce customer
service by rescheduling if you can't justify a full second shift.

Estimating

Develop a policy to base customer charges on estimates instead of actuals (excluding
change orders). This enforces problem solving if jobs cannot be completed within
estimates. Develop a procedure to base estimates on benchmarks and the depth of design.
Develop procedures to place estimates on dockets (Ziegfeld et al 1985). This contracts
workers to the production budget and also encourages faster feedback, allowing them to
renegotiate the contract if they believe the estimates are unrealistic.

Benchmarks

Develop production standards or rates for all of the various parts of a job. Use work
measurement techniques such as statistical sampling, stop-watch studies, and tables of
industry norms. Develop procedures for devising and monitoring internal benchmarks for
activities like writing, graphics* programming. Use meaningful time frames for
benchmarks. For example, it may be unrealistic to expect a writer to do 1500 words/day
but realistic to expect 7500 words/week. Don't base benchmarks on completed
components such as frames/day, pages/day, etc. Review benchmarks monthly.

Depth of Design

An important consideration is the depth of design (good-enough factor) the customer
requires, eg. a Chevrolet or a BMW. Many CBT projects are budgeted simply by working
backward from the customer's budget. If he/she wants three hours of courseware for
$30,000, and the instructional company charges $60/h, then the depth of design is 200:1
hours. This is fine if the customer truly understands what 200:1 means, or if 200:1 is
good enough. Use QFD to determine customer needs and depth of design (see IMPROVE
QUALITY); use simpler measures such as reverse calculation only for reality checks.

Cycle Time

Reducing cycle time (also called time compression management, time-based
competitiveness and time-based innovation) for products or projects, a quality method in
place in Japan since 1981, delivers major financial benefits, while allowing rapid response
to changing market conditions (Musselwhite 1990). It is closely allied with concurrent
engineering and JIT.

Shorter cycle time, a response to quickly changing markets, gives quicker market entry,
allows organisations to charge a premium price for new products, reduces costs and
improves quality (Musselwhite 1990). It creates opportunities to increase market share,
market leadership and profits (Vesey 1991). An additional benefit is that turning over
projects more rapidly increases contact with the market and increases the rate of learning
within the organisation.

Cycle-time reduction cannot be implemented incrementally. Incremental change is
efficient in building tools, like supporting computer systems, but will not significantly
affect cycle time.

MANAGING CBT DEVELOPMENT

199

Major gains are only possible through process elimination and innovation, especially at
boundaries between departments. At this level there are two primary objectives: speeding
new products to market and quickening response to customers in production and delivery
(Sheridan 1991). Set targets like reducing cycle time by 50%. One very interesting
approach is to concentrate on the time to break-even, rather than the entire product or
project cycle (IM Management Roundtabie 1990), Break-even time encompasses all cash
flow from a project and incorporates the entire development process from concept to sales
(Inglesby 1991).

Start by speeding up existing processes as much as possible (eg. use E-mail). This gives
faster feedback, too, so problems are dealt with promptly and customer satisfaction is
increased. Next, re-design the existing process, through organisation and procedures.
Supporting business and computer systems must evolve simultaneously. Scalable solutions
based on open systems are a prerequisite. They must be built with fourth-generation
languages not aligned with any platform, operating system or database. The underlying
data model must permit long-term growth of the department's database (Inglesby 1991).

IMPROVE QUALITY
House Style Guide

Develop policy and procedures for creating and maintaining a house style guide. Develop
procedures for customising the guide to meet individual customer/project needs.
Incorporate the guide into style sheets and templates linked to authoring tools.

Glossaries

Develop procedures to build and maintain glossaries of nomenclature peculiar to the fields
your organisation services.

Control Points

Make every workstation a quality control point, reducing the bad product passed forward.
Develop check lists for each station.

Immediate Evaluation

Evaluate every module right after it is completed. Develop automated shells that
completely compile CBT programs overnight, so c valuators have the latest version each
day.

Make everyone responsible for quality: give each worker responsibility to flag defects
even if not in their area of responsibility.

Defect Review

Develop procedures to trace defects to their source immediately upon detection. The
purpose is to devise a system improvement - not to blame the individual. Where defects
are not found out until a later process, or until the customer uses the product, provide fast
feedback right to the originators.

20£

200

QUALITY IN TEACHING AND TRAINING

Defect Correction

Make every workgroup responsible for correcting its own defects. Sometimes imagination
is required. At a car factory in Canada, the purchasing department bought the wrong size
of tires. They were mounted on 50 cars before the defect was caught. To correct the
defect, the entire purchasing department was mustered in the parking lot to change the
tires, using the standard tools found in the trunk.

Internal Customers

Develop a policy making each individual/group a customer/service provider to other
individuals/groups.

Internal customer service items: develop a procedure for defining internal customer
requirements and a related benchmark and/or checklists.

Pre-Design Planning

Invest in pre-design screening, evaluating and planning. The larger the design project, the
more important the planning phase (Ziegfeld et al 1985). As a rough guide, aim for
distribution of effort like planning (40%), CBT design (25%), process design (15%),
production, test, delivery (20%). Obviously distributions like this cannot be achieved with
traditional ID and CBT methodologies.

Quality Function Deployment

QFD is a set of planning and communication routines that facilitate concurrent
engineering and give focus to meetings of quality teams. It assumes that marketing,
design and production people will work closely together from the first conception of a
product. It allows organisations to learn from customer experience and to reconcile what
customers want with what instructional designers can reasonably build (Hauser and
Clausing 1988). The procedures in QFD are simple:

1. Develop customer attributes: find out what customers want, expressed in their own
words. These items, typically 30-100 of them, are customer attributes (CAh They may
also include demands of regulators (eg. ISO 9000, MIL specs). Some attributes may
be gathered into groups called bundles.

2. Weight customer attributes: weight each attribute in percent, and place the data in a
table.

3. Analyse competitive advantage: on the right-hand side of the table, place a perceptual
map giving ihe customer's perception of your capability of meeting an attribute (a),
compared to the competition (b). Rate attributes on a scale of 1-5 (5 = best). This
may relate closely to the type of authoring tools used or the expertise of your
instructional-design team. One impediment here is that many design organisations are
largely tool driven, since few existing tools are modular or extensible in any
significant way. Competitive advantage data is usually collected using marketing
methods. Identify areas of advantage (plan to maintain them) and opportunities for
improvement.

2VD

MANAGING CBT DEVELOPMENT

201

4. Develop design characteristics: along the top of the table, develop design
characteristics (DC) affecting each customer attribute. A DC may affect more than
one CA. Express design characteristics in quantifiable terms. Analyse carefully,
brainstorm, avoid vagueness.

5. Assess design characteristics vis a vis attributes: determine what is the strength and
direction of the relation between each DC and its associated CAs. Fill in the body of
the table with this information, linking the DCs and CAS. Check for consistency
between customer evaluations of CAs and measures of the related DCs.

6. Ascertain what are the interdependences among DCs: set up a matrix depicting these
relationships.

7. Assess design characteristics vis a vis competitive benchmarks: furnish objective
measures for current DCs.

8. Set design targets: determine what are ideal values for the DCs.

9. Assess design targets vis a vis desired cost and quality levels: determine whether
design targets are within the quality level required by the customer and the
appropriate cost bracket. Refer to the matrix depicting interrelations among DCs for
assistance in making decisions regarding trade-offs to meet these constraints.

10. Design development and production processes: establish development methodology
and plan production based on target design.

Design for Production

Design CBT applications so a) they readily fit your existing authoring shells, and b)
production can be automated as much as possible. If customers want features not readily
accommodated by the shell, don't shoehorn the application. Extend the shell, or develop
a new one. Use tools like HyTime, the SGML-based hypertext and multimedia structuring
language.

Robust Design

Robust design (Genichi Taguchi methods) builds in tolerances for variables that are
known to be unavoidable. Identify the controllable and uncontrollable factors in your
production process. Optimise the controllable factors within a range that minimises any
effect of the uncontrollable ones. Any deviation in quality, no matter how small, increases
the product's ultimate cost, including warranty liability and lost customer goodwill.

Concurrent Engineering

Concurrent engineering is one of the prime tools for cycle- time compression. It uses
cross-functional or multi -disciplinary teams to provide more effective product designs,
design products and production processes simultaneously, reduce time to market and to
link producible designs to highly productive processes (St. Charles 1990).
Cross-functional teams work concurrently on all aspects of the project.

Traditionally CBT uses a linear process: research, product design, process design,
production, sales and distribution. One department does its thing, then tosses the job
"over the wall" to the next (Shina 1991).

This linear process has many review stages and is costly. Product is recycled often to
correct errors or incorporate changes. Manufacturing studies have shown that 70-80% and
75-80% of production costs are locked in during the design stage (Creese and Moore
1990; Musselwhite 1990: Sutton 1991).

210

202

QUALITY IN TEACHING AND TRAINING

Concurrent engineering is similar to rapid prototyping except ihat it emphasises
management as well as design skills, focussing on customer needs (Creese and Moore
1990). One of its benefits is that typically most design changes occur before production
starts (Turino 1991). A useful aspect of rapid prototyping is to use automated shells to
load all completed modules every night, so e valuators can work each day with the current
version.

Concurrent engineering involves all departments simultaneously in : parallel process.
Practice has shown there should be a minimum of five people on a team, to prevent one
person from dominating. Such cross- functional teams have reduced new-product
introduction time by 50-75% (Musselwhite 1990). According to Hauser, "...the use of
interfunctional teams benefits design.... [but] what should these people talk about? How
could they get their meeting off the ground? This is where the house of quality (QFD)
comes in," (Hauser and Clausing 1988) or the committee approach advocated by Nandyal
(Nandyal and Welch 1991) [see IMPLEMENTATION] .

Run Diagrams

Develop procedures for individuals to monitor their own performance using run diagrams.
Identify and investigate system problems.

Technical Planning

A major strategic issue facing CBT organisations is responding to technological change.
Research has shown that organisations having a high-level technology plan are more
successful than others (ITAC 1990). A technology plan has many aspects.

Products can be brought to market very quickly by having three design teams
simultaneously develop solutions, but deliver them to market on different time lines. In
some large-scale CBT projects, this may be a means of version planning. An original
product is delivered quickly to the customer; significant change orders are incorporated
into V.2.n; meanwhile more sophisticated versions are in development.

The same model may be used in planning for CBT technology, in the sense of pacing
the technology. An organisation producing text and graphic frames could plan to migrate
to multi -media. At the moment, muhi -media is expensive and suited only to certain
training situations. However, as multi-media functions are incorporated into operating
systems, the cost will decrease while user expectations increase.

One problem is that we get locked into the paradigm of existing operating systems.
DOS users, and even Windows users, don't understand the seamless environment of
Macintosh, and thus cannot envision the type of products it enables. Similarly, those of
us who have not fooled around with desktop video cannot truly envision the applications
it enables. The technology plan should therefore include a small budget for experimental
technology.

PROCESS IMPROVEMENT

The previous sections of this paper have addressed techniques for reducing cost,
improving quality and assessing quality. Many of the techniques and mechanisms (e.g.,
JIT) presented are designed to expose problems in the processes underlying design,
development and production functions, as much as to reduce cycle time. The following
sections focus in more detail on issues relating to the conception and execution of these

211

MANAGING CBT DEVELOPMENT

203

processes, with a view to making suggestions about how the cycle time for creation of
CBT products might be compressed.

Time compression is a major topic in the current literature on TQM and industrial
engineering (cf Inglesby 1991; Musselwhite 1990; Sheridan 1991; Vesey 1991). The
advantages of quicker product delivery in a manufacturing context are obvious: there are
potential cost reductions and the possibility of greater market share through touching the
market first, or more often in a given time span. Shorter cycle time also allows for a
more incremental approach to product development and innovation and this brings lower
risks. The analogy between manufacturing and CBT development is not perfect, since in
most cases CBT products are custom tailored solutions rather than mass market products.
However, the benefits of time compression are still clear: instructional development is one
area in which, paradoxically, advances in technology have had the effect of significantly
increasing the time and costs associated with design, development and production phases.

There are two kinds of decisions which can be taken be taken with respect to time
compression: total and incremental (Vesey 1991). Incremental decisions modify existing
procedures to produce minor or modest gains in efficiency and time compression. Total
decisions concern the radical redesign (or elimination) of processes and yield major
improvements. Vesey reports a number of striking cases to illustrate the kind of impact
total decisions can have. For example, the Ballistic Systems Division of Boeing
Aerospace Corporation reduced design analysis from two weeks to 38 minutes.

Both kinds of improvements - radical and incremental - are discussed below. Arguably,
CBT development has a great deal to gain from incremental decisions. Even in
sophisticated manufacturing which involves intensive design engineering efforts, product
design typically accounts for 20% or 25% (Sutton 1991) of the overall cycle. In CBT,
design typically represents up to 75% (Merrill 1988) of the overall cycle. Putting aside
claims concerning the complexity of instructional design and the ill-defined nature of
instructional design problems, these comparative figures suggest that existing procedures
have not been optimized.

Incremental Improvements

Areas where improvements can be made include: the management of information
generated in design and development activities, the design and implementation of
error-proof processes with automatic checking devices in all phases of ISD and
production, the introduction of new approaches in the knowledge acquisition phase of
ISD, the elimination of prototyping, the automation of portions of the ID and production
processes, and explicit strategies to develop expertise among instructional designers.

Information Management in ISD

There are many kinds of information generated during CBT development relating to
initial conceptualization, feasibility, planning, development, production, delivery and
evaluation. These include: the results of task and content analyses, objectives and
sequences, media selection and instructional strategies analyses; test item banks; syllabi
or outlines; scripts, course and lesson design specifications; graphics and text components,
libraries of computer code. This information is highly interrelated. In fact many categories
are the direct result of a transformation process that takes another category as input: tasks
are converted to objectives, objectives are translated into test items, objectives are
sequenced into lessons, and so forth.

204

QUALITY IN TEACHING AND TRAINING

In small scale CBT development projects, with experienced teams, it is often possible
to adopt a relatively informal approach to documenting the results of design and
development activities and managing this information. In large scale projects, a systematic
approach t information management founded on the construction and maintenance of
linked databases associated with each stage in the ID process can provide efficiencies and
reduce the risk of major, costly blunders.

1. In the first place, information retrieval capabilities associated with databases can be
used to support (and in some cases, to automate) analysis and design. For example,
a database which can provide information concerning the relative proportion of
different kinds of objectives can facilitate the design of a generic lesson structure.
Another example: information concerning redundancy of tasks across different jobs
could result in a more rational approach to training which consolidated these
redundant tasks in one package. From a "performance engineering" standpoint the
same information might be used to redesign jobs and reduce the need for training.

2. Reports generated from databases can also assist in estimating production and
development costs more accurately. For instance, a complete breakdown of objectives
according to the media and instructional strategies selected for them will facilitate a
more accurate projection of production time and costs. If it should prove necessary
to compromise media or strategies selections, owing to constraints, then the same
information will enable instructional designers to make the necessary adjustments
more rapidly, consistently and rationally.

3. Linked relational databases ensure that design and development decisions are
consistent across the ISD process. The database system can be designed so that a
change in an element associated with one phase in the ISD process can result in flags
being set against related elements that are products of other phases. A change in an
objective, for example, would result in flags being set against related artifacts such
as test items, graphics selection, strategy selection, sequencing information and so
forth. Automating the regulation of consistency in the ISD process obviously has
even greater benefits when the content is not entirely fixed or frozen at the time
design activities are undertaken. This is often the case in large scale projects in
technical fields, where development of systems operation and maintenance training
may have to begin before all systems are fully engineered and in production. The
implication for training development based on initial design engineering specifications
is clear: there may be many revisions. Under this condition it is essential to have tools
which can register these changes and maintain the integrity of the ISD process by
identifying inconsistencies as they are introduced and by facilitating the process of
rippling required changes through the various subsystems of the evolving instructional
design. This use of instructional design databases to ensure consistency is also an
example o* tlie utilization of an error-proof process with automatic checking built in.

4. Finally, ID databases with automatic report generation that will track the degree of
task completion for various phases can obviously be used for project management
purposes ?tid to assist in producing intermediate deliverables in the form of
documentation of the various ID phases.

Barriers to Implementing Information Management Systems in
ISD

Despite the obvious applicability of database approaches to managing development-related
information in ISD, only a small proportion of commercial developers exploit the

213

MANAGING CBT DEVELOPMENT

205

possibilities. An informal survey of five Canadian CBT development operations indicated
that not one employed database management systems (DBMSs) in the ISD process in a
significant way (ie. in a way that links ISD phases and which exploits the potential of
DBMSs for design decision support). Even in large scale projects, the operations surveyed
routinely used DBMSs only for one phase (task analysis), and in some instances for
project management purposes.

One reason for the limited use of DBMSs in ISD is the lack of commercially available,
iSD-specific, applications. Commercially available authoring systems often include as an
option a DBMS that can be used to implement computer management of course delivery
("computer-managed instruction") but they do not offer DBMS tools for ISD.

A barrier to creating custom DBMS systems in-house is the issue of the different
requirements which would have to be addressed. Different phases of the ISD process
require different kinds of interactions with information and correspondingly different
forms of representation and different interfaces (Gibbons and O'Neal 1989).

In the analysis phase, for example, it is crucial that the developer be able to manipulate
structural representations (course structures, lesson structures, unit structures) as well as
the data associated with individual elements in a structure. In media selection, individual
objectives are associated with ranked choices from among available media. With the
kinds of analyses associated with this phase, wherein scenarios associated with alternative
mixes of media are compared on the bases of instructional value and overall costs, a
spreadsheet representation is more appropriate than a structural representation.

A fully integrated tool for ISD would thus have to combine flexible outlining functions
with relational database capabilities, and ideally would also handle graphics elements as
data. In addition, there would be requirements for handling multiple versions and possibly
collaborative work (multiple developers). ' A tool with these features and suitable
interfaces is feasible (at the boundaries of the current envelope of technology) , but would
require a significant research and development effort.

Knowledge Acquisition

Standard ID practice is based on a separation of subject matter expertise and pedagogical
or instructional design expertise. Instructional designers work with subject matter experts
(SMEs) to extract content, but apply their own expertise based on experience,
instructional theory and instructional design models to make decisions regarding how to
best structure and present that content. Knowledge acquisition is very often a major
bottleneck in ID where complex knowledge or skills are concerned.

Traditionally, the central tools of knowledge acquisition (and knowledge representation)
in ID have been procedural and hierarchical task analyses. One problem associated with
these tools is that where cognitive tasks are concerned experts' knowledge is often highly
proceduralized and telescoped, meaning that several explicit procedures have been
collapsed into a single step. As part of the knowledge acquisition job an instructional
designer working with a highly skilled SME must facilitate the explicit recall of steps or
procedures that have become tacit or highly automated.

In a typical scenario, the instructional designer will observe the SME performing a
series of tasks, or listen to an account of the performance of those tasks, and will prompt
the SME for further details or explanations where these appear to be missing. This
activity is time consuming and can be highly problematic, depending on such factors as
the quality of communication between the SME and the instructional designer, the level
of knowledge of the SME, and the fluency the instructional designer is able to acquire
in the language of the SME's domain.

214

206

QUALITY IN TEACHING AND TRAINING

More meticulous and detailed approaches to knowledge acquisition and knowledge
representation have emerged in the field of knowledge engineering (KE) which is
associated with the development of expert systems, decision support systems and
intelligent tutoring systems. Both KE and traditional ID use interviews and protocol
analysis as their basic methodologies. The basic difference between the two concerns the
emphasis in KE on how the expert structures his knowledge. Task analysis focuses on the
tasks performed by an expert; KE also emphasizes the expert's knowledge base (which
provides an explanation for why things are done in a particular way).

KE is even more labour intensive than conventional ID knowledge acquisition and
representation and is warranted only when the instructional goals necessitate the
additional effort (ie. when quality is defined at that level). However, having said that, it
is worth considering whether any innovations in methodology that have arisen in KE
could be transplanted to conventional ID practice to improve efficiency. KE is so time
consuming that the impetus driving process improvement has been higher than in ISD.
Since the basic methods are the same, many advances in KE would potentially be
transportable to ID practice.

KE has taken two directions that are relevant. Both involve the attempt to transfer more
of the effort in knowledge acquisition on to the expert. This makes considerable sense on
an intuitive level, since the element contributing most to the bottleneck in knowledge
acquisition is the instructional designer's ignorance of the domain.

The first approach attempts to eliminate the instructional designer's role, replacing her
with a set of computer-based, possibly interactive, knowledge acquisition tools that the
SME works with alone to build at least a first-cut explicit representation of her
knowledge. In the area of knowledge-based systems development there has been some
success in the strategy of providing shells to SMEs for producing rule-based systems.
SMEs are then able to develop and test the rule base themselves, without the requirement
of learning a complex language and programming environment.

Another example of this approach deals with declarative rather than procedural
knowledge. Shaw and Gaines (1987) have developed a methodology which they call
Personal Construct Technology (PCD. In the PCT approach, an SME begins by
identifying all the major elements in the domain. This domain is then decomposed into
all possible triads. The SME then is required to identify, for all triads, how two elements
are similar and how they differ from the third. At the end of this process an exhaustive
taxonomy of the domain has been produced and the categories or features that underlie
the taxonomy have been made explicit. Shaw and Gaines have developed computer-based
incarnations of PCT that will guide the SME through this process in an interactive mode.

PCT is very much a bottom-up approach, and real efficiencies in KE rather will likely
come from top-down approaches which present a framework that can guide and focus KE
activities in a given type of domain. Some work, such as Breuker's concerning generic
models of diagnostic tasks, is being carried out in this direction (Breuker and Wielcnga
1987). While results to date have not indicated radical breakthroughs, this would seem
to be an area worth watching.

An alternative to approaches such as PCT which seek to replace instructional designers
or knowledge engineers is one which utilizes SMEs to a greater extent in a number of
variations on the standard appnjach to SME interviews and collection of protocols. In one
such variant the knowledge engineer observes two SMEs who, in turn, pose representative
problems and provide solutions for those set by their colleague, while the protocols are
recorded. The technique is based on the premise that the task of defining a representative
and comprehensive set of problems or scenarios usually can be carried out much more
efficiently by two experts than by a SME working alone with an instructional designer.

2 IZ

MANAGING CBT DEVELOPMENT

207

Training and the Development of ISD Expertise

The knowledge acquisition bottleneck is one area where appropriate training can often
play a role in increasing efficiency (Wielenga and Breuker 1986). Even those who have
had formal schooling in ID have typically never received any specific training regarding
knowledge acquisition from SMEs. There is reason to believe that training in a structured
methodology for knowledge acquisition would improve the process.

Lack of general ISD training and related design experience is also frequently a problem
in instructional development operations. Many instructional designers lack formal training
in their field; those with formal training who prove to be very competent often move
quickly from their roles as instructional developers to assume supervisory or management
positions. In the area of CBT in particular, there are apparently only a relatively small
number of designers and developers with significant work experience. In cutting edge
are?^ such as multimedia systems and intelligent computer- assisted instruction the
situation is even more serious: there is little cumulative knowledge and experience to
draw on in : [tempting to formulate design theory and development methodologies. Such
advanced areas also seem to entail redefinitions of the traditional roles of instructional
designers.

Several responses to the problem posed by insufficient numbers of expert instructional
designers have beer, advanced in the field of CBT development. On the one hand, there
have been attempts to engineer systems with built-in expertise that might eliminate or at
least reduce the need for expert designers. This is the thrust of the SOCRATES project
(Doucet and Ranker 1990) and also to some extent of Merrills *s IDexpert project (Merrill
and Li 1989). IDexpert. the better known of the two, provides instructional design
decision support and aims at a level of automation of CBT development through the
creation of a series of "transaction shells," each of which is intended to handle a specific
type of instruction by adapting reusable course component structures.

Conversely, some commentators have argued that the key to improved performance and
quality in development must lie in developing appropriate skills in instructional designers.
Duchastel insists that the instructional designers must be in "creative control" (Duchastel
1988). If you accept this position, the question becomes how one can develop that
expertise as quickly as possible. In addition to conventional team- building strategies,
technology can perhaps play a role here, also. Duchastel has argued that designers need
to have access to their own design experience in some expedient fashion, as well as to
the cumulative experience of design teams. Misselt (Misselt 1989) has similarly argued
the need for a courseware design library and has provided some analysis of the
characteristics required of such a library. Other, more explict strategies have also been
explored. RRI (Research Reference Interface) is a component of IDexpert that provides
"robust" explanations justifying the design prescriptions provided by IDexperfs
instructional design engine (Li et al 1991). IDE is a hypertext-based tool that allows
developers to record reasons and explanations for design decisions along with course
content. It provides for the management (compilation and retrieval) of this information
as well as the means to represent it in different ways. These capabilities facilitate group
decision making, make developers more reflective through requiring an explicit statement
of design decisions and their underlying rationale, and provide a database of design
decisions and explanations from which to learn.

Cross training of CBT development teams is another area where gains can be made.
Communication of requirements across functions will be facilitated if team members have
a better understanding of the different tasks. Recent developments such as hypermedia
and CD-I and DV-I applications require greater cooperation and collaboration among

208

QUALITY IN TEACHING AND TRAINING

team components, a larger, more central, role for graphics design, and a different, more
creative (narrative), approach to course design. With this increased interaction among
different functions, cross training may be especially beneficial. In some forms of
development, cross-training has been shown to cut new product introduction time by as
much as 50% (Mussel white 1990).

What Do Instructional Designers Really Do?

ID theory is not an area of knowledge that is highly developed. While there exists a large
number of procedural models, which are very similar to one another, for doing
instructional design there is little research and hard data concerning what instructional
designers actually do. The standard procedural models provide a general methodology for
developing instructional artifacts; they do not address the specific strategies and tactics
and representations that an instructional designer utilizes when performing ID tasks.
Better understanding of ID as a design activity per se is needed if major breakthroughs
are to be made. To date only a very little research has been carried out in this area (eg.
Pirolli and Greeno 1988; Pirolli and Greeno 1990; Jonassen and Wilson 1990).

Eliminating Prototyping

ID is generally justified on the grounds of effectiveness, not efficiency, and it is well
known that under normal "real world" constraints all steps of the conventional ID models
cannot be carried out in the degree prescribed. Recent improvements in methodology have
been made in CBT development by combining elements of rapid prototyping with some
components of traditional ISD. Major gains in time compression can be achieved if the
prototyping stage, which is itself highly intensive, can be eliminated.

Prototypes play several roles in CBT development: feasibility (a prototype can provide
a very good sense of the total development effort required to produce the complete
product, where new tools are being used and/or new designs are being implemented);
verification of the effectiveness of the design (prototype evaluation with end users);
communication of standards to production teams and agreement on standards with the
customer. Ideally, prototypes should not play a role as a medium for extensive design
experimentation, unless the context is R & D, rather than a CBT production operation.
Properly exploited, prototyping eliminates any possibility oi the need for expensive
revisions arising during the production phase.

Elimination of prototyping as a major stage will only be possible w'ien: (a) production
tools allow for more modular construction of CBT, to facilitate revisions; (b) clients have
greater familiarity with CBT and hence are more fluent in de&criblng the quality level
(instructional design features, aesthetic and functional characteristics) they require and
better able to understand what they are being offered, and (c) our knowledge progresses
to the point where cumulative experience, new procedural models, and detailed
prescriptive theories can replace the role of prototypes in ensuring effectiveness.

QUALITY INDICATORS

A number of examples of quality indicators are given below. But don't monitor
everything. Select a few indicators (or devise your own) that are meaningful and focus
on them. Don't build a bean-counting bureaucracy.

MANAGING CBT DEVELOPMENT

209

Overhead

Develop procedures to monitor overhead as a percentage of processing cost
(activity-based accounting). Take total costs for direct labour, materials, processing and
overhead. Subtract labour and materials, and divide the balance by processing costs. Now
for each project, multiply its processing costs by the remainder to get a contribution to
overhead. Add project costs for labour, materials and processing to get total project costs.

Customer Turnover

Monitor the percent customer turn-over. Long-term customers are easier to sell on
services, reducing marketing costs. A high customer turnover indicates dissatisfaction
with quality in some sense.

Development Cost

Monitor the ratio of development hours to course hours. A decreasing ratio shows an
improvement in quality, probably in the design process.

Unit Cost

Monitor unit cost. A declining cost indicates improvement.
Unit Effort

Monitor unit effort. A declining rate indicates improvement in cycle time. Monitor both
effort and cost because in some cases effort may decline while cost does not (wage
settlements may increase costs).

Utilization Rate

Monitor the job queue using the utilization rate, U = A/u, or percent of time busy.
Utilisation rate is the ratio of the mean arrival rate of jobs (units/h) to the mean service
rate. For example, an arrival rate of 2.1 units/h and a service rate of 1.9 yields a
utilisation rate of 1 .10. Anything over 1.0 means »^hs cannot be completed during normal
hours and overtime is necessary. A rate of 0.85, representing undercapacity scheduling,
is suggested as an experimental target.

Cost Index

Monitor the cost index for a CBT project. It is the ratio of actuals/estimates. A value
greater than 1.00 represents over expenditure and the need for corrective action.

210

QUALITY IN TEACHING AND TRAINING

Achievement Index

Monitor the achievement index. It is the ratio of actual percent complete on a project to
the planned percent complete at any given time. A value less than 1.00 indicates
underachievement, ie. the project is slipping.

Status Index

Monitor the status index. It is the ratio of the achievement index to the cost index. As a
rule of thumb, 0.9-1.1 is a normal range, 1.1-1.3 and 0.7-0.9 requires management
attention/action, >1.3 and <0.7 requires immediate management action (Gore and Stubbe
1988). Plot the cost, achievement and status indices on the same chart.

Change Orders

Monitor the number of change orders. A declining number shows an increase in quality
in the design process.

Benchmark Trendlines

Monitor the trendline in benchmarks. A decreasing trend shows an improvement in
quality. Note that by the time problems show up in trendlines. they may be fairly serious.

Break-even lime

Monitor the number of days to break-even on a given class of project.

CONCLUSIONS

We have presented a basic model for considering TQM. This should be regarded as a
framework of interrelated considerations. The reader may select, at her own discretion,
various elements or configurations of elements from among the various approaches and
techniques that were outlined, to develop an approach that suits her particular
organization.

However, in the last analysis any approach to TQM must be systematic and systemic.
Several of the techniques which are discussed above will yield incremental benefits if
introduced in isolation. These gains not only may be incremental but may also be
transitory.

Those contemplating a move to TQM clearly do need to think carefully about the
culture of their organizations with regard to innovation. The fullest benefits of TQM can
only accrue when there is an organizational strategic commitment to the paradigm.
However, much can be accomplished at the departmental level.

It takes time to successfuly implement TQM and entrench a philosophy of continuous
improvement. The need to improve competitiveness, and initiatives such as the
widespread adaptation of ISO 9000, will eventually force many organizations to make
rapid changes that may be very disruptive and counterproductive; it is better to plan
implementations now, to ensure success.

21C

MANAGING CBT DEVELOPMENT

211

The concepts underlying the approaches encountered in TQM are comfortable ones to
those in the field of instructional technology who have adopted a performance technology
orientation - which deals with performance related factors in general, including job
design, job support, technology, organization, procedures, as well as explicit training
solutions. Training departments and organizations are thus perhaps in a position to lead
the way in the introduction of TQM philosophy, principles and methods.

References

Albregtse R J, Hejka J A and McNeley P K (1991) Quality Management: How do you
do it? Automation August pp 30-32

Alonso R L and Frasier C W (1991) JIT Hits Home: a case study in rescuing
management delays Sloan Management Review Summer pp 59-67

Bellis-Jones R and Hand M (1989) Are Total Quality Management Programmes a Fact
or a Management Fad? Management Accounting May pp 36-37

Breuker J A and Wielenga B J (1987) Use of Models in Interpreting Verbal Data In A
Kidd (Ed.) Knowledge Acquisition for Expert Systems: a practical handbook NY
Plenum pp 17-44

Brown D H (1991) Product Information Management: the great orchestrator Computer-
Aided Engineering May pp 61-61, 66

Collins A (1990) Quality Control as a Model for Education: it would improve our output
Engineering Education May/June pp 470-471

Creese R C and Moore L T (1990) Cost Modelling for Concurrent Engineering Cost
Engineering June pp 23-27

Cronk R D (1992) Tributaries and Deltas Byte January pp 172

Doucet R M and Ranker R A (1990) SOCRATES, an Instructional Design Advisor
Program Computers in Human Behaviour 6 pp 201-206

Duchastel P C (1988) Models for AI in Education and Training Artificial Intelligence
Tools in Education pp 17-28 Amsterdam North-Holland

Duchastel P C (1990) Cognitive Design for Instructional Design Instructional Science
19 pp 437-444

Edosomwan J A and Savage-Moore W (1991) Assess Your Organization's TQM Posture
and Readiness to Successfully Compete for the Malcolm Baldridge Award Industrial
Engineering February pp 20-24

Gibbons A S and O'Neal A F (1989) New Work Surfaces for Instructional Systems
Design J of Interactive Instructional Development Spring pp 16-20

Gore M and Stubbe J W (1988) Elements of Systems Analysis Wm C Brown Publishers
College Division

220

212

QUALITY IN TEACHING AND TRAINING

Hauser J R and Clausing D (1988) The House of Quality Harvard Business Review
May-June pp 65-73

Higgens R and Messer G H (1990) Improving Instruction Using Statistical Process
Control Engineering Education May/June pp 456-470

IM Management Roundiable (1990) Concurrent Engineering, Global Competitiveness,
and Staying Alive: an industrial management roundtable IM July/August pp 6-10

Inglebsy T (1991) Time Waits for No Manufacturer) HP Manufacturing Systems June
pp 4,6,9,12,14,17,19,20

ITAC (1990) Putting the Pieces Together Information Technology Association of
Canada Toronto

Jonassen D H and Wilson B G (1990) Analyzing Automated Instructional Systems:
metaphors from related design professions In M R Simonson and C Hargrave (Eds.)
Proceedings of Selected Research Paper Presentations at the 1990 Annual Convention
of the Association for Educational Communications and Technology (pp 309-328) Ames
Iowa Instructional Resources Center College of Education Iowa State University

Keys D E (1991) Five Critical Barriers to Successful Implementation of JIT and Total
Quality Control Industrial Engineering January pp 22-24, 61

Lee C (1991) The Customer Within Training July pp 21-26

Li Z, O'Neal H F and Baker E L (1991) Developing a Research Reference Interface for
Knowledge-Based Instructional Design Tools Educational Technology 31(8) pp 7-16

Li Z and Merrill M D (1990) Transaction Shells: a new approach to courseware
authoring J of Research on Computing in Education 23(1) pp 72-86

Merrill M D (1988) Implementation of an Expert System for Instructional Design:
Phase 2 - design document and technical report (Contract No. OPM-87-9041)
Alexandria VA: Army Research Institute for the Behavioral and Social Sciences

Merrill M D and Li Z (1989) An Instructional Design Expert Journal of Computer-
Based Instruction 16(3) pp 95-101

Misselt A L (1989) Needed: a courseware design library CBT Directions 2 pp 16-22

Musselwhite W C (1990) Time-Based Innovation: the new competitive advantage
Training & Development Journal pp 53-56

Nandyal R K and Welch D (1991) Goals-Orientation, SPC and Incentives Quality June
pp 12-14

Pirolli P L and Grceno J G (1988) The Problem Space of Instructional Design In J
Psotka, L Massey and S Mutter (Eds.) Intelligent Tutoring Systems: lessons learned
(pp 181-201) Hillsdale NJ Laurence Erlbaum.

MANAGING CBT DEVELOPMENT 213

Pirolli P L and Greeno J G (1990) The Instructional Design Environment: technology
to support design problem solving Instructional Science 19 pp 121-144

Ranker R A and Doucet R M (1990) SOCRATES: A Computer-Based Lesson
Development Advisor Educational Technology 30(3) pp 46-50

Russell D M (1988) IDE: The Interpreter In J Psotka, L Massey and S Mutter (Eds.)
Intelligent Tutoring Systems; lessons learned (pp 323-349) Hillsdale NJ Laurence
Erlbaum

Russell D, Moran T and Jordan D (1988) The Instructional Design Environment In J
Psotka, L Massey and S Mutter (Eds.) Intelligent Tutoring Systems: lessons learned
(pp 203-228) Hillsdale NJ Laurence Erlbaum

Schonberger R J (1985) Operations Management - Productivity and Quality Business
Publications Inc Piano Texas

Shaw M L G and Gaines B R (1987) An Interactive Knowl'^e-Elicitation Technique
Using Personal Construct Technology In A Kidd (EdO Knowledge Acquisition for
Expert Systems: a practical handbook NY Plenum pp 109-136

Sheridan J K (1991) Racing Against Time Industry Week June 17 pp 22-28

Shina S G (1991) New Rules for World-Class Companies IEEE Spectrum July pp 23-26

St. Charles D (1990) Don't Toss it Over - Break Down the Walls Automation June
pp 68-69

Sutton J R (1991) New Cost Management Tools Offer Competitive Advantage
Industrial Engineering September pp 20-23

Tag.chi G and Clausing D (1990) Robust Quality Hansard Business Review Jan-Feb
pp 65-75

Tripp S D and Bicheimeyer B (1990) Rapid Prototyping: an alternative instructional
design strategy Educational Technology Research & Development 38(1) pp 31-44

Turino J (1991) Making It Work Calls for Input from Everyone IEEE Spectrum July
pp 30-32

Vesey J T (1991) The New Competitors: they think in terms of "speed-to-market u
Academy of Management Executive * 5(2) pp 23-33

Wielenga B J and Breuker J A (1986) Training of Knowledge Engineers Using a
Structured Methodology In T Bernold (Ed.) Expert Systems and Knowledge Engineering
Elsevier Science Publishers

Ziegfeld R, Hawkins R, Judd W and Mahany R (1985) Preparing for a Successful
Large-Scale Courseware Development Project In R Barrett (Ed.) Text, Context and
Hypertext The MIT Press Cambridge MA

222

31 Towards quality management in
training design

Meurig Williams and Graham Carr, Training Development Group, Lloyds Bank
pic, Hindhead, GU26 6QZ, UK.

INTRODUCTION

An integrated quality management policy is applied throughout the training development
cycle, using a consistent approach across all training media. This strategy is based on
all-round commitment to quality and on sensitively defined standards. Quality
management techniques and independent quality reviews and audits successfully
complement established validation techniques. The paper outlines the development of the
strategy, the skills and conditions necessary for its success, its benefits and its relationship
to international standards, and includes a case study of the use of quality management
tools in CBT courseware production.

Lloyds Bank Training Development Group (TDG) designs training for 35,000 staff
throughout the United Kingdom, using both traditional and technology-based delivery
methods. Quality management techniques, originally introduced in the development of
computer-based training (CBT) and owing something to software engineering, have been
successfully extended to other media.

The forerunner of TDG, Lloyds Bank Distance Learning Centre (Slade 1989) was
rapidly expanding the production of CBT packages and courses to meet the needs of the
Bank's Branch Information Technology (BIT) project, a major internal customer. Both
design and programming of many courses were being subcontracted to external suppliers.

Project teams were following the conventional systematic training design process. A
training requirements study led in turn to an outline design and detailed design. Then the
programmer received a script specifying screen layout, content, program branching,
course navigation, user interface and any other responses to student interaction. Staff
external to the project approved the requirements study, and tested the completed course
before its release, but there was no other independent review.

In pre-release testing, reviewers recorded observations on course design, content and
operation. An unacceptable proportion of defects (defined as negative observations
requiring re-programming) was occurring. These ranged from typographical and
grammatical errors to serious software bugs, and included factual errors and perceived
poor training design. They contributed substantially to project costs and delays in
completion.

ATTACKING THE PROBLEM

A member of staff (the first author) was appointed, and given the necessary independence
and authority, to tackle the quality problem. Technical problems that could only have
been introduced at the programming stage were the most urgent and important, so a
technical production guide (Lloyds Bank 1990) was developed to ensure that all
concerned were aware of relevant screen design conventions and equipment limitations,
and to facilitate communication between designers and programmers. Both internal staff
and external subcontt actors received copies.

In the meantime observations of defects were being systematically recorded, classified
and graded according to criticality, later using Pareto analysis. This allowed informed
decisions to be taken about which areas it was possible to standardise. Concurrently,

QUALITY IN TRAINING DESIGN

215

independent reviews were introduced on completion of critical intermediate stages in
courseware development, such as the outline design and the programming script. These
were intended to trap potential difficulties early enough to prevent their becoming
expensive problems.

Thus the classic quality management techniques of process observation, event recording
and design review were introduced.

DEVELOPING THE SOLUTION

As a means of specifying technical limitations the technical production guide was a
success, but in non -technical matters it did not reflect a consensus of design views and
so was not widely accepted. A parallel investigation of the difficulties of communicating
design information (England 1990) showed that in a fast-changing field like training
technology, individual projects might have to adopt their own conventions. It became
clear that a standards-based approach to quality was valid but would succeed only if
environmental influences were taken into account.

Time and cost, of development, observance and enforcement, compared with benefit,
help to determine whether or not a standard is issued. So does demand, whether from
designers, quality assurance (QA) or production staff. What any one standard covers is
determined by delivery medium, demand and enforceability, while the principal factors
affecting effectiveness are usability and acceptability.

As described above, the first effective standard, the technical production guide, was
developed empirically. A pre-release review of a number of CBT courses revealed
defects. Classification enabled attention to be given to critical areas and an ideal situation
to be determined. Hence a standard for the critical areas could be defined and guidelines
issued. The next step was assessment of subsequent projects against the standard.

Currently, when a new standard is required, an example of the item (process or
product) concerned undergoes an initial assessment, by means of inspection, to determine
what constitutes a defect. All observations are recorded. The resulting Pareto analysis
leads first to revealing what needs standardising, and then to discussions with interested
parties and the definition of a standard. Guidelines for new designers are then produced,
to help them understand how the standard is intended to be interpreted and in what
circumstances exceptions may be advisable. This cycle is repeated as many times as
necessary.

The actions described - inspection, recording, analysis, addressing standards, producing
guidelines, and assessment, re-inspection or validation - resemble the traditional
courseware development cycle and also the classic quality control loop (Figure 31.1), with
the assessor or tester being the sensor and the reference being the standard.

The review at each stage of course development is a logical part of the training
validation plan which is built in lo each project. It often uses familiar techniques like
course piloting with designers* peers or with the target population. Assessment against
a standard may be eased by using checklists and, sometimes, testing plans. It is
nevertheless a task requiring a variety of well-developed skills and judgment in the
assessor, depending on the medium used for the training. In most areas of training design
there will be borderline cases, where technical infringements do not affect training
outcomes or would cost a great deal to correct. However, each assessment should produce
a record of findings, a correction agenda and a possible improvement action programme.

*• *w -4.

216

QUALITY IN TEACHING AND TRAINING

Actuator

Sensor
(tester)

input

Process

Reference
(standard)

4
I
I
I

Comparator

Measurement j

Figure 31.1 Vie classic control loop
THE ROAD TO SUCCESS

A combination of independent review and acceptably defined standards and
well-presented guidelines thus form the foundation of a workable quality management
system. Its success requires acceptability, independent assessment and a means of
completing the improvement action loop.

Acceptability is assisted by involving project teams closely in the review process and
in the definition of standards, and by ensuring that the reviewers are appropriately skilled.
The standards themselves must be easy to understand and apply.

Independence is provided through the existence of a quality assurance team whose
members, although Lloyds Bank TDG staff, are not members of project teams.

The improvement action loop is addressed by feedback meetings open to all design and
production staff. These address specific topics and plan subsequent action, such as a
project for writing guidelines.

QUALITY IN ALL TRAINING MEDIA

In 1991 Training Development Group extended its quality effort to media other than
CBT. The techniques described above are now applied to all training design. Guidelines
for writers and editors have been produced, and in early 1992 observation data was being
gathered from classroom sessions in order to analyse the delivery of face-to-face training.

In the process of extending the system to other media QA staff acquired new skills,
and began to be consultants to the four design teams, rather than inspectors. They need
skills in writing, editing, proof reading, software testing and interface design, and
observation, as well as in quality management. Coaching and giving feedback is important
in the review process and good inter-personal skills are vital, as is a full knowledge of
the training design and delivery process.

QUALITY IN TRAINING DESIGN

217

BENEFITS

Directly quantifiable benefits are discussed in the case study. Among the indirect benefits,
the approach is extensible as required, uses many existing skills, can be incorporated into
initial project costings and so can be closely monitored, is widely applicable and is
sensitive to designers' feelings. External suppliers can easily be involved, without
conflicting with their own quality systems. Quality audits can be incorporated, and vast
amounts of paperwork are not generated. The visibility of the system is an encouragement
to TDG's suppliers and customers alike.

Unlike the approach adopted in BS5750 (BSI 1987) TDG's approach is not wholly
inspection-based and can be implemented fairly gradually in particular areas. Unlike the
British Standard also, it is results-based rather than purely activity-based (Schaffer and
Thomson 1992) as its primary focus is on improvement action, not fidelity to procedures
alone.

CASE STUDY: USING QUALITY MANAGEMENT TOOLS IN
CBT COURSEWARE PRODUCTION

Extensive data from the production and testing phase of CBT courseware development
has been collated and analysed. The major source of quality problems have been
identified by using simple statistical techniques. Designers and programmers have thus
been enabled to improve quality by taking particular care in critical areas. The case study
illustrates the benefits and problems arising from the previously largely untried use of
quality management tools in developing CBT courseware.

The quality improvement process adopted during CBT courseware development consists
of four distinct stages: identification, analysis, improvement via communication, and
monitoring.

Identification

The identification stage can be the part most organisations find difficult to accept. The
fact tha. the identification process is considered necessary indicates there are problems
which need addressing, and there is thus room for improvement. As a corollary, this
suggests that both processes and people will need to change which can be cause for
significant resistance.

Accepting the sensitive position the quality assurance (QA) team are in, it is realised
that it is not always "what you say" but "how you say it" that counts, and much effort
is expended in this type of approach. For instance specific training on coaching and
feedback techniques considerably assist this role. In TDG, designers and programmers
want to produce better training material and are therefore willing to consider and
implement improvement ideas. It is for these reasons QA can feel at ease in offering
genuine remedies to common problems and remain confident in the fact they will be
welcomed and implemented.

Many inconsistencies (observations) occur throughout CBT production, and the basic
method of recording these is on a QA report form. The columns show the observation
number (sequential), where in the course the observation occurred, what it was, and
recommendations (if any). The "importance" column is described in detail later. The
remaining four columns show the process of correction until final recheck by QA. The
forms are not for the sole use of QA members as the designer or other interested parties

218

QUALITY IN TEACHING AND TRAINING

are actively encouraged to use them.

Each course will generate a number of these forms and therefore a significant number
of observations. Following a 1t brainstorming H session the lengthy "classification of
observation" list shown below was produced.

A Key Functionality (D)

B Key Functionality (P)

C Help (D)

D Help(P)

E Interactions ie questions (D)

F Interactions ie questions (P)

G Graphics (D)

H Graphics (P)

I Application Simulation (D)

J Application Simulation (P)

K Spelling and Punctuation

L Technical Accuracy

M Tutorial Text (D)

N Tutorial Text (P)

O Mouse

P Observation Readvised

These classifications are intended to cater for all possible inconsistencies likely to be
encountered during procl iction of a CBT course and, for clarity, have been assigned
individual letters. The (D) and (P) refer to designer and programmer respectively.

Analysis

The analysis stage uses numerous quality management techniques to ascertain the extent
(ie frequency) of the observations being made throughout course production, and identify
major areas of concern.

A simple "tally chart" was devised to collate and classify the observations from three
CBT courses, one internally designed and internally programmed (course one, released
in 1989), one externally designed and internally programmed (course two, 1991), and one
internally designed and externally programmed (course three, almost ready for release in
April 1992). This revealed remarkable consistency in the distribution of different types
of deficiency.

From the sample of observations from course one, the classification H M" ie Tutorial
Text-Design provided the most frequent comments. This is where the programmer ha3
correctly followed the designer's script but for example, a sentence or paragraph does not
read correctly, and the observation is consequently highlighted and attributed to the text
design. The next highest was the "K" classification (Spelling and Punctuation) with "B"
and "F" classifications (Key Functionality - Programming and Interaction (Questions) -
Programming respectively) also featuring highly. The latter two classifications refer to
the usability of the keys lor navigation throughout the course, and although the design
script can sometimes provide incorrect instructions for the programmer, it is more often
found that the observation relates to the programming related deficiency.

The second course included a "simulation" of the application as a method of testing the
learning. This shows similar priority to course one in terms of Tutorial Text - Design but
Key Functionality - Programming came second pushing Spelling and Punctuation into

QUALITY IN TRAINING DESIGN

219

third position. Understandably the design and programming of the "Application
Simulation" also feature highly. Another interesting point is the appearance of Tutorial
Text - Programming ("N") in fourth spot, as this indicates inability to reproduce the text
on screen from the designer's script.

Course three, designed and produced most recently, again shows similarities to the other
two courses. The spelling and punctuation has in fact reduced in significance but the
appearance of the programming of graphics ("H") is now a feature. Being the first
"Mouse" controlled course to be monitored the appearance of comments under
classification "O" was, to a certain extent, anticipated The latest classification added to
the list (ie "P" Observation Readvised) was a direct result of analysing the QA report
forms and noticing the recurrence of previous comments.

Understandably an abundance of information has been produced, which renders
concentrating on many areas at once, totally impracticable. We therefore prioritised the
observations in terms of frequency, which allows comparisons across individual courses.
This was achieved with the aid of Pareto analysis,

The overall results from course three confirm the findings in terms of priorities
mentioned above (ie M,B,N,H,0,K). We were also able to focus on the observations by
importance as touched on previously in the "QA report form". Each observation is
categorised in terms of importance as follows:

Category 1 Critical Defect

Category 2 Major Defect

Category 3 Minor Defect

rendering release of course highly unlikely ie execution
error (software failure) where the course has "crashed"
and needs reprogramming, or where some text has been
incorrectly keyed which could seriously inhibit the
transfer of learning.

making release unlikely but not impossible, ie key
malfunction on one screen but fully functional
elsewhere, or where a graphic does not hinder the
learning but could be improved on, or difficulty in
understanding how to proceed,
where release is still likely if insufficient time to
correct, ie spelling or punctuation error.

Each level of observation is likely to be corrected before release, but this level of
importance prioritises the work involved.

The analysis of observations by category assist the QA team in identifying where to
focus attention. By analysing Category 1 observations only, we found the top three
classifications all focus on programming inaccuracies.

Many more of these charts have been produced which gives us clear indications where
to concentrate our initial efforts in order to make the most impact. Another technique
being investigated is that of the "cusum" chart. This refers to the cumulative sum (hence
"cusum") of observations found in a sample number of screens. In essence a reference
or "target value" is subtracted from each successive sample observation and the results
cumulated. Values of this cumulative sum are plotted and "trend lines" are drawn on the
resulting graph. If this is approximately horizontal, the value of the variable is about the
same as the expected value. An overall slope downwards shows a value less than
expected, and if the slope is upwards it is greater.

Another technique is that of the "cause and effect diagram" (Ishikawa) as shown in Fig.
31.2, which provides an overall view of the processes that could lead towards a
nonconformity report.

220

QUALITY IN TEACHING AND TRAINING

Failure to
follow
design

Text design
Outline \ Spelling
design \ /

Inadequate-
pilot

Inadequate
detailed —
design

Design

USABILITY /SOFTWARE

Programming ^ of c,arit V

SIMULATION \ LEARNING

ERROR \ PROBLEM

Testing Programming

NONCONFORMANCE
REPORT

Failure
to

observe
standard

poor oversight

Figure 31.2 Cause and effect diagram (Ishikawa diagram)

Improvement Via Communication

As mentioned, it is not possible to improve all the problems at once, so we have tended
to concentrate effort on a selection of the more important and frequent observations. The
improvements currently being worked on or already achieved include:

Tutorial Text - Design ("M")

Improved reviews of designer's script. Many of the design observations should be
picked up at the design review stage, as should Technical Accuracy C'L"). This would
dramatically reduce the classification of observation list so that (ideally) only pure
programming deficiencies remained.

Many reviewers find difficulty in visualising the paper version of text on a screen
resulting in more comments when the material is later reviewed on the screen. This
is being investigated by producing the pure script on screen but without the usual key
functions so as to restrict comments to content, screen design and layout only.

Key Functionality - Programming CB")

If the above were successful, much more could be achieved in this area by further
developing tools such as "Fault Tree Analysis" (FTA*s) which would help to pinpoint
exactly why a programmer was allowing inaccuracies. It could be inexperience, lack
of time, insufficient testing, etc.

QUALITY IN TRAINING DESIGN

221

The creation of a "shell" for navigation purposes (ie "Exit" key "Help" key "Next"
key etc) will standardise the process, and significantly reduce this classification
without stifling design.

Many courses are now becoming "Windows" based, and the use of the recorder
function in "Windows" allows selected key presses to be automatically remembered
and reproduced on demo; A . This encourages a more efficient testing process to be
maintained, thereby increai 'ng the speed of course release.

Spelling and Punctuation ("K")

This has been improved in recent courses by using a programming language that is
able to import the designer's script thereby cutting the keying-in process by half (and
consequently virtually eliminating Tutorial Text - Programming ("N") classification
in the process).

These improvements are only likely through effective communication and a willingness
to improve. It is up to QA to identify, analyse, encourage, support and coach so that both
the training provided and the method of producing that training are, and continue to be,
of the highest quality.

Monitoring

We need to continue the improvement process by constant monitoring. As technology
advances so do the problems. The difference in complexity of, for example, course one
and course three is enormous. This is proven by comparing the type and frequency of
classification of each course. It is true to say that as soon as you have agreed a standard
(or preferably - a guideline) so the parameters change. It is for these reasons a monitoring
process can, in itself, present difficulties. However we are able to compare the number
of observations per sample screens, and also indicate whether we have been successful
in shifting the percentage of observations from importance (category) one to two and
from two to three, to show improvement. This information is regularly communicated to
other TDG members whereupon the Identification stage will restart.

References

BSI (1987) British Standard 5750: Quality systems BSI London

England E (1990) Streamlining Computer Based Training Production: improving
communication between commissioners and contractors Aspects of Educational and
7 raining Technology XXIV

Lloyds Bank (1990) Technical Production Guide Distance Learning Centre (internal
document) Lloyds Bank Hindhead

Schaffer R H and Thomson H A (1992) Successful Change Programs Begin with Results
Harvard Business Review Jan/Feb pp 80-89

Slade A (1989) How Lloyds Bank Utilises Distance Learning Training Technology
July/August

23Q

32 Achieving quality in networked
interactive video

Peter Willis and John Eary, Scottish Police College, Kincardine-on-Forth, FK8
4BE and National Computing Centre, Manchester, Ml 7ED, UK

INTRODUCTION

The undoubted growth of computer based training and interactive video in the past ten
years has been almost exclusively concerned with individualised learning. By contrast,
classroom based training methods during this period have increasingly used group centred
and team approaches to promote more active participation. The National Computing
Centre (NCC) has developed an interactive group training system (VISTRAIN) with the
Scottish Police College to help train police super/isory officers in crowd control at major
spectator events, such as football matches.

VISTRAIN is a multimedia, multi-user network training system using video based
scenarios under the control of a facilitator. The system is designed to test decision
making and communications skills in time-critical situations, A multimedia approach was
chosen because of the difficulty of mounting realistic training exercises. The trainees will
be able to act out their roles in a multimedia simulation. Video material shot at major
football matches has been extensively used to give an authentic feel to the decisions
taken.

It is believed to be the first known application of networked interactive video, within
a co-ordinated group learning environment. This approach involves technology enhanced
training in a group learning environment, and is expected to have a significant impact on
management training methods since it will enable new levels of sophistication and realism
to be incorporated into exercises to practice decision making and communication skills
under pressure.

BACKGROUND

The control of crowds at major spectator evenX such as football events involving around
40,000 spectators, requires a high level of decision making and communication skills.
The incidence of violent disturbances at football matches in many European cities is
unfortunately a common occurrence. Police forces are being faced with increasing
problems in planning and policing major football matches and it is recognised that senior
officers need to be trained to high standards.

Effective training in crowd control and similar situations requires the intelligent
application of procedures in realistic circumstances. Football match commanders,
responsible for the safety of many thousands of people, are required to make many rapid
decisions and communicate clear instructions in an often "highly charged" atmosphere
where noise levels are extremely high, making good communication very difficult.

Little formal training in crowd control has previously existed because of the difficulty
of mounting realistic practical exercises which would simulate conditions experienced at
a real football match.

One approach that has been tried is paper based exercises where teams of officers are
given notes describing a situation and asked how they would respond. These exercises
have been observed by the authors. They lack realism and do not put sufficient pressure
on the participants to prepare them for actual and often dangerous events.

NETWORKED INTERACTIVE VIDEO

223

THE PROJECT

The Scottish Police College, in seeking a means of developing decision making and
communication skills which, in addition to testing these skills, would also simulate the
atmosphere and conditions experienced at a football match consulted the National
Computing Centre (NCC). The consultation with NCC resulted in the joint submission
of a competitive bid for funding to the UK Government's Employment Department,
which was successfully accepted as part of the Technology in Learning programme. An
award in the region of £100,000 was made, and the project to plan and develop the
exercise commenced on 1 December 1989.

The Scottish Police College is the central police training establishment, serving all eight
police forces in Scotland, representing some 14,500 officers, and all levels of training are
provided. The college has considerable experience in the use of computer based training
and interactive video, particularly when this is applied to learning in groups.

The Scottish Police College role in the project ha'j been to provide the subject matter
expertise and the supply and production of video material. The college has also
conducted formal evaluation trials of the system as it is used as part of senior training
courses at the college.

The National Computing Centre is a membership organisation run on a commercial
basis. Profits from products and services are invested in the future of effective
information technology for users. NCC Training has developed innovative CBT and
interactive video courseware for over seven years.

NCC Training has provided the project management, instructional design, software
development and integration of the system. The two organisations have worked closely
together sharing expertise.

The system is used in a specially prepared exercise which forms part of senior police
training courses. Sixteen senior officers participate. The officers attend a briefing at
which the layout of the stadium and match details are given. They are split into four
groups of four officers taking one of four senior police officer roles: match commander;
officer in-charge inside the ground; officer in-charge outside the ground; divisional
control.

On the screens of their team workstations they see video scenes that relate to their role.
They have only a partial view of the situation, eg the group taking on the role of officer
in-charge inside the ground can only see views inside the ground and cannot see what is
happening outside the ground. The network is used to synchronise the video scenes.

The teams are required to respond to various "cues", ic potentially serious incidents to
which they should respond. These incidents include moving video, radio and telephone
messages. The teams are required to discuss the actions they should take in response to
these incidents and relay these decisions via a personal radio to the appropriate teams,
whose "view" changes accordingly. Other teams in turn must respond to the messages
that are received and decide upon their actions and so on.

A member of the college staff acts as facilitator to keep overall control and provide
advice. The facilitator is able to monitor the teams* progress by listening in on the
personal radio channels. He or she can also observe what the teams arc viewing on their
video display on a bank of four duplicate monitors.

The facilitator is required to z^sess the teams* performance during the exercise. The
facilitator is provided with an on-screen storyboard indicating the appropriate actions

THE SYSTEM IN USE

224

QUALITY IN TEACHING AND TRAINING

that each team should take in response to the events. By the use of a mouse and
especially designed puii down windows, the facilitator is able to quickly record the
actions taken by the teams.

MATCH
COMMANDER

OFFICER-IN-CHARGE
OUTSIDE THE GROUND

OFFICE R-IN-CHARGE
INSIDE THE GROUND

l ^i =3

Figure 32.1 The network

Technology is also employed to assist the participating teams. The teams* workstations
make use of a WIMP environment to offer support to the teams. Using a mouse the team

NETWORKED INTERACTIVE VIDEO

225

can select context sensitive help. There is also a browse facility so that the team can
recall text and pictorial information from the briefing to assist them in their decision
making.

The exercise is divided into four scenarios, before kick off, first half of the match,
second half and dispersal after the match. There are some eight or nine events in each
scenario which can be selected by the facilitator from the workstation. Examples of these
events include the selling of forged tickets, illegal entry and the explosion of a tear gas
canister.

After the first scenario has been played a debrief session is held. The system assists
the facilitator to select those events where wrong decisions have been taken or have been
communicated incorrectly.

During the debrief session, which involves all sixteen officers, the selected events are
constructed by combining the video cues together with the teams' radio messages. These
have been digitally recorded onto a personal computer incorporated into the system.
During the debrief participants are able to review the actions taken and identify
alternative actions and their likely result.

The remaining scenarios are played in a similar way, but the teams change roles. Thus
by the end of the exercise they have had the opportunity to experience all four roles.

A formal evaluation of the system was carried out over a 9 month period at the College.
The resulting report has demonstrated that the delivery of a high quality learning
experience has been achieved. Those who have had previous experience of policing
football matches have indicated that the degree of realism achieved was outstanding.
Those without such experience have developed a real insight into the many pressures and
problems facing a match commander and feel much better equipped to cope with such
a responsibility. All indicated that there was no comparison between this exercise and a
"paper feed" exercise and all found the technology easy to use.

The system that has been developed for the project is called VISTRAIN (Video-based
Integrated System for Training Applications). VISTRAIN is a multimedia, multi-user
network training system using video based scenarios under the control of a facilitator.
An essential feature of VISTRAIN is that it allows users to participate in training
exercises where each has only a partial view of the total situation upon which decisions
must be taken.
The VISTRAIN system has three main components:

• a network of interactive video workstations linked to a facilitator workstation

• a two channel dedicated intercom network

• a software "shell" which, in addition to synchronising the video displayed,
enables the facilitator to control the triggering of various training incidents; mark
and record the performance of the participants; and reconstruct the events and the
participants responses during the debrief session.

The equipment used for the system comprises four Commodore Amiga 2000 computers,
fitted with Commodore genlocks and attached to Sony 3300 laservision videodisk players.
The facilitator's workstation is based on an Amiga 3000. These workstations are linked
in a star network using ASAP's Amiox network boards. A PC compatible Commodore
Bridgeboard computer is fitted to one of the Amiga 2000 computers to digitally record
the personal radio messages and replay these at the debrief.

EQUIPMENT USED

226

QUALITY IN TEACHING AND TRAINING

QUALITY APPROACH IN THE DESIGN OF THE SYSTEM

The design of the system followed NCC Training's standard methodology to ensure high
quality was achieved in the design of technology based training courseware:

• agree training objectives

• determine training "events"

• produce an outline design

• identify suitable video and textual material

• conduct a technical feasibility study

• produce a functional specification of the system.

An early meeting was held at the Scottish Police College to formulate training objectives.
The meeting was attended by a number of college staff, including the Commandant and
Deputy Commandant. Two senior Scottish police officers responsible for crowd control
at major football matches also contributed. South Yorkshire Police were represented by
an officer with considerable experience of crowd control.

Some thirty detailed objectives were identified. They related to three major objectives -
that during the exercise participants will demonstrate the ability to:

• communicate adequately and properly with all parties involved

• take appropriate action within areas of responsibility

• apply appropriately their knowledge of relevant legislation and regulations.

The more detailed enabling objectives were further refined and incorporated in the outline
design documentation.

During the discussion of the training objectives a number of typical incidents were
identified which would provide suitable training opportunities. The incidents were
collated to form a series of training events. It was decided to categorise these into four
main scenarios:

• pre-match

• first half of the match

• second half

• post match.

The initial plan was to allow fifteen minutes for the playing of each scenario. This was
found insufficient given the need for the large number of interactions between the
participating teams.

The general background was established against which these events would take place,
eg an all-ticket football match, capacity crowd, known history of animosity between
supporters etc.

An important decision taken at this stage was that the ground would be fictitious
composed of elements of actual grounds. It was felt to base the scenarios on a specific
ground would risk alienating officers from other forces who may not feel the training
events were relevant to their own situation.

TRAINING OBJECTIVES

TRAINING EVENTS

ERIC

NETWORKED INTERACTIVE VIDEO 227

OUTLINE DESIGN

The training objectives and training events were incorporated in an outline design
document. This document also included the agreed target audiences, a primary audience
of police officers, and secondary audience drawn from the other emergency services and
staff from the football clubs and authorities. The roles of the participant were identified.
To accommodate the sixteen attendees of a typical course, it was decided to allocate four
officers to each of four roles:

♦ the match commander, the officer in overall command in the control room

♦ the officer in charge inside the stadium

♦ the officer in charge outside the ground

♦ divisional control (responsible for officers beyond the immediate vicinity of the
ground).

Each team was to be shown video sequences and receive radio messages. The teams
were given a limited time in which to discuss a situation and determine the best course
of action. Teams were allowed to have the ability to call up information presented at the
briefing for the match and pictorial information including street maps and layout of the
stadium. Context sensitive help facilities would also be available to assist the teams in
their decision making.

The role of the facilitator was also defined. The facilitator would control the exercise
by triggering the training events and could monitor the exercise by listening in to the
radio messages sent by the teams and keep track of the resources deployed.

The facilitator needed to log the actions taken and mark the performance of the teams.
The main categories identified were:

♦ correct action taken and decision clearly communicated

♦ correct action taken but inadequately communicated

♦ incorrect action taken

♦ no action taken where action was required.

FUNCTIONAL SPECIFICATION

A functional specification was prepared, based on the outline design document and
consideration of how the system would operate. The specification covered the following
aspects:

♦ video source

♦ team workstations

♦ central facilitator's workstation

♦ verbal communication

♦ data networks

♦ video and audio distribution

♦ help facilities.

This document was to form the basis for an investigation of the technical opions and
therefore focused on the essential aspects of the system.

The functions of the team workstations were described and it was decided that the
remaining decision time would be displayed to add pressure on the decision making.

|ERjC

228

QUALITY IN TEACHING AND TRAINING

Alternative ways for the teams to communicate were considered. Typing was regarded
as an unnatural way for formulating verbal commands and the participants could not be
assumed to have keyboard skills. A displayed menu of possible actions would limit the
number of options available to the teams. It was therefore decided to simulate a personal
radio system. NCC investigated the use of a proprietary intercom system which utilised
the 240 volt main as the carrier signal. However this could not simulate a two channel
radio network.

The various tasks required of the workstation to support the facilitator were identified
for both the playing of scenarios and in the debrief. It was decided that a time log would
be used for recording the radio messages so that they could be played back in sequence
with the video sequences interspersed.

The initial design was based on a storyboard showing the preferred actions by each
team in response to a particular event. The project team was fortunate to receive
excellent co-operation f rom two experienced officers responsible for policing major crowd
control events. The first scenario was developed as a prototype in which various
conventions were developed. These included:

♦ an event number and time;

♦ the cues that would trigger decisions;

♦ the actual text of pre-recorded personal radio messages or descriptions of video
cue;

♦ the expected actions of the four teams.

A significant decision was to determine the "model answers" rather than all possible
options as this was likely to be confusing to the facilitator. A specially designed
storyboard was devised to show the interactions of the four teams.

MANAGEMENT TRAINING APPLICATIONS

VISTRAIN addresses a training requirement common to all police forces. It has already
attracted attention from police forces throughout the UK, Europe and the USA. The
system has now been successfully used on twelve courses and has won three awards
including the European Technology Training Award.

Although designed to help train police officers in crowd control at major spectator
events, the approach used to produce VISTRAIN has significant potential applications to
other areas. VISTRAIN is a unique software framework onto which any scenario could
be scripted.

The approach is applicable where training is required in decision making and
communications commonly required in management training. In particular, where one
or more of the following training factors apply:

♦ situations where decision makers have only a partial view of the total situation;

♦ role playing is required;

♦ training takes place in groups;

♦ facilitation is preferred to instruction;

♦ communication skills need to be practiced;

♦ training needs to have a high level of realism.

The significance of the VISTRAIN approach is that it takes the sophisticated technology
of multimedia from being training for the individual to being applied to the equally
important development in participative teamwork training.

237

Section 6: Quality in Technology

This final section is concerned with quality as it impinges on the technology that we use
in education. We begin with David Hawkridge's fundamental and controversial appraisal
of the state and place of educational technology as we approach the end of the twentieth
century - are we still locked into the behaviourism that originated much earlier in this
century? How can we assure "Quality in modernising educational technology"?

Perhaps some evidence for development (or lack of it) might be found by a careful
inspection of the study of current trends undertaken and reported by Donald Ely in
"Trends and issues in educational technology".

In "Training quality and new technology", Antonio Bartolome describes current
initiatives being taken in Spain to discover the levels of utilisation of media in the
classroom and the training initiatives being implemented to improve the quality of use.
An application of CAI methods and technology to the laboratory teaching of electronics
in Japan is addressed by Yoshikazu Araki in his consideration of "Connecting lectures
and laboratory learning with CAI".

With "Graphical routes to quality courseware" we move into the realms of multimedia
and hypermedia, where Charles Lamont and Philip Barker illustrate techniques that can
help us to produce quality courseware for CBT applications both more efficiently and
more effectively. Finally, in "Enhancing the quality of student- centred mathematics",
David Bowers and Rod Burrell look at applications specifically in the area of
mathematics workshops and describe their use of software to set up meaningful dialogue
between computer and student.

23S

33 Quality in modernising educational
technology

David Hawkridge, The Open University, Milton Keynes, MK7 6AA, UK
INTRODUCTION

Educational technology purports to be modem, even futuristic, yet uses 1960s
psychological theory as its basis. Is the perceived quality of educational technologists '
performance and products being adversely affected by this cracked and obsolete
foundation? How can educational technology courses be successfully modernised in the
1990s? Can we borrow from cognitive science? Are we integrating information
technology? Can we rise to the political and moral challenges?

My thesis is that it is time to modernise educational technology. This is not an easy
task. I shall first examine the 1960s theory, drawn mainly from behavioural psychology,
now discredited, and the thinking of the audio- visual movement. Familiar tools such as
behavioural objectives are blunt, but continue to be used. Many UK educational
technology courses, usually at the postgraduate level, follow the example of similar
courses in the US by teaching behavioural psychology and the systems approach. I shall
speculate on the extent to which we can borrow ideas from cognitive science instead,
bearing in mind problems being experienced in that discipline. I shall look at how well
we are integrating information technology into our theory. Finally, I shall identify
political and moral challenges being flung at educational technologists in general.

A CRACKED AND OBSOLETE FOUNDATION

The term "educational technology" came into general use in the late 1960s, after the
heyday of programmed learning. Ely (1968) had two definitions: technology (hardware)
use in education, and the application of behavioural science to education. If we include
the period in which programmed learning was being developed, it seems as though the
dominant paradigm has been the same for about forty years. To put it another way, it
is now a cracked and obsolete foundation.

It is time to modernise educational technology. This is not an easy task. We probably
all agree with Gagne (1987) that educational technology involves the systematic
application of knowledge derived from scientific research. It is a fact t however t that the
majority of educational technologists have clung to behavioural psychology and continue
to advocate the use of familiar tools such as behavioural objectives. Many UK
educational technology courses teach behavioural psychology and the systems approach.
The same psychology underlies much of the thinking about performance-based models
of education and the definition of competencies. Behind the jargon of this conference are
hidden assumptions about the nature of learning and what constitutes success. My own
opinion is that we are in danger of saying to the world of education "never mind the
quality of educational technology, just feel the width".

Consider the books for 1980s courses about educational technology in British
universities, polytechnics and colleges. Romiszowski (1981) built entirely on ideas of the
1970s, derived from North American behaviourism and the systems approach. As Eraut
(1989) commented, these ideas do not constitute established knowledge, though they may
be useful to us, nor do they take account of constructivist views of learning. Percival and
Ellington (1984, revised in 1988) took a similar line, presenting a handbook long on

232

QUALITY IN TECHNOLOGY

techniques, short on theory. Rowntree (1974, revised in 1982), widely used in teacher
education, was anything but theoretical perhaps because he preferred to appeal to readers'
commonsense, not their knowledge of philosophy or psychology. Spencer's (1988) book
on the psychological roots of educational technology depended greatly on behaviourism.
He showed how Thomdike and Pavlov laid the foundations for a science of learning, but
gave Watson and Skinner, both eminent behavioural psychologists, most of the credit.
He also drew particularly on Bloom, Briggs, Gagne, Keller, Mager, Popham and Tyler,
all Americans in the behaviourist mould, and he was right to do so, even if it underlined
how much British educational technology had been influenced by them. Spencer did go
further than this to consider the work of Piaget, Bruner and Vygotsky, all of them
interested in how children internalise what they learn as cognitive structures or models.
Finally, he referred to Papert and Salomon's research, which may yet link educational
technology to cognitive science.

A couple of years ago, Romiszowski (1990) summarised the position. The educational
technology movement had been criticised, he said, for insisting on too specific,
product-based, uniform objectives; for placing too much emphasis on prior design and
development of materials, followed by dissemination of the standard messages to all
learners indiscriminately: for emphasising behaviours mastered rather than ideas
processed and correction of errors rather than reflection on the implications of viewpoints;
and for h Jng shallow and superficial rather than encouraging the processing of complex
multi -faceted content.

Our neglect of cognitive science may be due to its rhetorical flavour. Behaviourists
stick closely to the analysis of behaviour, and do their best, even in the case of humans,
to develop their discipline along strictly scientific, positivist lines. Cognitive scientists
have much less empirical data available on which to base their theories. Like certain
nineteenth century psychologists, they are attempting to analyse what happens inside
human beings' brains, not outside. They derive their ideas from linguistics, artificial
intelligence and psychology, but as yet cognitive scientists can tell educational
technologists very little about how to guide learning according to the principles of their
science.

The search is on for a science of learning, a set of normative rules that, if followed, will
lead to successful teaching. After four decades of effort by cognitive scientists, however,
it is clear that these ideas have been slow to reach children and their teachers, even with
the advent of microcomputers. There is a large gap between the research laboratories and
classrooms. For example artificial intelligence specialists admit that some of their work
has been misguided, and that they have explored blind alleys. Their field data are hard
to interpret for practical purposes.

Intelligent tutoring systems, which act like a good tutor, represent the extreme example
of cognitive science applied to education. Analysis of possible errors lies at the heart of
the development of intelligent tutoring systems, yet, as the behaviourists found, human
learning is extraordinarily complex and predictions are very difficult. Tom (1984) does
suggest that even if there are no breakthroughs, cognitive scientists may have an impact
on teaching by encouraging teachers to regard their pupils more as thinkers and less as
bch avers.

It is clear from many of the papers at this conference that educational technologists are
making great use of information technology. That is not surprising. As technophiles
rather than technophobes, we welcome the technology. In doing so, we are continuing

INTEGRATING INFORMATION TECHNOLOGY

MODERNISING EDUCATIONAL TECHNOLOGY

233

the line of thought we inherited from the audio visual movement that started in the 1920s.
The challenge to us, however, is how will the field of educational technology be changed,
to incorporate information technology where desirable, and to exploit its special
characteristics for the sake of education? At the moment, we are assailed by
technology-led solutions to educational problems. That is not what educational
technologists stand for: we believe in student-led solutions in which information
technology plays an appropriate role. It is a pity that in the United States and to some
extent in this country too, the introduction of information technology into education has
simply enabled educational technologists to apply the "failed" behavioural psychology all
over again. Perhaps this was inevitable, because many technological innovations are used
at first to do faster what was done before, rather than to change the nature of what is
done. I need only mention the drill-and-practice programs that flooded the US education
market in the late 1980s and you will see what I mean.

Of course, British schools are now using computers for many other purposes besides
drill-and-practice. Some educational technologists now advocate the use of Logo, which
is based on construct vist psychology. Many others expect children to learn how to use
databases, word processing programs and spreadsheets, if not graphics and other software.
In doing so, however, these latter educational technologists appear to be atheoretical,
advocating practical courses of action without theoretical justification. We should also
consider at least a few of the critics of information technology's ideology in education.
For example, Chandler (1989, 1990) argued that computers project an ideology that
conflicts with the purposes of the educators who put them into schools. The nature of
information technology is such that we cannot avoid its influence on teaching and
learning. He gave three examples: data-handling, simulation and word- processing. Let me
take the first of these, in particular.

Data-bases are introduced into schools to teach children data-handling: how to record,
sort, search for and display data. As Chandler said, these students learn
"information-processing", using information technology, despite the fact that computers
only contain data, not information. Data are changed into information by humans, not by
computers. Computers deny the human origin of information. Humans negotiate meanings
through discourse. Chandler said this is not merely a semantic argument: the language of
computers threatens to redefine the world in its own terms.

Chandler also pointed out that because humans give the computer power and authority,
data stored elsewhere appear to lose some of their significance. Data-handling systems
also distort information, diminishing the value of things they cannot record, including
much everyday knowledge. Creators of databases have to be willing to tailor their
intentions. Searchers have to limit the questions they ask.

Chandler was worried that using computerised databases may lead students to believe
that thinking is data processing. He asserted that thought and memory require building
and rebuilding of models of the world, not through passive data capture but through
"interpretation and elaboration of information according to changing hypotheses ... We
create ideas: computers can't". Databases disregard meaning.

Computers promote the notion of their own objectivity, said Chandler. Their users often
ascribe more authority to databases than to the printed word. Because databases contain
standardised data, without details of sources, they seem misleadingly objective: each item
carries equal weight, equal certainty. The creators' biases are invisible, the items are
stripped of their origins and context. Data-processing requires fragmentation of the whole,
with loss of meaning, declared Chandler. It prohibits a holistic perspective.

Saettler (1990) suggested that there was a hidden agenda in incorporating information
technology into education. He said that improved managerial efficiency was the goal:
more students taught in less time by fewer teachers using less space and at lower cost.

24:

234

QUALITY IN TECHNOLOGY

This agenda is no longer hidden. Politicians and educators in the UK and US, if nowhere
else, have turned to information technology as a potential means of increased efficiency,
despite the fact that the evidence from studies of costs is not reassuring. The technology
seems always to be an add-on cost. So educational technologists must ask themselves
whether they are bringing information technology into their work in response to such an
agenda, or because the information technology actually serves the theory they hold.

POLITICAL AND MORAL CHALLENGES

If educational technology is to be truly modem, its practitioners must also respond to
political and moral challenges, which happen to come mainly from the Left.

The Frankfurt School of critical theory was formed in 1923. Its founders, including
Marcuse, developed studies of authoritarianism. In the 1930s they began a critique of
new mass media, particularly radio, and of how they were being used by the fascists.
After World War II, they mounted a critique of mass culture, which they saw as
affirming the rule, or hegemony, of bourgeois values because it offered escapism and
gratification, not insight. Habermas and others who came after Marcuse used critical
theory to attack capitalism more directly. In particular, some critical theorists argued that
capitalism was a system in which objects and the processes that produced them were
valued by marketplace exchange, that is, they were turned into commodities or
commodified. The commodification of the whole of society was proceeding apace, said
these critics, recalling that the function of marketed objects was to profit the capitalist.

But what does critical theory have to do with educational technology? Burrell and
Morgan (1979) state that proponents of critical theory "seek to reveal society for what it
is, to unmask its essence and mode of operation and to lay the foundations for human
emancipation through deep-seated social change" (p 284). Translated for us, this might
read "seek to reveal educational technology for what it is, to unmask its essence and
mode of operation and lay the foundation for learners* emancipation through rooting it
out". Harris (1987) thinks of educational technology as a species of positivism, and says
that all the major authors in the field of critical theory consider positivism to be a
particularly acute and modem form of domination. Bowers (1988) asserted that
educational technology, far from being neutral, is a powerful force that will alienate
individuals, destroy community values and, through promoting a falsely objective view
of the world, serve right-wing political and economic forces.

Left-wing critics like these find intellectual support in the work of Lyotard (1984),
whose message about the "post-modern condition of knowledge", in present day western
industrialised society, is that capitalists are turning knowledge into a commodity, to be
bought and sold in a marketplace which they dominate. Fox (1989), who aims at open
and distance learning, rather than directly at educational technology, sees these as a
branch of the "knowledge industry". He suggested ihat "knowledge and the opportunity
to learn are benefits to be bought by private individuals rather than fundamental rights
belonging to every member of a democracy" (p 270). In the circumstances, open and
distance learning become a "dream instrument" for the control of the ruled. This may be
a surprise to those of you like me who are engaged in designing this learning. Burt
(1991) suggested that "educational technologists cannot escape the epistemological crisis
which confronts all the other social disciplines. The new critical theory aims to address
the crisis by making previously suppressed voices heard" (p 229). Burt saw the
educational and training technology literature as tending to express the ideologies of
technological optimism and national economic competition, close to the dominant
ideology of British and US society.

24C

MODERNISING EDUCATIONAL TECHNOLOGY

235

This sort of criticism is hard to take for educational technologists who feel that they
have frequently participated in benevolent, well -intended educational reform. They are
forced to admit that educational technology can be used for the wrong ends, but not that
educational technology is always used thus. Even in general it does not have such a
malign influence, we hear them say. Critical theorists do not deny the good intentions
of educational technologists: they simply say that the results of such endeavours enhance
the promotion of "falsehood and negative value".

The best defence against the critical theorists is to point out that they propose
substituting their own hegemony for the capitalist one, and that educational technologists
desire to be slaves to neither. Critical theorists are essentially nihilistic: they offer no
constructive ideas to those they accuse. Nevertheless, modern educational technologists
would do well to develop their own political consciousness, just as they should modernise
their theoretical foundations, and examine why they are integrating information
technology.

References

Bowers C A (1988) The Cultural Dimensions of Educational Computing: understanding
the non-neutrality of technology Teachers College Press New York

Burrell G and Morgan G (1979) Sociological Paradigms and Organisational Analysis
Heinemann London

Burt Gordon (1991) Opinion: culture and ideology in the training literature Educational
and Training Technology International 28(3) pp 229-237

Chandler D (1989) The Purpose of the Computer in the Classroom In Beynon J et al
(eds.) Computers into Classrooms: a critical appraisal Falmer Press Lewes

Chander D (1990) The Educational Ideology of the Computer British Journal of
Educational Technology 21(3) pp 165-174

Ely D P (1968) Educational Technology as Instructional Design Educational Technology
8 4-6 (referred to by Saettler 1990)

Eraut M (1989) Conceptual Frameworks and Historical Development In Eraut M (ed.)
The International Encyclopedia of Educational Technology Pergamon Oxford

Fox S (1989) The Production and Distribution of Knowledge Through Open and
Distance Learning Educational and Training Technology International 26(3) pp

Gagne R (ed.) (1987) Instructional Technology Foundations Lawrence Erlbaum
Hillsdale New Jersey

Harris D (1987) Openness and Closure in Distance Education Falmer Press Lewes

Lyotard J-F (1984) The Postmodern Condition: a report on knowledge University of
Minnesota Press Minneapolis

269-280

236 QUALITY IN TECHNOLOGY

Percivai F and Ellington H (1984) A Handbook of Educational Technology Kogan Page
London

Romiszowski A (1981) Designing Instructional Systems Kogan Page London

Romiszowski A (1990) Shifting Paradigms in Education and Training: what is the
connection with telecommunications? Educational and Training Technology International
27(3) pp 233-237

Rowntree D (1974 revised 1982) Educational Technology in Curriculum Development
Harper and Row (1974 edition) Chapman (1982 edition) London

Saettler P (1990) Tiie Evolution of American Educational Technology Libraries
Unlimited Englewood Colorado

Spencer K (1988) The Psychology of Educational Technology and Instructional Media
Croom Helm Beckenham UK

Tom A (1984) Teaching as a Moral Craft Longman New York

34 Trends and issues in educational
technology

Donald P Ely, ERIC Clearinghouse on Information Resources, Syracuse University,

There are many ways in which trends couid be identified: expert opinion, panels of
specialists, or informed observation. This study chose content analysis as the primary
vehicle for determining trends based on earlier work of Naisbitt (1982) and his model
(Jar.owitz 1976). The basic premise of these works is that current trends can best be
determined by what people are saying publicly, through newspapers and magazines. This
study, and those that preceded it, used the same basic procedure: identification of
emerging topics in key publications over a period of one year. The literature of the field
has been chosen as the best comprehensive coverage of current thinking and events in the
field. A selected body of literature has been reviewed using a team of educational
technology specialists.

To maintain consistency from year to year, the same sources of information were used
as in two earlier studies, Ely (1989, 1990), with a few exceptions. To aid in the selection
of sources, the Moore and Braden (1988) report was used. This source reported the
people, publications, and institutions of "high Prestige" that were identified by a survey
of personnel in the field. The highest ranking journals and the dissertations produced by
American universities that ranked highest served as two major sources of literature.

Additional sources of data were the papers given at major national and international
conferences and the input to the ERIC database in the field of educational technology.
The ERIC system solicits unpublished materials such as reports, evaluations, studies and
papers for review and, following evaluative criteria, selects the best for inclusion in the
database. The Clearinghouse on Information Resources is responsible for the field of
educational technology and therefore documents selected from that source are likely to
represent current developments in the field.
The complete set of sources used was:

Journals (October 1990 - September 1991)

British Journal of Educational Technology (United Kingdom)

Education and Training Technology International (United Kingdom)

Educational Technology

Educational Technology Research and Development
TechTrends

Dissertation Sources (1991)

Arizona State University Syracuse University

Florida State University University of Southern California

Indiana University

Conferences (1991)

Association for Educational Communications and Technology
Educational Technology International Conference (United Kingdom)
National Society for Performance and Instruction
ERIC Input (October 1990 - September 1991)

Syracuse, NY 13214, USA

INTRODUCTION

LITERATURE SOURCES

238

QUALITY IN TECHNOLOGY

From the reviews of four coders, discussing more than 1300 items, came a list of "topics"
that were most frequently presented in the literature. The recording units offer a first
indicator of trends. Further anaiysis of each category and subcategory reveals sharper
distinctions. At this point the key literature is added to the mix. Key literature consists
of policy papers, reports and statistical data for each topic area that was published during
the time period of the study. This information, together with the content analysis, was
studied by the author who then drafted the trends and sent them to external reviewers.
Changes were made based on their comments.

CONTEXT

This study answers the question, "Where is educational technology headed?" Technology
does not move alone apart from the society in which it exists. Information and
communication technology is being used in the home and workplace at all levels - local,
state, regional, national, and international. To separate it from the context is to highlight
its products rather that its uses and impact. Therefore, much of the discussion in this
study involves the total fabric of technology in society rather than technology as an entity
in itself.

TRENDS

Using a content analysis of the 1991 educational technology literature, the following
trends have emerged:

1 , The creation of technology-based teaching/learning products is based largely upon
instructional design and development principles.

There appears to be more evidence that materials developed for the purpose of teaching
and learning use design principles that have their roots in cognitive psychology and
instructional science. More than 15% of the items reviewed for this year's study were
devoted to design and development. Major subheadings included message design, product
development and course development.

I'. Evaluation ha: taken on greater importance as the concept of performance technology
has been further c* \ sloped

More than 1 09c \>i the 1991 literature was concerned with some aspect of evaluation:
proc< ;ss evaluation, product evaluation, cost/effectiveness assessments and formative
evaluation.

3. The number of educational technology case studies is growing and provides general
guidance for potential users.

More than 1 1% of the literature reported on specific use of media and technology in
teaching/learning settings. Almost all the case studies were "successful " and many could
serve as models for potential users. Very few reported "failure" or negative outcomes.
About one-half of all the case studies related to computer use in teaching and learning.
There were almost no cases of traditional media use or instructional procedures that have
been "proven" in the past.

4. Distance education is evident at almost every educational level in almost every sector.
Distance education has become a major instructional force in education. A recent

estimate is that 25%-50% of U.S. students are reached by distance learning technology

243

TRENDS IN EDUCATIONAL TECHNOLOGY

239

(School Technology News, 1991). There is probably no other single trend that
encompasses the theory and practice of educational technology better than distance
education. Its frequency in the literature confirms this observation.

5. The field of educational technology has more and better information about itself than
ever before.

Eighty surveys about various aspects of the field were reported during the time frame
of this study. For example, studies of the most frequently published.textbook authors,
a list of current dissertations, the extent of microcomputer penetration in the schools and
other such reports help to paint a quantitative picture of the profession.

6. Computers are pervasive in the schools. Virtually every school in the U.S. has
microcomputers.

Computer applications permeated the literature of educational technology in 1991.
Purposely omitted from the analysis of trends were 14 journals associated with computer
assisted instruction (CAI) and conferences that focused on the computer as an
instructional medium. Sometimes the items were directly focused on the use of
computers in the classroom for direct subject-matter instruction but most referred to
learning about the computer as a tool. Many items discussed the resistance or
"roadblocks" to the use of computers in schools.

7. Telecommunications is the link that is connecting education to the world.
Telecommunications is an overarching term that describes electronic point-to-point

connections between individuals and groups. Translated into electronic delivery terms,
telecommunications technology includes connections that utilize existing telephone lines,
dedicated lines, cable and satellite transmission. Some messages are intended to be
interactive, such as electronic mail (e-mail), computer conferences and two-way audio and
video conferences. Some are intended to be one-way, e.g., television directed to
classrooms through cable and satellite systems. The dominant trend within
telecommunications is networking - the electronic connection of individuals who have
common interests.

8. The teacher's role in the teaching and learning process is changing as new
technologies are introduced into the classroom.

Positive attitudes toward technology are emerging from teachers' unions and from
teachers themselves. Introduction of such new approaches as Integrated Learning
Systems (ILS) bring about new roles for the teacher.

9. There is increasing pressure for the schools to consider the adoption of technology
while, at the same time, concern is expressed for the impact of technology on children
in society at large.

This trend is a two-edged sword. Picas for the use of technology in the schools are
increasing in frequency. Simultaneously, there are cries of concern over the impact of
technology, especially television, on children and youth.

10. Professional education of educational technologists has stabilized in size and scope.
There may not be much uniformity in the titles of academic programs that prepare

individuals to serve in the field of educational technology and the academic "homes" are
not consistent from university to university but, in general, the field is holding its own.
Programs tend to include similar content, are primarily offered at the graduate level and
prepare people for similar positions.

240

References

QUALITY IN TECHNOLOGY

Ely D P (1989) Trends and Issues in Educational Technology 1988 Syracuse NY
ERIC Clearinghouse on Information Resources

Ely D P (1990) Trends and Issues in Educational Technology 1989 Syracuse NY
ERIC Clearinghouse on Information Resources

Janowitz R M (1976) Content Analysis and the Study of Sociopolitical Change Journal
of Communication 26(4) pp 20-21

Moore D M and Braden R A (1988) Prestige and Influence in the Field of Educational
Technology Performance and Instruction 21(2) pp 15-23

Naisbitt J (1982) Megatrends Warner Books Inc New York

School Technology News (1991) "Wade Right In," Manager tells Educators (8)9 p 1

35 Training quality and new technology

Antonio R Bartolome, Dpt. de Diddctica i Organitzacio Escolar, Universitat de
Barcelona, SPAIN

INTRODUCTION

The aim here is to describe an interactive course that attempts to improve the competence
of teachers in the use of videos and computers and to discuss the outcomes. The material
is being used in research concerning interactivity levels, path control, communication
channel and cognitive styles. The paper includes results from research work undertaken
in Catalonia, and the design of the interactive video course.

Nine years ago, Chadwick (1983) announced the "Law of Hammer": if you give a small
child a hammer, he will immediately conclude that all around needs a good hammering.
In the educational field, give a trainer a new resource and he will immediately conclude
that ...

Perhaps it is the "Law of Hammer" that turns trainers upside down in computer assisted
instruction (CAI), as they take the big leap to artificial intelligence, flying over hypertexts
on their way towards new authoring languages. In my opinion, it is at this moment that
they should emerge into the street from the multimedia sewer in order to get a glimpse
of the unreal virtual reality world. Today we find ourselves in such fields as these, with
more projects than realities.

At Barcelona University, the Department of Curriculum and School Management has
been training the teachers of the future in the use of educational technology for years,
both with undergraduate and postgraduate courses. Several methodological tools have
been developed for these courses. Among them, we have the "New Technology for
Teaching", hypermedia with interactive videodisc support.

NEEDS ANALYSIS

From 1987 to 1992 several different studies were carried out in Catalonia about the use
of technological resources in schools. In fact, this line of study has been followed by the
introduction of the New Technology Work Group (GTNT). The research was financed
by the Service for Popular Culture Foundation. This study investigated the way in which
teachers used video in their teaching practice and was based on field work that was done
by 6 observers in 50 schools chosen as a good cross-section of all the different schools
in Catalonia (Bartolome and Ferres. 1992).

Amongst the general conclusions were: video equipment was a resource that teachers
handled a lot, but to which students had much less access (see figure 35.1). Even where
pupils had access to video equipment, this was largely as viewers - 92.3% had viewed
video, whereas only 18.5% had used a camera, out of a total of 4000 pupils.

Here we must point out that, in relation to computing, the situation is different: the
utilisation has a much smaller base, but the relative involvement of the students is much
higher.

And so, what is video used for? Out of 15 possible Video uses, the most accepted was
"to transmit information to the classroom group". So we should not be very surprised
when we realize that 78.8% of the 150 teachers who were asked had never even
considered the video as a resource for consulting or studying on an individual basis.

Thus, pupils get little access to the sets of equipment nor can they use the playing or
recordings systems, nor can they get to a viewing system in order to do individualized

24*

242

QUALITY IN TECHNOLOGY

or group work.

We obtained other conclusions from this work and these have provided the basis for the
design of the course "New Technology for Teaching".

H- Yes
| No

Schools in wliich teachers Schools in which pupils

use video use video

Figure 35.1 School utilisation of video

CONCLUSIONS ABOUT VIDEO USE

We can conclude from the research that the teaching method is not modified by the
simple introduction of video technology in the teaching-learning process. Really, it is the
complete opposite. There is a tendency to "domesticate" the media, forcing it 10 act as
a second teacher.

But this study suggests that there is room for more innovative uses, both creative and
participative, where teachers actually integrate the technology in this way.

Generally speaking, we cannot say that teachers have reached the stage at which the
video has been completely integrated in the classroom. Most pupils see video as an
isolated experience, and they do not look upon it as an integral part of every day school
activities.

Despite the relatively small amount of time for which the video is used in schools,
pupils value its incorporation in a very positive way, as an aid to learning. In this sense,
it is a pity that it is not used in a better way, both from a quantitative and a qualitative
point of view.

In general, the study shows that it is the most passive and the most one-way video uses
that are incorporated in schools: the video player is used more often than the camera, and
it is handled more by the teacher, than by the pupils; programme- viewing is more typical
than other activities; more video lessons are watched than motivating programmes.

This tendency towards a passive use of video can be explained by three factors: firstly
the lack of teacher-training about more imaginative and participative uses, secondly the
lack of an adequate technological infrastructure in schools, and thirdly because there is
no time available for creating new "types of use.

A further factor that must also be taken into account is that the manufacturers of
educational video-programmes practically always produce the same video- programme
model, without offering any other alternatives.

250

ERjC

TRAINING QUALITY AND NEW TECHNOLOGY

243

Among the positive values, we ought to point out the fact that the use of the video
workshop is on the increase in schools. Despite the lack of means for this activity and
the fact that it only can be used by a small number of pupils, this is a very creative and
participative formula.

This study has shown that there is a significant deficiency in schools regarding
audiovisual literacy and critical reading of media. Very few schools offer education in
this field, and this means there is a very big reparation between school and society in use
of media.

Incorporating video in teaching practice do;s require an extra effort from the teacher,
and therefore only those teachers who are particularly interested, or who are very
enthusiastic, are likely to put this means into practice.

A PROJECT TO IMPROVE TEACHING QUALITY

From this and other research, some action plans were drawn up. One of these was to
develop material about new technology use that laid emphases on the methodological
aspects of this use. The final products can be defined as follows:

♦ An optical videodisc entitled "New Technology for Education" incorporating
motion pictures and still images about the introduction of video and computers in
both primary and secondary education.

♦ A control software that allows access to the different sequences, helps the user to
get to specific information, gives access to a dictionary of technical terms, and that
also includes written textual information about certain aspects of the subject.

♦ Some versions include individualized and group learning programmes.

Three aspects were taken into account in the course design:

♦ The technical aspects of equipment

♦ The communicative aspects of the media: media language

♦ The educational aspects relating to how to use these media: resources
methodology.

COMMUNICATIVE ASPECTS

The programme has two types of sequence:
Base-sequences

Present key aspects of computing, video and telecommunications in an intuitive and
general way. They do not offer detailed information, but try to give clues tha'. will lead
to further discussion, to suggestions for group-work, to encourage individualized study.
They are video-sequences with moving images and sound effects, and their lengths vary
between 20 and 120 seconds.

Complementary-sequences

Present certain aspects in an accurate and detailed fashion. In order to do this, they use
a written text in the computer, which is illustrated and is based on still images shown

244

QUALITY IN TECHNOLOGY

from the videodisc.

Using any of these sequences, users can access the Dictionary and other buttons are
available as illustrated in figure 35.2.

Macintosh HD:CIDE/Soft:CIDE/Soft BflSICQ:NNTT.SoftUDJENG

Computers in Education

^panish

Computers achieve different results in education

IQI?* It is an object for studying : pupils learn how to get to know its technical characteristics;

they also learn how to program it.
U^^* It is also another means for learning - , it transmits information, it develops skills, it

changes attitudes.

iTj3> As a didactic means it is a tool in the hinds of teachers and pupils.

Finally it is a tool for administration; economic or academic, in the same classroom.

Contents of this chapter

Computers in Education
Setting Up the Equipment
Computers Room
The computer the classroom
Acquiriring programmes

; T>arJw-Tr*-imnn

Figure 35.2 Sample video screen display

DIDACTIC PLANNING

This programme has been designed to meet different needs:

• As a tool to support teaching explanation in specific subjects - the teacher can
offer support to his demonstration by using still images or by employing
whichever video-sequences he considers appropriate.

• As a means to generate small-sized or average-sized group dynamics the -
video-sequences lead to group discussion or other role-playing type activities.

• As support to group learning - the pupils use written guides in which different
themes are presented. Each individual uses the programme to collect information
about these themes, as well as using other resources such as books. Later, together
with the teacher, the results they have achieved in their work are discussed.

• A support to individualized learning - a computer programme directs the learning
by asking a series of questions in which the pupils must use the videodisc, with
different levels of interactivity, to answer these questions.

25°

TRAINING QUALITY AND NEW TECHNOLOGY

245

SHOULD EDUCATIONAL TECHNOLOGY BE USED IN
SCHOOLS?

The last aspect that we need to consider is to ask if it is really worthwhile to use these
resources.

We should like to be able to say that pupils, with the help of technology "learn more,
in a better way and faster ...\ Unfortunately, according to Clark (1983), five decades of
research have shown that there are no benefits to be gained in learning from using
different teaching media. With regard to reducing the amount of necessary learning time,
Clark points out that the new media hold a slight advantage over more conventional ones.
Kulik et al (1980) carried out a meta-analysis between traditional courses and courses that
used computer-based training, with results showing an effectiveness in favour of the latter
type. This greater effectiveness was considerably reduced when the same teacher prepared
and taught both the experimental and control group. In the long-term results, this
effectiveness practically disappeared.

But these considerations make a basic mistake: they try to make a comparison to show
what happens when we teach old material with new resources. MacLuhan was probably
right when he related contents and media: new media teach new contents, they address
different aims than traditional media.

The use of new technology is related in a strange manner to studies about the hidden
curriculum and with the school that we present to our pupils: a shool which is separated
and different from the world outside in which they live (Treffel, 1981). A school without
computers teaches something to students: "this school is not related to your world, to
your needs, to your future; we are not interested in your society". It does not matter that
teachers do not think so: this is the content of the hidden curriculum.

References

Bartolome A Y and Ferres J (1992) L'Us Del Video a les Escoles de Catalunya Serveis
de Cultura Popular Barcelona

Chadwick C B (1983) Los actuales desafcos para la Tecnolog^a Educativa Revista de
Tecnologga Educativa 8(2) pp 99-109

Clark R E (1983) Reconsidering Research on Learning from Media Review of
Educational Research 53(4) pp 445-459

Kulik C et al (1980) Effects of Computer Based Teaching on Secondary School Students
Journal of Educational Psychology 75 pp 19-26

Treffel Jacques (1981) Presents et Fuiurs de l' Audiovisual en Education La
Documentation Franchise Paris

36 Connecting lectures and laboratory
learning with CAI

Yoshikazu Araki, Saitama Institute of Technology, 1690 Fusaiji Okabe Saitama,
Japan 369-02

INTRODUCTION

As the number of college and university students rapidly increases, their educational
needs, motivations, levels and abilities become more diversified. Also their attitudes
towards their studies and their interests in college life are quickly becoming more varied.
To make matters worse, the number of students who are unable to fully understand the
lectures, and are therefore less likely to be willing to study, is gradually increasing. It
may be because the students do not understand the materials being presented to them or
that teachers are not making an effort to examine their students* progress. Perhaps the
fault is with the teaching methods. The use of textbooks and handouts does not suffice
to ensure the students' comprehension of a lecture. A professor {particularly of
Engineering) cannot expect the students to easily grasp theories and schematic
explanations without the use of visual aids. In Engineering, if we can explain theories
with data derived not only from simulation but also from the actual device or circuit,
lectures will be more persuasive and practical. Using personal computers, the author is
introducing a technique to show real experimental data that verify the theories presented
in the lectures.

PROBLEMS WITH LABORATORY

It has been usual in Japanese universities and colleges for the lectures and seminars to
be given independently of the laboratory courses. The same teacher rarely does both.
In order to provide the students with more practical knowledge, the laboratory courses
are particularly important in the university curriculum for engineering.

In order to maximise effectiveness, it is highly desirable that the relationship between
the lectures and relevant laboratory learning be significantly strengthened.

In an average Japanese laboratory course, a student is obliged to work with several
other students because of the shortage of measuring equipment and space. Thus , some
students are unable to participate in all the projects of the laboratory. Unhappily, the
result is that those students tend to rely on their classmates for the material in their
reports, and do not develop their own ability to research the subject thoroughly.

APPROACH TO SOLVE THE PROBLEMS

In order to solve these problems and to improve this situation. Imamura et al 1989 have
suggested that teachers should make much more effort to improve their teaching
techniques, by utilizing various new media. The author is developing such a new
teaching system. It consists of three parts: first, the Pre-Laboratory-CAI Test (Pre-Test);
second, the laboratory; and third, the Post-Laboratory-CAI Test (Post-Test). We call this
system RECALL (Real Experience Computer Assisted Laboratory Learning). Figure 36.1
shows the fundamental concept of RECALL system.

The Pre-Test is done to check basic knowledge of experimental theories, equations and
methods which are necessary for understanding and finishing the laboratory successfully

2^/

CONNECTING LECTURES AND LABORATORY LEARNING 247

within a limited time. If the result of the Pre-Test is less than 70%, the student should
study the subjects using tutorial type CAI. Then he should take the Pre-Test again. If
the result is over 70%, he is permitted to begin the laboratory. He is permitted to take
the Pre-Test three times within 45 minutes.

Pre-

Test — d

Quiz-type CAI
Tutorial type

Test
CAI

Laboratory

Experiments |

Post

-Test — d

Quiz-type CAI
Tutorial, type

Test
CAI

[ Making Reports 1
Figure 36.1 Fundamental concept of RECALL system

COMPUTER ASSISTED MEASUREMENTS

We have been developing a system, using a computer, to measure voltage-current
characteristics of a bipolar junction transistor (BJT), a junction type field effect transistor
(JFET) and a metal-oxide-semiconductor type field effect transistor (MOS FET).

The hardware configuration of the system consists of common hardware, a curve tracer
unit and signal generators. The common hardware consists of a personal computer, an
I/O interface card and an analogue interface unit. The personal computer is PC-9801
(NEC) with 8086 family CPU. The analogue interface unit converts external analog
signals into digital data and vice versa with 12 bits A/D, D/A converters. The curve
tracer unit is used literally as a curve tracer for the measurement of voltage-current
characteristics of BJT and FET. The unit contains V-I converters and voltage amplifiers.
The converter converts voltage supplied from the D/A converter into current (e.g. I B of
a BJT). The amplifier is used for providing voltage (e.g. V CE of a BJT).

Common software consists of some controlling programs and the main program. The
former control data flow between analog interface unit and personal computer for each
application. The latter integrates application softwares.

Before starting the laboratory, students should know the theories behind these devices.
In the laboratory, students not familiar with measurement equipment often acquire
insufficient or incorrect data due to errors in operation, thus wasting time. This system
reduces measurement time allowing students more time to spend on consideration of the
main theme of the laboratory.

In the laboratory, input, output and mutual transfer static characteristics of the devices
are measured. All measurements are done automatically by following messages displayed
on the CRT of the personal computer. Table 36.1 shows an example of measuring

248

QUALITY IN TECHNOLOGY

procedures. Measuring steps for V DS -I DS output static characteristics of FET are as
follows;

1 Inputting student's number in order to store learning record.

2 Inputting measuring conditions such as weather, temperature and humidity.

3 Displaying the explanation of a measuring operation.

4 Inputting a name of a device.

5 Preparation for measurement.
Setting i) output voltage V DS

ii) maximum input voltage V GSo)tx

iii) maximum output current 1^^

iv) step increase in output current I DS

6 Measuring Vx> s -los output static characteristics.

Here V DS and V GS are applied from the D/A convener. Values of I DS are taken
into the A/D converter.

7 Outputting V DS -I DS output static characteristics on a CRT and in a printed form.

8 Measuring other characteristics.

Alter taking the measurements students should learn the characteristics of the
device by using their own data to answer questions such as the calculation of h-
parameters, input resistance and output resistance. For example, from the result
of Grounded Emitter output static characteristics, a student should calculate hpp
at a certain point on the output characteristic curve according to a message
displayed on a CRT. After calculating the h^ the student inputs the value and the
computer checks it. These processes are repeated until the right answer is
obtained. Thus students should come to understand the actual characteristics by
the use of real data.

After all the processes mentioned above are finished, the Post- laboratory- CAI Test (Post-
Test) is taken to check understanding of the completed section of the laboratory. If the
result of the Post-Test is less than 80%, the student should study the subjects using
tutorial type CAI. Then he should take the Post-Test again. If the result is over 80%, he
is permitted to finish the laboratory. The time allotted for the Post-Test is limited to 45
minutes. At the end of the laboratory, students submit reports including data, problems,
results of the Pre -Test and other considerations.

Heasuring Flow

( 1 ) 1 npu t t i ng

(2) Preparation for neasurement

(3) Measuring

(4) Outputt ing

Laboratory Experiment Condition

Measurement f

or Voi Ids output static characteristics of FET

(D output voltage V os

(3) aaxiaua output current Ios. t .

Measurement

for V t Ic output static characteristics of BJT

OD output voltage Vt a

(?) aaxi&um input voltage Vun.

(3) aaxiausi input current It...

0) step increase in input current Ah

Tabic 36.1 Measuring procedures

CONNECTING LECTURES AND LABORATORY LEARNING 249

CONCLUSION

The author has developed a new learning system where students measure characteristics
of real semiconductor devices such as V DS -I DS or h parameters using personal computers,
then confirm the theoretical characteristics by CAI. This system combines practical data
and phenomena with theory which is taught in a regular lecture course.

Through the RECALL System (the Pre-laboratory-CAI Test, the laboratory and the
Post-laboratory-CAI Test) and CMI system, students* understanding of laboratory is
accurately and easily evaluated.

ACKNOWLEDGEMENTS

The author wishes to thank Professor A Nakajima, Mr H Imamura, Mr Y Kobori and Mr
T Ono for their advice. Also he would like to thank Ms E Chatillon for correcting the
English.

References

Araki A (1991) CAI and CMI Considering Student's Achievement and Consciousness
in Winterburn R (Ed) Aspects of Educational Technology Realizing Human Potential
Kogan Page London

Imamura H, Kobori Y, Ono T and Araki Y (1989) A New Teaching Method Using
Computer Processed Real Data 1989 Frontiers in Education Conference Proceedings
pp 105-110

257

37 Graphical routes to quality courseware

Philip Barker and Charles Lamont, University of Teesside, Cleveland, UK
INTRODUCTION

Computer-based training (CBT) is concerned with the automatic delivery of training and
learning material using a computer system. In the past, screen-based text has been the
main method of communication. However, it has been shown that augmenting basic
textual material with appropriately designed static graphic imagery produces a
communication mechanism of significant pedagogic effectiveness. The use of moving
pictures (in the form of TV quality video images and animated graphics) has similar
potential for improving the quality of multimedia instructional delivery within CBT
systems. Further improvements in the quality of CBT courseware can be achieved
through the incorporation of hypermedia techniques (Richards and Barker 1992).

Unfortunately, the development of multimedia and hypermedia courseware can be highly
problematic unltss suitable development tools are used. For this reason a considerable
amount of research has been devoted to the development of tools to automate the
production of courseware (Barker 1990: Barker and Lamont 1992). The motivation for
using these tools primarily arises for four basic reasons: first, the need to decrease the
number of steps and/or phases involved in the overall courseware development process;
second, the need to reduce the level of expertise needed to develop effective and efficient
courseware; third, the growing necessity to cut down courseware development costs; and
fourth, the need to reduce the amount of time devoted to producing instructional software.

In this paper we describe an approach to courseware automation which we have found
to be extremely effective. It is based upon the use of "custom editors" that are designed
to augment and enhance the facilities provided by the graphical user interface to the
PRO PI courseware development system (ASYS Computer 'Services 1988). PROPI is
essentially a courseware generator that produces PC/PILOT code (Barker 1987; Conlon
1984). In the remainder of this paper we provide a short description of the PROPI
graphical user interface and then illustrate how custom editors (written in C) can be used
to provide high-level, easy-to-use authoring facilities to ease the production of
high-quality multimedia and hypermedia courseware.

GRAPHICAL USER INTERFACES

People interact with computers by means of appropriately designed human-computer
interfaces (Barker. 1989). Screen-based interfaces fall into two basic categories: character
orientated (based upon the use of a keyboard); and graphical user interfaces. A graphical
user interface (GUI) relies upon the use of icons, images and pictures to support pointing
operations based upon the use of a mouse. There is considerable evidence to support the
claim that graphical user interfaces are easier to use and that they can lead to substantial
increases in end-user productivity.

A number of CBT authoring systems are now supplied with graphical user interfaces
which act as 'front-ends' to an existing development tool. TenCORE and PC/PILOT are
two examples in which this approach is used. In many other systems (such as
HyperCard, ToolBook and Authorware Professional) the GUI is the only interface that
is available to courseware authors. Because of their ease of use, systems of this sort are
becoming extremely popular - mainly because users need not have any technical
knowledge of computers or do not need to know how to write computer programs. There

GRAPHICAL ROUTES TO QUALITY COURSEWARE

251

is another reason why these GUI-based authoring systems are growing in popularity: they
provide a relatively easy way of developing high-quality courseware.

The ease of use of a GUI-based authoring system stems from the fact that each generic
type of courseware authoring function is represented on the author's computer screen by
means of a reactive icon. PROPI, for example, has seven basic classes of icon: input,
output, decision, mark, question, group and external. A fundamental property of these
reactive icon. c , is that they can be selected (using a mouse) and cloned. A cloned icon can
then be positioned on the CRT screen so as to form part of some more complicated
graphical structure.

In order to produce a piece of instructional software, a courseware author uses the
screen -based menu of icons in order to construct a "lesson map". This is simply a
pictorial representation of the logic flow and functionality that is to be embedded within
the courseware. The lesson map is essentially a graph structure which is very similar in
appearance to a computer flow-chart. Each type of node within a lesson map has
associated with it a series of content "editor" templates. These arc used either to specify
the pedagogic content of a node or to give control information that specifies how a node
is to be delivered. Once a graphical lesson map has been produced it can be
automatically converted into CBT code.

CUSTOM EDITORS

Authoring tools of the type described in the previous section provide a powerful
mechanism for generating courseware provided, of course, that they are able to offer all
the major functions that authors require. Invariably, due to the very nature of progress,
there will always be some things that are missing. Some mechanism must therefore be
provided which will allow users to "extend" an authoring environment in ways that are
relevant to their own particular needs and requirements. This mechanism is provided in
PROPI, for example, by means of "custom nodes" (ASYS Computer Services 1988). The
purpose of custom nodes is to facilitate the automatic generation of PC/PILOT code for
performing tasks which are not directly catered for by PROPI itself.

A custom node (within a PROPI lesson map) is a generic node which, when invoked,
causes the activation of an external (user written) "custom editor". The purpose of a
custom editor is to generate CBT code from parameters entered by the courseware author
while a CBT lesson is being developed. Obviously, there is no limit to the range of
custom editors that a particular PROPI authoring environment can make available to
courseware authors.

The program code used to produce a custom editor can be written in any programming
language (we use C for most of ours). During its execution a custom editor must
generate a user-interface to facilitate the entry of the parameters that are needed in order
to build the target code that is to be generated. Obviously the design of end -user
interfaces to custom editors offers considerable scope for the application of GUI
techniques and the provision of mechanisms that will minimise any errors that authors
make with respect to the entry of code generation parameters.

In the work that we have undertaken using PROPI, a wide range cf custom editors have
been built for use with electronic books (Barker 1991) and for controlling various
multimedia presentation devices. The following section presents a case study which
describes one of the courseware automation projects in which we have made extensive
use of PROPI custom editors.

252 QUALITY IN TECHNOLOGY

A CASE STUDY - MULTIMEDIA AND HYPERMEDIA CBT

In this case study we describe some of the ways in which we have been using custom
editors (within the context of courseware automation) in order to: (a) provide "user
friendly" interfaces to "high technology" information display devices; (b) make available
a standard and consistent graphical user-interface (GUI) to a range of different peripheral
devices; and (c) improve the quality of CBT courseware through the incorporation of
interactive multimedia and hypermedia techniques. The essential features of this
courseware automation project are illustrated schematically in figure 37.1.

Philips vP-705 Custom Editors

video D.se Ed.tors

Slngtl Iirt40l Dliptay

VTdae Clip DltpUy

Ravarsa Play

Fcst Play

SlSir Play (FhJ)

Autto t ConUal

Audio U Control

Vldao MuU OrJOU

ate

Cu«or*. Ed *0' utti'y

Hitachi CDR-1503S Editor*

Stauc Plitur* D:tP:ay

CDROJ.t Editors

Functional Love!

Video
Disc

Corrajct

Disc

Grsprrcs
Boards

Otr-cr
Devices

PROPI
Courseware
Automation

System

Generic Level

System Level

Figure 37,1 Multimedia custom editors for use in PROPI

In order to "insulate" courseware developers from the complexity of programming these
devices the GUIs employed in our custom editors are standard ones for all devices (or
technologies - such as CD-I and DVI) within a given generic class. That is, no matter
which actual device is employed for information delivery, the GUI allows a consistent
style of use, has a standard appearance and provides access to a standard repertoire of
cont' 1 options. Courseware authors can access the particular editor of their choice either
by a menu selection strategy or directly by entering its name.

To date we have implemented a full set of custom editors for the various types of video
disc equipment that we use. Custom editors arc also available for basic compact disc
equipment, selected graphics boards and one or two special devices such as a touch
screen and a concept keyboard. Descriptions of the design, implementation and evaluation
(from the courseware authors* point of view) of these custom editors are presented in
detail elsewhere (Lamont 1991). Our current work with custom editors is now directed
at extending the range of editors that we have available for use with digital optical
storage technologies. Because tiiese technologies are likely to have a significant impact
on courseware development for CBT, we are now extending our range of eduors to
include facilities for handling digital video interactive (DVI), compact disc - interactive
(CD-I) and compact disc extended architecture (CDROM-XA). Further details of this
work are described in Lamont and Barker (1992). One very important aspect of this
particular courseware automation project from both the courseware authors* and the

2ii0

GRAPHICAL ROUTES TO QUALITY COURSEWARE

253

students* point of view is "image reactivity". That is, the ability to embed and use
reactive areas (or hotspots) within images and text so that when one of these areas is
selected (using a pointing device such as a mouse) other related processes are initiated.
Image and text reactivity is an essential requirement for the creation of hypermedia
structures and their subsequent incorporation into CBT courseware. To this end we have
designed (and are currently in the process of implementing) a range of custom editors to
facilitate: (1) the insertion of hotspots into images; and (2) the linking together of image*
text and sound within a hypermedia knowledge corpus that is held on digital optical
storage media (primarily, CD-ROM). We believe that these custom editors will
substantially improve the productivity of our authors and the quality of the courseware
that they produce.

Multimedia and hypermedia techniques are playing an increasingly important role in the
development of quality courseware. However, the incorporation of these techniques into
instructional software requires significant technical (and pedagogic) skills if they are to
be used in an effective way. While it is difficult to automate the pedagogic aspects of
authoring, the technical requirements of authors can be substantially reduced if
appropriate courseware automation techniques are made available. We have found that
graphical user interfaces (used in conjunction with custom editors) can play a significant
role in easing the burden of courseware authoring for multimedia and hypermedia CBT.

The authors are indebted to both the Science and Engineering Research Council (SERC)
and to A.P. Chesters and Associates, 17 Newhall Court, George Street, Birmingham, B3
1QA, for the financial support to enable this work to be undertaken. Financial
contributions from University of Teesside, Eyetech Ltd and the Training, Enterprise and
Education Directorate of the Department of Employment are also gratefully
acknowledged.

References

ASYS Computer Services (1988) PROPI User's Manual ASYS Computer Services
2601 North Shore Road Bellingham WA 98226 USA

Barker P G (1987) Author Languages for CAL Macmillan London UK

Barker P G (1989) Basic Principles of Human-Computer Interface Design
Century- Hutchinson London UK

Barker PG (1990) Automating the Production of Courseware in Farmer B, Eastcott D
and Lanz B (eds) Aspects of Educational Technology- Volume XXIII: Making Learning
Systems Work Kogan Page London UK

Barker P G (cd) (1991) Electronic Books Special Edition of Education and Training
Technology International 28(4)

CONCLUSION

ACKNOWLEDGEMENTS

254

QUALITY IN TECHNOLOGY

Barker P G and Lamont C W (1992) Courseware Automation Through Custom Editors
in The CTISS File Special Edition on Courseware Authoring Systems

Conlon T (1984) PILOT - the language and how to use it Prentice Hall Englewood
Cliffs New Jersey USA

Lamont C W (1991) Human-Computer Interfaces to Reactive Graphical Images Draft
PhD Thesis University of Teesside Cleveland UK

Lamont C W and Barker P G (1992) The Design of Custom Editors for Digital Optical
Storage Media Working Paper University of Teesside Cleveland UK

Richards S and Barker S (1992) Improving the Quality of CBT Through Hypermedia
paper submitted to AETT '92 International Conference University of York 6th-9th April
1992

262

38 Enhancing the quality of student-centred
mathematics

David Bowers and Rod Burrell, Suffolk College, Ipswich, UK
INTRODUCTION

This workshop addresses how aspects of information technology can influence the quality
of the learning experience and in particular how IT can promote a flexible response to
individual student need. This will be related primarily to delivery within the area of
mathematics.

The need to meet customer demands in delivering a quality flexible response has led
Suffolk College to investigate the use of authoring software which allows a one to one
relationship between the student and the lecturer through the use of such software.

The need to identify the diagnostic element of the student lecturer exchange can be
satisfied by the logic built into the authoring solution. More sophisticated communication
is non-linear, expressing at least a two way conversation: the use of a quality IT solution
delivers this.

The aim of the workshop was to encourage participants to identify quality delivery
within the mathematics area, specifically related to information technology solutions; and
to allow hands-on experience of a number of examples of related software.

The objectives were:

i) to set the scene within die wider educational arena;

ii) to explain how Suffolk College has harnessed a number of organisational and
technical solutions to deliver mathematics to a diverse student population;

tii) to explore the use of authoring software as a link between the flexible response
and the individual student need.

SETTING THE SCENE
Suffolk College

Suffolk College is a classic example of a mixed economy institution. It plays an essential
role in the delivery of Suffolk's Post- 16 education and training. This includes
school-linked A level and vocational provision, non- advanced further education, adult and
community education, and advanced further education.

The Academic planning figures for the next three years project an increase in student
numbers of 31 percent and all departments within the College arc addressing ways of
enhancing the quality of student provision.

Mathematics

Recent developments in the qualitative nature of mathematics provision were discussed,
as well as advances in computer technology. Together, these provide motivation for
investigating the opportunities for computer based learning in mathematics, and workshop
participants were invited to contribute their own experiences. In particular, the following
have had an impact on mathematics provision:

263

256

QUALITY IN TECHNOLOGY

♦ Common Skills, Core Competences, and Curriculum Entitlement - this new
awareness in course design requires a formal response from maths education

♦ Syllabus changes - these result in potential inconsistencies at course "interfaces"
(eg GCSE. BTEC OND)

♦ Mathematics Workshops - these necessitate a re-appraisal of delivery methods (eg
resource-based learning)

♦ Mechanical calculating devices - yesterday hand calculators, today
microcomputers, tomorrow ...?

Information Technology
Hardware Developments

Hardware performance has increased dramatically whilst, at the same time, costs have
fallen. This has led to the availability of hardware to perform complex processing tasks
at affordable prices.

Software Developments

The release of Microsoft's Windows 3.1 together with its multi -media extensions kit, will
allow the ease of use of such media as CD-Roms, still and moving video, and sound, in
many application areas.

Delivery of the Curriculum

The use of Computer Based Learning has already allowed the curriculum to be delivered
in non- traditional ways, the coming of improved hardware and software products will
allow the curriculum to be delivered in an even greater variety of ways. The College has
to be able to facilitate this delivery.

SUFFOLK COLLEGE CASE STUDY

This case study served as a basis upon which participants could relate their own work.
It also provided examples for critical evaluation and as a springboard for new
developments.

A recent HMI report noted that: "Almost half the colleges in England have...
Mathematics Workshops, where students may study on their own or under guidance with
learning materials which suit their individual needs ... a balance between structured
sessions and workshops is a particularly good approach to teaching mathematics at this
(FHE) level".

However, it continued: "The use of IT in mathematics teaching is still not widespread-
there is scope for using the microcomputer more extensively and imaginatively in
mathematics ... IT equipment must be made more accessible to students".

The facilities of the Mathematics Workshop at Suffolk College were outlined, where an
open plan area contained a 24 station NOVELL pc network as well as a BBC
microcomputer ECONET network.

Two examples of currently available commercially produced mathematics
leaching/learning software packages were available for demonstration on BBC
microcomputer.

264

STUDENT-CENTRED MATHEMATICS

257

These packages ("Revise Maths" by A VP Computing, and "Mathspad" by Longman
MicroSoftware) had proved particularly successful at Suffolk College. Participants in the
workshop could evaluate this software, and in doing so become aware of; examples of
good practise; areas with the potential for further development.

SHARING OF PARTICIPANTS* EXPERIENCE
Group Activity 1

The participating groups were asked to share examples of good and useful software that
they have used.Why did they find it so? Does it have any drawbacks or limitations?

Group Activity 2

To establish the criteria for good software packages to aid the teaching/learning of
mathematics, the Group were invited to answer the question "effective teaching and
learning software should ensure that
The following criteria were established by the group;

• use of the software should be transparent to the user

• the user should attain a sense of achievement by using the software

• the software should make full use of visual display capability (colour,
graphics etc)

• the software should be "tried and tested"

• users should be encouraged by the software to consider options and make choices
at regular stages

• the user should be allowed to respond freely (ie not have to make a choice from
pre-defined responses)

• the software should offer different "difficulty levels" at the outset

• the software should include a "help" option ("panic button") to give the user
clarification and advice

• the objectives of the software solution should be pre-determined

• the software should not cause too many irritating bleeping noises

• the software should allow precise definitions of the "task" by the teacher

• the user should be given "diagnostic" response by the software

• the user should be given "recap" facilities (return to start; tc other pages)

• the software should give summary and detailed feedback to the user

• the user should be given "mapping facilities" by the software

• the software should comment on what the hardware is doing.

INTRODUCTION TO AUTHORING SOFTWARE

There is a need for effective methods of developing material for computer based training
systems.

In particular, there is an increasing need for authoring to;

• promote Open Learning, which is widely acknowledged as a method of delivery
which provides flexibility and improved utilisation of staff and student time

Aw ^ I J

258

QUALITY IN TECHNOLOGY

♦ improve delivery methodology by incorporating technology and multi-media to
enhance the presentation

♦ allow different areas to be customised to specific needs, so allowing an
organisation or individual to fill in the gaps that more traditional methods cannot
provide.

Authoring is now viable because of:

♦ the technological developments in PC technology

♦ the development of multi-media technology - sound, picture, video

♦ the reduction in cost of hardware, so making authoring economically viable

♦ and the need for improvements in the quality of teaching and training.

What makes for good authoring software?

♦ ease of development of the teaching material - the interface of the authoring
system must be highly interactive, and require non -programming skills

♦ a high level of interaction between user and system including monitoring and
assessment of the learner

♦ interface to multi-media technology - the ability of the systems to use interactive
laser disk, CD-Rom, sound and scanned pictures if required.

DEMONSTRATION OF THE SOFTWARE SOLUTIONS

The participants were able to try a number of simple authoring packages. In particular,
the overhead displays used throughout the session were generated using IBM's
LINKWAY, an authoring solution which is particularly appropriate for presentation
purposes.

A number of illustrations of mathematics solutions using LINKWAY were available,
which demonstrated the ease of use and effectiveness of simple features such as "pop-up
screens" and "go-buttons". More advanced features were discussed after the
demonstrations and brought the session to a successful close.

oqan Page
120 PentonviiieRoad
London N1 9JN

Published m the USA by
Nichols/GP Publishing Company
PO Box 6036, East Brunswick, New Jersey 08816

ISBN (UK) 07494 08979
ISBN (US)Q-89397-385-8

25'

Internet Archive Audio

Featured

Top

Images

Featured

Top

Software

Featured

Top

Books

Featured

Top

Video

Featured

Top

Mobile Apps

Browser Extensions

Archive-It Subscription

Save Page Now

Full text of "ERIC ED358830: Quality in Education and Training. Aspects of Educational and Training Technology. Volume XXVI."

See other formats