Environmental labs measure the effectiveness of conforming to international standards
Active conformance to international
standards such as
ISO/IEC9001:2000 is intended
to help organizations consistently
produce a quality product
(material or service) and instill a culture
of continual improvement in their
operations. The fact that registration to
ISO standards is a growing phenomenon
(greater than 500,000 worldwide)
strongly suggests that the process is
effective.
But the question must be asked: Is it
successful because it measurably
improves the performance of the operation,
or is it successful because it has
become a powerful marketing
tool - a cost of
doing business? The analytical
laboratory industry
in Canada is an excellent
case study in the true
and measurable effectiveness
of conformance to
an international standard
(specifically, ISO/IEC
17025).
Figure 1. A comparison of proficiency testing failure rates
between accredited and non-accredited laboratories.
Before 1990
Twenty years ago,
shopping for analytical
laboratory services was
like buying a car without
road testing it. You may
have ended up with a
good quality vehicle or
you may have gotten a
lemon. Anyone with a
few dollars and a basic
knowledge of analytical
chemistry (or not) could
set themselves up to provide analytical
chemistry services to the general public.
This resulted in, not only a great
diversity in quality between laboratories,
but often a great diversity in quality
within the same lab. The highly
competitive nature of the analytical
chemistry market and the general lack
of accountability occasionally resulted
in laboratory practices that would not
be considered acceptable by today’s
standards. These included the use of
methods that were not properly validated,
the use of non-qualified or poorly-
trained staff to perform testing,
skipping of critical steps in analytical
processes, use of out of date standards
and reagents, and numerous other
questionable practices.
This situation was not helped by the
practices of the predominant client for
many of these services, the engineering
community. Too often, the decision
on what laboratory to use was based
solely on two criteria, cost and turnaround
time. The perception being presented
was that the signed report was
more important than the quality of the
results on the report. Presumably,
some clients felt that having a signed
report absolved them of any liability
resulting from erroneous data. We now
know that legal liability cannot be
rationalized this easily.
1990 to 2004
In 1989, the laboratory community
cooperated to form the Canadian
Association for Environmental
Analytical Laboratories (CAEAL).
The main goal of the association was
to improve the quality of environmental
information in Canada. It is also
fair to state that individual laboratories
were motivated by an attempt to level
the playing field amongst analytical
laboratories and a desire to improve
how the laboratory industry was being
perceived by the general public.
Membership required a laboratory to
abide by a set of by-laws, a code of
conduct, and client confidentiality.
CAEAL-registered laboratories could
also expect increased business as
(hopefully) clients preferentially
selected registered laboratories over
non-registered laboratories.
The first program to be established
by CAEAL was a Proficiency Testing
(PT) program in which samples were
sent to member laboratories twice per
year. A laboratory’s performance for
each parameter was evaluated against
all other laboratories. Successful participation
resulted in proficiency certification
on a parameter level basis. In
1993 the accreditation program was
added. This program, for which PT
participation is mandatory, assesses a
laboratory’s conformance
to ISO/IEC 17025 (updated
in 1999 from ISO/IEC
25). Both the proficiency
testing program and the
accreditation program
have grown significantly,
to the point where there
are now more than 350
participants in the PT program
and over 200 laboratories
in the accreditation
program.
Many laboratories,
both government and
commercial, have decided
that accreditation and PT
participation is a good
business decision. This
has been reinforced by
recent government regulations.
But the question
still remains as to whether
these programs, when properly implemented
and maintained, actually result
in a measurable improvement in the
quality of analytical results or whether
their benefit is limited to their marketing
value.
The CAEAL proficiency testing
program has been in operation since
1989 and has grown in size and scope
ever since. This extensive database is a
valuable tool for evaluating the effect
of various factors on laboratory performance.
In 2004, A. Morris and D.
Macey published a paper clearly
demonstrating that accredited laboratories
perform better on proficiency
testing samples than non-accredited laboratories (Accred Qual Assur 9:52-54). Given that laboratory accreditation is contingent on demonstrating conformance
to an international standard, this result is not unexpected.
This leads to the question: If accreditation results in
improved performance, is there an added value to proficiency
testing or is this program superfluous? Again, the
CAEAL proficiency testing database contains the information
necessary to answer this question. The database was
queried to retrieve all laboratory/parameter combinations
where first time PT participation started after October 1994
and participation continued for at least ten consecutive PT
rounds. For each laboratory/parameter combination, the
first participation was assigned to study one. In total,
29,480 individual scores were used in this examination.
When the average PT scores were plotted, the trend was
obvious. Average performance (as measured by PT score)
improved over the first few rounds, after which it leveled
off. Since this data-set includes accredited labs, nonaccredited
labs and labs that obtained accreditation within
the ten study window, it can be stated with confidence that
the observed trend is due to PT participation, not due to
other factors that may affect quality.
This phenomenon is supported by observations made for
clinical laboratory PT programs. For example, in 1994, PT
participation was mandated for clinical laboratories in the
United States. This resulted in a decreased percentage of
acceptable PT scores, due to the influx of a large number of
first-time participants, followed by improved numbers over
the next several PT rounds (Hassemer, D. 1996. Wisconsin
State Laboratory of Hygiene, newsletter # 10). The curve is
very similar to the one observed for the CAEAL data.
Figure 2. Average CAEAL proficiency testing score for the first ten PT rounds after the first participation event.
2004 and on...
In the aftermath of the Walkerton tragedy, provincial
governments enacted new legislation and regulations with
the goal of protecting the safety of drinking water. A
requirement in almost all of these regulations has been that
water utilities use laboratories that have demonstrated proficiency
(i.e., PT programs) or are accredited to ISO/IEC 17025. Although this is a significant
step forward in assuring the quality of
drinking water, it only provides a
means of quickly and accurately identifying
problems after they occur. It
does little to prevent these problems
from occurring in the first place. This
is because much of the chemical testing
conducted in order to optimize and
monitor the treatment process (e.g.,
chlorine, turbidity, colour, fluoride,
pH, etc.) is not performed in accredited
laboratories. Generally, these
process tests are governed under operator
certification or through specific
analyst certification. There is little or
no direct oversight of the testing procedures
used, whether they are suited to
purpose or whether they perform as
designed. This is disconcerting given
the fact that results from these tests are
used in day-to-day operational decisions.
To illustrate the importance of
proper controls at the process testing
level, an analogy to loss control theory
can be drawn. Most managers and
supervisors are familiar with the iceberg
or pyramid models of accidents.
For every accident there is a large number of near misses and an even
greater number of substandard conditions
that could lead to an accident.
The best means of reducing the number
of accidents is to directly address
these conditions. Following the same
logic, the most effective way of reducing
the number of adverse water quality
conditions (or C of A exceedences)
is to address substandard process conditions,
including the process testing.
As has been demonstrated with analytical
laboratories, two ways to
improve the accuracy and reliability of
test results is to actively participate in
a proficiency testing program and to
be part of an operation that conforms
to an applicable standard. The trend
towards the management and operation
of utilities to international standards
started prior to the Walkerton
tragedy but has been accelerated in
recent years. Ontario has enacted legislation
that will require drinking water
utilities to become accredited to a specific
standard (still to be defined).
However, proficiency testing, a tool
with a proven track record, has not yet
been considered in the field of process
testing. In large part, this is due to the
general belief that analytical testing
and process testing are fundamentally
different. This is a misconception. It is
true that the average process test may
not need the accuracy expected for
analytical labs, but this does not justify
a complete lack of control procedures.
At present, many utilities lack or
have inadequate procedures in place to
ensure that the process tests are performing
as designed. Sometimes, the
first indication that there is a problem
with the test is when the results
obtained for samples are observably
different from those typically measured.
By now, any problems with the
testing procedure have been reflected
in the entire treatment process.
To conclude, the experience of the
environmental laboratory industry in
Canada clearly demonstrates that proficiency
testing and accreditation to an
international standard provide real and
measurable benefits to laboratories
that undertake these strategies.
Contact Ken Middlebrook, CAEAL,
e-mail: kmiddlebrook@caeal.ca.
See our home page on how to order your subscription. We regret we can
only accept orders from Canada and the United States.
