For the proficiency testing scheme Progetto Trieste* a new version of the evaluation criteria has come into effect since 2024. These changes have been implemented to align more closely with ISO 13528:2022 ‘Statistical methods for use in proficiency testing by interlaboratory comparison’. Before these changes, Test Veritas was already following ISO 13528 but, as several accredited PT providers, we were not completely aligned with it.
A laboratory may think that every UNI CEI EN ISO/IEC 17043 accredited proficiency testing provider follows the same approach in the statistical design and interpretation for laboratories’ assessment. In reality, it is a common practice of UNI CEI EN ISO/IEC 17043 accredited proficiency testing providers, to elaborate their statistical plan using several sources, not necessarily only ISO. In conclusion, by participating to two UNI CEI EN ISO/IEC 17043 accredited PT schemes, laboratories can obtain different kind of assessment since not all statistical plans are equivalent.
Is it possible that the UNI CEI EN ISO/IEC 17043 standard allows not to follow the same rules and approaches for assessing the performances of the participating laboratories? Is it possible that participants may not be aware of so? Let’s try to get some light on the subject.
IF YOU DON’T HAVE TIME TO READ NOW THE ENTIRE ARTICLE, FOLLOW US ON LINKEDIN.
NEXT WEEKS WE WILL PUBLISH DEEPER DETAILS ABOUT THE HIGHLIGHT ASPECTS.
The UNI CEI EN ISO/IEC 17043 ANNEX B, B.1.1 only indicates that the statistical methods used to analyse the PT results must be appropriate for each situation. The statistical analysis techniques that can be applied to most of the types of PT schemes described in ANNEX A are addressed in ISO 13528. ISO 13528 recognizes that other methods may be used when they are statistically valid and are described in detail for all participants. So, in case the choices are statistically valid and appropriated, they may differ from ISO 13528.
It should also be considered that often the PT providers were already using a statistical plan before the UNI CEI EN ISO/IEC 17043 standard came into force, for this reason, many of them have not yet fully adapted to the ISO environment and still use not ISO sources (e.g. IUPAC Technical Report The International Harmonized Protocol for the Proficiency Testing of Analytical Chemistry Laboratories).
One of the reasons why we choose to align completely with ISO 13528 and to stop using sources other than ISO is because the statistical approach of ISO 13528 is considered by ISO/IEC 17043 to be statistically valid and appropriate for the most types of PT schemes. Therefore, the reflection to be made is: are the laboratories informed about which other sources are used by their current proficiency testing provider in their statistical plan? How can laboratories know that these approaches are statistically valid and appropriate? Are these other sources mentioned in the ISO/IEC 17043?
Another not less important reason to align with ISO 13528 is that we believe, along with the approach of several accreditation bodies, that the direction of ISO/IEC 17043 is to prefer the use of an ISO standard, with a strive for standardization that will allow the harmonization in the assessment of laboratory performances, guaranteeing reliability of participants’ assessment.
Although other approaches may be applied by PT providers, following an ISO standard is a guarantee of quality because the used procedures are standardized, shared and easily verified by accreditation bodies in different Countries.
Two strengths of our statistical plan are undoubtedly the alignment to a single resource, exclusively ISO and considered valid by ISO/IEC 17043, as well as the transparency of our statistical approaches, with evaluation criteria always published in detail that allow participants to be aware of the statistics used in each situation.
However, we want to go deeply to what aligning our statistical plan with ISO 13528 involves and what strengths it provides:
Transparency of the statistical plan.
The ISO/IEC 17043 requires the PT provider to share the statistical plan with the participant in detail. Not all PT providers share the same level of detail on the decision-making and statistical processes used. Test Veritas has always chosen to be transparent about this and to make all information available in a clear and comprehensive manner. Our evaluation criteria are detailed and always accessible on our webpage, allowing the participant to choose us as a proficiency testing provider with awareness about how their performances will be assessed. For example, with other PT provider it isn’t always clear if a limit for the uncertainty of the assigned value is set, which is and how the evaluation of the performance is managed when the uncertainty of the assigned value is not negligible.
Single exclusively ISO source.
Our statistical plan is fully adapted to the ISO environment, providing standardization of the approach, considered valid by ISO/IEC 17043. The ISO/IEC 17043 indicates the statistical analysis techniques that can be applied to most of the types of PT schemes described in ANNEX A are addressed in ISO 13528, moreover ensures that the approaches in this standard are statistically valid and appropriate for the most types of PT schemes. When PT providers use different sources other than ISO, are the participants informed about the prove of the adequacy of the statistical approach? Moreover, whether a PT provider is proposing a new particular PT, not included in the type of PT schemes to which ISO/IEC 17043 refers, it could be justified the use of a different statistics; but how the use of statistics different from ISO 13528 can be justified for PTs that takes place from long time now and that are highly defined by ISO/IEC 17043?
Choose of the Estimator for the assigned value based on the robustness.
As indicated in the point 6.5.1 of ISO 13528, in general, robust methods should be used in preference to methods that delete results labelled as outliers. Among the robust methods stated by the ISO 13528 there are:
– the median and the scaled median absolute variation (MADe) (point 6.5.2): the median is a simple and highly outlier-resistant estimator of the population mean for symmetric distributions (ANNEX C.2.1).
-the robust means and standard deviation (point 6.5.3): Algorithm A with iterated scale, the algorithm yields robust estimates of the mean and the standard deviation of the data to which is applied (ANNEX C.3.1).
Only the value of concentration obtained from Algorithm A or from the median are chosen by Test Veritas as estimators for the assigned value.
Some PT providers, use for example the arithmetic mean as an estimator for the assigned values: the annex D 2.2. of ISO 13528 standard indicates that the arithmetic mean and its standard deviation have poor resistance to minor modes, it means that they have resistance to low proportion of outliers. Instead, the robust method using for example median and MADe can tolerate a very large proportion of outliers.
The standard uncertainty u(xpt) is calculated as: u(xpt) = 1.25 [s*/ sqrt(p)].
When the assigned value is derived as a robust average, the u(xpt) standard uncertainty of the assigned value may be estimated as u(xpt) = 1.25 [s*/sqrt(p)] (point 7.7.7). Test Veritas aligns to this requirement, while with other PTp you often may encounter a different formula, that omits the factor 1.25 or uses a lower one. This 1.25 factor contributes to the determination of the accuracy of the assigned value, by using a lower one, the standard uncertainty obtained with the same conditions will be smaller. In this way, the participant can not have the accuracy of the assigned value as required by ISO.
u(xpt)<0.3 spt used as limit for the uncertainty of assigned value.
Point 9.2.1 of ISO 13528 reports: if the standard uncertainty u(xpt) of the assigned value is large in comparison with the performance evaluation criterion (in our case spt), then there is a risk that some participants will receive action and warning signal because of inaccuracy in the determination of the assigned value, not because of any cause of participant. For this reason, the u(xpt) shall be determined and shall be reported to participants (ISO/IEC 17043:2010, 4.4.5 and 4.8.2).
If the u(xpt)<0.3 spt criterion is met, then the uncertainty of the assigned value may be considered negligible and need not to be included in the interpretation of results of the round of PT scheme.
If this criterion is not met, the PT provider should consider to use the uncertainty of the assigned value in the interpretation of the results. When u(xpt)>0.3 then the uncertainty can be taken into account by expanding the denominator of the performance scores, that will be calculated as z’-score (9.5 z’-score (point 9.5).
ISO/IEC 17043 accredited PT providers often don’t give indications about the uncertainty of the assigned value, therefore participants are not aware of the uncertainty level in their assessment. Sometimes uncertainty indications are not complete, for example it may be missing a criterion for the evaluation of the uncertainty of the assigned value. In other cases, it is chosen a more permissible criterion than the one required by ISO 13528 (e.g. u(xpt)<0.35 spt). In this last case, it may happen that the uncertainty of the assigned value should be considered by ISO 13528 as not negligible, while for the not ISO 13528 PTp it would be used as acceptable, so this high uncertainty will be not taken into account in the laboratory assessment.
In case u(xpt)>0.3, the evaluations will be calculated as z’-scores.
When the uncertainty of the assigned value is not negligible, the calculation of the evaluation takes it into account; in case the criterion u(xpt)<0.3 is not met, the z-score is converted into z’-score, by expanding the denominator with u(xpt):
z-score = (xi – xpt) / sqrt(spt) (point 9.4 of ISO 13528)
z’-score = (xi – xpt) / sqrt(spt^2 + u(xpt) ^2) (point 9.5 of ISO 13528).
Some ISO/IEC 17043 accredited PT providers are not using the z’-score or other types of z-scoring ever (e.g. En scores, z-scores), so probably they are not taking into account the higher uncertainty of the assigned value for the laboratories’ assessment. In this case, ISO 13528 requires to inform participants that the uncertainty of the assigned value is not negligible, and evaluations could be affected (point 9.2.2 of ISO 13528). Consequently, how should the participant interpretate his assessment if has been informed that the uncertainty of his assigned value is not negligible? Shouldn’t be preferred to put participants in the condition to have the evaluation in z’-score terms for example, without delegating them how to manage the not negligible uncertainty? This aspect is remarked by both ISO/IEC 17043 and ISO 13528, warns PT providers that they could release unsatisfactory assessment to the laboratories due to reasons that are not addressed to the participant itself: if the standard uncertainty u(xpt) of the assigned value is large in comparison with the performance evaluation criterion, then there is a risk that some participants will receive action and warning signal because of inaccuracy in the determination of the assigned value, not because of any cause of participant (ISO/IEC 17043:2010, 4.4.5 and 4.8.2; Point 9.2.1 of ISO 13528).
If the PT provider decides to use only a z-score calculated as (xi – xpt) /sqrt(spt) because they decide to use a different approach that allows to meet the criterion u(xpt)<0.3, this approach should be communicated to participants and the appropriateness should be demonstrated, always in the respect of transparency ISO/IEC 17043 requirement.
Homogeneity checked with the formula indicated in ISO 13528.
The criteria for sufficient homogeneity are established using the formula reported in the point B.2 of the ANNEX B of ISO 13528. The homogeneity check strictly follows the standard ISO 13528.
spt expanded in case of inhomogeneity
ISO 13528 standard recommend to take into account the inhomogeneity effects in the interpretation of the performances. We have chosen one of the proposals of point B.2.5 of ISO 13528 to use the following approach: if the criteria of sufficient homogeneity are not met, include the between sample standard deviation in the standard deviation for PT assessment, by calculating spt as follows (s’pt = sqrt (spt^2+ Ss^2)).
p>8 set as minimum number of participants.
The point 5.4.1 of ISO 13528 states that the statistical design for a proficiency testing scheme shall consider the minimum number of participants that are needed to meet the objectives of the design, and state alternative approaches that will be used if the minimum number is not achieved (ISO/IEC 17043:2010, 4.4.4.3 b). … Concerns are that statistics determined from small numbers of participant results may not be sufficiently reliable, and a participant could be evaluated against an inappropriate comparison group. Considering the indication of point 5.4.2 of 13528, the minimum number of participants need for the various statistical methods will depend on:
– the statistical method used (e.g. robust method or outlier removal)
– the experience of the participants with the particular PT scheme
– the experience of the PTp with the matrix, measurand, methods and group of participants
-whether the intent is to determine the assigned value or standard deviation (or both)
Comparing this criterion with other PT provider, we encountered different situations: often a minimum number of participants is not even established or if it is, it corresponds to a lower threshold.
Other aspects of our statistical plan that are not included in ISO 13528 but are relevant to meeting the fit-for-purposes concept of ISO/IEC 17043 are:
- Separate evaluation data for screening and confirmatory methods, with the assigned value deriving only from quantitative confirmatory results. This feature is applied to our PT schemes for the lines of Veterinary Drug Residues, Mycotoxins, Freshness Parameters, Processing Contaminants where the screening methods are widely used, so the distinction between the screening and confirmatory PT results is necessary. This criterion makes the assigned value more reliable and the assessment more adequate. Even if you are using a screening method, the PT simulates what you would do in the routine (comparing your results with confirmatory ones). Are the participants of a PT aware of the number of screening results that contributes to calculate the assigned value? This percentage is known by participants? Is it reliable to assess confirmatory methods results on the basis of an assigned value obtained with 20% or more of screening results? Shouldn’t be preferred an assigned value that completely exclude screening results?
- Dedicated evaluation criteria for not quantitative screening results, while the quantitative screening and confirmatory results are evaluated through z-score. This feature is applied to our PT schemes for the lines of Veterinary Drug Residues, Mycotoxins, Freshness Parameters.
- Assessment for False Positives. When there are several molecules under study, an evaluation in terms of False Positives for the not present molecules is released. Moreover, in some of our PT schemes 2 different Test Materials are provided (they can be both contaminated or one of them could be not contaminated). In the PT, as in the routine, the laboratory technicians do not know if they are testing a contaminated or a blank material. In this way, the laboratory could be assessed also for False Positive results or for method performances at different levels of concentrations.
What about your current PT providers? Do you have the details about the statistical plan used? For all the above-listed points, do you know which are their approaches? If they use an approach different from ISO 13528 standard, how can you be sure that it is statistically valid and appropriate as required by ISO/IEC 17043?
All the above-mentioned strengths should be a reason why to choose us as your proficiency testing provider, come and try our quality.
IF YOU LIKED THIS ARTICLE, FOLLOW US ON LINKEDIN.
NEXT WEEKS WE WILL PUBLISH DEEPER DETAILS ABOUT THE HIGHLIGHT ASPECTS.
If you want to see more details about our PTs, such as the schedule, prices… go to this page to download the entire programme.
Here you can find information on HOW TO ORDER.
For any further information, do not hesitate to contact us at info@testveritas.com.
Follow us on LinkedIn to get the last update! Stay tuned.
* For the lines of Progetto Trieste of Veterinary Drug Residues, Freshness Parameters, Pesticides, Metals and Additives the new evaluation criteria are already available on the website. For Mycotoxins and Processing Contaminants it is still online the old version (April 2022), but it will be updated soon.