How well do you manage measurement systems? Find out with this interactive self-assessment.
This self assessment is roughly compatible to the guide "Measurement Systems Analysis", 3rd edition, produced under the auspices of the Automotive Industry Action Group and the American Society for Quality
Scroll to bottom for results.
| Selection of instruments | Have you ascertained the instruments you use on key parameters provide the best characterization of the attributes you wish to measure? | Field=sel1 0: Not at all 1: In some cases 2: In most cases |
| Before you acquire new instruments, do you determine the required resolution and repeatability specifications (measurement error) prior to acquisition? | Field=sel2 0: Not at all 1: In some cases 2: In most cases |
|
| After you acquire the instrument, do you conduct tests to determine whether the instruments meet these specifications before putting them into use? | Field=sel3 0: Not at all 1: In some cases 2: In most cases |
|
| Have you ascertained that the instruments you use on key parameters have a sufficient resolution to detect process or product variations that may be significant? | Field=sel4 0: Not at all 1: In some cases 2: In most cases |
|
| When you upgrade instruments or purchase new instruments, do you perform regression studies on the new and old instruments do determine whether the difference in the way the two instruments measure? | Field=sel5 0: Not at all 1: In some cases 2: In most cases |
|
| Do you update your process control and quality control procedures according to the difference in the two instruments? | Field=sel6 0: Not at all 1: In some cases 2: In most cases |
|
| Do you read the instructions that come with the instruments and develop your own measurement, instrument testing and calibration procedures based on these instructions? | Field=sel7 0: Not at all 1: In some cases 2: In most cases |
|
| Your Partial score for Instrument selection | Subtotal percent | |
| Instrument Repeatability | Have you determined the repeatability of the instruments? (this means that you know the standard deviation of the measurement device when used by a single appraiser to measure the same characteristic on the same part). | Field=repeat1 Fail: No 1: In a few cases 2: In most cases 3: For all instruments |
| Have you established a criterion for determining whether the instrument standard deviations are acceptable? (for example, Acceptable if s.d. < one scale unit). | Field=repeat2 0: Not at all Fail: Do not know s.d. 1: No, evaluate informally 3: Yes |
|
| Have you verified that the instruments meet the criterion? | Field=repeat3 0: No 1: On a case by case basis 1: In some cases 2: In most cases 3: Yes |
|
| In situations where they do not meet the criteria do your measurement procedures call for repeated measurements, and the use the mean of these measurements, to increase measurement precision? | Field=repeat4 0: No 1: In some cases 2: In most cases 3: Yes |
|
| Partial score for Instrument repeatability | Subtotal percent | |
| Measurement Methods | Have you performed statistical studies to determine whether differences in methods for sampling, preparation and measuring may contribute to measurement variation? (e.g. differences in collection, preparation, handling or storage of samples or use of measuring device). | Field=im2 0: No 1: We do some informal tests 2: Some statistical studies 3: Completed statistical studies |
| Have you modified your methodologies as a result of these studies | Field=im3 NA: NA 0: No 1: In some cases 2: In most cases 3: Thoroughly |
|
| Do you perform audits or inspections to determine whether the appraisers continue to use the methodologies you have developed? | Field=im4 0: No 1: Some informal checks 2: Some formal checks 3: Carefully verified |
|
| Have you verified that users of the instruments can read the scales properly (e.g. reading without parallax errors, reading the correct scale on an instrument with multiple scales, not rounding up or down) | Field=im1 0: No 1: Some informal tests 2: Some formal tests 3: Yes, they can |
|
| Partial score for Measurement methods | Subtotal percent | |
| Reproducibility and stability | Have you determined the sources of variation in the measurement devices and in measurement methods that may affect reproducibility and stability (e.g. appraisers, instruments, wear, environmental variables etc.)? | Field=repro1 0: No 1: In some cases 2: In most cases 3: Thoroughly |
| Have you conducted a reproducibility study to estimate the contribution to variability of each of these sources? | Field=repro2 0: No 1: In some cases 2: In most cases 3: Thoroughly |
|
| When there was more than one source did you use Design of Experiments or related techniques to determine whether these sources contributed a statistically significant amount of variation? | Field=repro3 0: No 1: In some cases 2: In most cases 3: Thoroughly |
|
| Have you developed controls (e.g. procedures, climate control, etc.) for these sources of variation to ensure the stability of measurement systems and methods? | Field=repro4 0: No 1: In some cases 2: In most cases 3: Thoroughly |
|
| Do you regularly conduct tests or studies on measurement systems to determine whether they are in statistical control? | Field=repro5 0: No 1: In some cases 2: In most cases 3: Thoroughly |
|
| Partial score for Reproducibility and stability | Subtotal percent | |
| Linearity and bias | Have you determined that the instruments and measurement methods are unbiased over the full range of measurements they encounter in production operations? | Field=lin1 0: No 1: In some cases 2: In most cases 3: Thoroughly |
| Do you use traceable standards when you determine linearity and bias? | Field=lin2 0: No 1: In some cases 2: In most cases 3: Thoroughly |
|
| Partial score for Linearity and bias | Subtotal percent | |
| Product testing | Do your product testing (e.g. QC testing) procedures stipulate that appraisers should take a second measurement if the first one indicates the product may be out of specification, before rejecting the product? | Field=test1 0: Yes, in general 1: In several cases 2: In a few cases 3: No |
| Do you have internal criteria that define the acceptable measurement system variability compared to the product specification tolerance? | Field=test2 0: Not at all Fail: S.D. unknown 1: No, evaluate case by case 3: Yes |
|
| Have you tested your instruments to determine whether they conform to these criteria? | Field=test3 0: No 1: In some cases 2: In most cases 3: Yes, in general |
|
| Do your instruments conform to these internal criteria? | Field=test4 0: No 1: In some cases 2: In most cases 3: Yes, in general 4: Do not know |
|
| Partial score for product testing | Subtotal percent | |
| Process Control | Do you have internal criteria that define the acceptable measurement system variability compared to process variation or process specifications? | Field=pc1 0: Not at all Fail: Do not know s.d 1: No, evaluate case by case 3: Yes |
| Have you tested your instruments to determine whether they conform to these criteria? | Field=pc2 Fail: No 1: In some cases 2: In most cases 3: Yes |
|
| Do your instruments conform to these internal criteria? | Field=pc3 0: No 1: In some cases 2: In most cases 3: Yes, in general 4: Do not know |
|
| Partial score for process control | Subtotal percent | |
| Calibration | Do you regularly conduct scheduled calibrations? | Field=cal1 Fail: No 1: Do them sporadically 2: Fairly regularly 3: Yes |
| Do you have documented calibration procedures for instruments on key parameters? | Field=cal2 0: No 1: In some cases 2: In most cases 3: Yes, in general |
|
| Do your calibration procedures use statistical techniques (control charts, t-tests etc.) to determine whether the instrument has become biased or unstable before re-calibrating? (The incorrect method is to recalibrate based on a single measurement assuming this indicates bias, without regard to measurement error (repeatability)). | Field=cal3 0: No 1: In some cases 2: In most cases 3: Yes, in general |
|
| Do you maintain detailed and legible records of when (a) instruments were tested, and (b) when these tests resulted in performance of calibrations? | Field=cal4 0: No 1: In some cases 2: In most cases 3: Yes, in general |
|
| Do you use traceable standards for calibration, where such standards exist? | Field=cal5 0: No 1: In some cases 2: In most cases 3: Yes, in general NA: Not applicable |
|
| Do you use an internal standard, when a traceable standard does not exist? | Field=cal6 0: No 1: In some cases 2: In most cases 3: Yes, in general NA: Not applicable |
|
| Partial score for calibration | Subtotal percent | |
| Sampling error | Do you report the sampling error that occurs when samples are taken from production lines, as an overall part of measurement error? | Field=samer1 0: No 1: In some cases 2: In most cases 3: Yes, in general NA: Not applicable |
| Have you defined methods for determining sampling error? | Field=samer2 0: No 1: In some cases 2: In most cases 3: Yes, in general NA: Not applicable |
|
| Partial score for sampling error | Subtotal percent | |
Score |
Your result for measurement | Total percent |
Interpreting your score:
?sel2=0: Instrument Selection. We suggest you work with instrumentation/Q.C./Q.A./engineering/purchasing staff to develop specifications for the acceptable measurement error for instruments you acquire.
?sel3=0: Instrument Selection. Suppliers of instruments some times have strange ways of defining measurement system variation that may not correspond with your own. It is essential to conduct tests to verify that instruments meet your requirements before you put them to use.
?repeat1=fail OR repeat2=fail: Repeatability. If you do not know the standard deviation of the instruments that measure your key parameters, you cannot have any confidence in the measurements. This is a serious lack of information. Serious consequences may result. For example, you may reject perfectly good product, or you may over-control your processes, wasting time and money. Your score for the entire self-assessment is zero as a result. We suggest you initiate the required statistical studies immediately.
?repeat2=0: Repeatability. Without any sort of criteria to determine whether the standard deviation of an instrument is acceptable, you have no way of determining objectively throughout your organization whether the instruments you use are capable of doing the job they are supposed to do. We suggest you establish some criteria in conjunction with Q.C./Q.A., instrumentation and engineering staff.
?pc1=fail: Process control. You answered in the process control section that you did not know the standard deviation of your measurement system (s). You can not then use your measurement system for process control. Your score for the entire self-assessment is zero as a result. We suggest you initiate the required statistical studies immediately.
?im1=0: Instrument methodologies. it is surprising how people, even well-trained people, can have great difficulties reading instruments properly. It is certainly worth developing some kind of verification process to determine that users of the instruments can read the scales properly (e.g. reading without parallax errors, reading the correct scale on an instrument with multiple scales, not rounding up or down)
?im2=0: Instrument methodologies. Most people develop small differences in the methods they use with instruments, even though they are following procedure. It is worth performing statistical studies to determine whether differences in methodologies may contribute to measurement variation so that you can develop the best methodology and apply it uniformly (e.g. differences in collection, preparation, handling or storage of samples or use of measuring device).
? repro2=fail: Reproducibility and stability. If you do not know the extent to which your instruments and measurement processes are reproducible you cannot have any confidence in the measurements. This is a serious lack of information. Serious consequences may result. For example, you may reject perfectly good product, or you may over-control your processes, wasting time and money. Your score for the entire self-assessment is zero as a result. We suggest you initiate the required statistical studies immediately.
?lin1=0: Linearity and bias. If you have not determined whether the instruments are unbiased over the full range of measurements they encounter in production operations you may have serious inaccuracies in your measurement data. We suggest that you conduct the necessary bias and linearity studies.
?test1=0: Product testing. Usually when people proscribe the taking of a second measurement on a test if the first one shows an out-of-spec condition it is because they do not have confidence in their instruments. If there is no confidence in the instrument, why would the second measurement be any better? A two-measurement procedure decreases the probability or rejecting non-conforming product because the product must fail two consecutive measurements (thus you are more likely to ship non-conforming product). It is better to improve your measurement systems than to use such procedures.
?test2=0: Product testing. If you do not have internal criteria that define the acceptable measurement system variability compared to the product specification tolerance, you do not have a reasonable and consistent way to ensure that the instruments you use are giving an sufficiently accurate measure of product quality. You thus lack reliable knowledge about your product quality, with all the customer and financial implications.
?pc1=0: Process Control. If you do not have internal criteria that define the acceptable measurement system variability compared to process variation, you do not know, uniformly across the organization, how much of the process variation you observe is caused by the process and how much is caused by the measurement process. You therefore risk over-adjusting your processes, under the belief that they are varying excessively, when in fact they may be stable and the measurement process is what is causing the variations.
?cal1=fail: Calibration. If you do not conduct regular calibrations it is likely that your instruments are not giving you accurate information. Your score for the entire self-assessment is zero as a result. We strongly suggest that you initiate a calibration program based on sound statistical principles.
?cal3=0: Calibration If you do not use a statistical procedure to determine when an instrument must be re-calibrated you may re-calibrate it unnecessarily. This is the same principle as using control charts for process control. Do not adjust the process (measurement or production) unless it is out of control. By re-calibrating unnecessarily you are probably increasing, not decreasing measurement system variation.
?samer1=0 OR samer2=0: Sampling error. If you do not not know the sampling error of your measurement process you will under-estimate the total error. There is sample-to-sample variation in production lines (whether work-in-process or final product). If your measurement process is meant to make inferences about the characteristics of the samples in a production line, you must know the measurement error and the sampling error.
?total>0 and total < 40: Overall score. You have a very ineffective method of managing your measurement system. This is probably costing you significant amounts of money in firefighting, products rejected when they should not be, poor quality products sent to customers, inadequate process control, etc.
?total>41 and total < 60: Overall score. There are some strengths in your measurement system but there is still considerable room for improvement. You have a somewhat effective method of managing your measurement system. There are still probably many opportunities to save money incurred in firefighting, products rejected when they should not be, poor quality products sent to customers, inadequate process control, etc. as a result of your measurement system.
?total>61 and total<90: Overall score. There are many strengths in your measurement system but there is still room for improvement. You have an effective method of managing your measurement system. There are still probably a few opportunities to save money incurred in firefighting, products rejected when they should not be, poor quality products sent to customers, inadequate process control, etc. as a result of your measurement system.
?total>90: Overall score. Congratulations. You have a top-notch method of managing and analyzing your measurement system.