Gwets AC1 is the statistic of choice for two evaluators (Gwet, 2008). The Gwetsche compliance coefficient can be used in more contexts than kappa or pi, as it does not depend on the assumption of independence between evaluators. κ = (observed agreement [Po] – expected agreement [Pe])/(1-expected agreement [Pe]). For ordinal data where there are more than two categories, it is useful to know whether the ratings of the different evaluators varied by a small degree or a large amount. For example, microbiologists may assess bacterial growth on culture plates as zero, occasional, moderate, or confluent. Here, evaluations of a particular plate by two evaluators as “occasional” or “moderate” would imply a lower degree of deviation than if these scores were “growth-free” or “confluent”. Kappa`s weighted statistics take this difference into account. This results in a higher value if respondents` responses match more closely, with the maximum scores for a perfect match; Conversely, a larger difference between two odds results in a lower weighted kappa value. The techniques for assigning weighting to the difference between categories (linear, square) may vary. Weighted kappa partially compensates for a problem with unweighted kappa, namely that it is not adjusted for the degree of disagreement. Disagreements are weighted in decreasing priority from the upper left (origin) corner of the table. StatsDirect uses the following definitions for weighting (1 is the default): at this point, the attribute agreement assessment should be applied and the detailed audit results should provide a good set of information to understand how best to design the assessment. The statistic κ can take values from − 1 to 1 and is interpreted somewhat arbitrarily as follows: 0 = correspondence which corresponds to chance; 0.10–0.20 = slight chord; 0.21–0.40 = fair agreement; 0.41–0.60 = moderate chord; 0.61–0.80 = essential agreement; 0.81–0.99 = near-perfect match; and 1.00 = perfect chord.

Negative values indicate that the observed match is worse than might be expected by chance. Another interpretation is that kappa levels below 0.60 indicate a significant level of disagreement. Scatter plot showing the correlation between hemoglobin measurements from two methods for the data presented in Table 3 and Figure 1. The dotted line is a trend line (least squares line) through the observed values, and the correlation coefficient is 0.98. However, the individual points are far from the perfect chord line (solid black line) Often one is interested in whether the measurements taken by two (sometimes more than two) different observers or by two different techniques lead to similar results. This is called agreement or concordance or reproducibility between measures. Such an analysis examines pairs of measurements, categorically or numerically, each pair being performed on an individual (or a pathology slide or an X-ray). Cohen`s kappa (κ) calculates the interobserver agreement taking into account the expected random agreement as follows: In this example, an evaluation of repeatability is used to illustrate the idea, and it also applies to reproducibility. The point here is that many samples are needed to detect differences in an attribute matching analysis, and when the number of samples is doubled from 50 to 100, the test does not become much more sensitive. Of course, the difference that needs to be recognized depends on the situation and the level of risk that the analyst is willing to bear in the decision, but the reality is that with 50 scenarios, an analyst can hardly assume that there is a statistical difference in the repeatability of two reviewers with compliance rates of 96% and 86%.

With 100 scenarios, the analyst will barely be able to tell the difference between 96% and 88%. If the audit is planned and designed effectively, there may be enough information on the causes of accuracy issues to justify a decision not to use the analysis of award agreements at all. In cases where the audit does not provide sufficient information, the analysis of attribute agreements allows for a more detailed investigation that provides information on how to provide training and fail-safe modifications to the measurement system. Disagreement on a category and asymmetry of disagreement (2 reviewers) As stated above, correlation is not synonymous with agreement. Correlation refers to the presence of a relationship between two different variables, while agreement examines the concordance between two measures of a variable. .