Access Type

Open Access Dissertation

Date of Award

January 2014

Degree Type

Dissertation

Degree Name

Ph.D.

Department

Instructional Technology

First Advisor

Ingrid J. Guerra-López

Abstract

Background: In performance improvement, intervention selection is a complex decision that ought to be based on the best available evidence. Despite this, there is little research into what sources of evidence are used in intervention selection and what changes in belief occur during performance improvement professionals' decisions. Framed in decision theory, this study aims to resolve these problems. Methods: Sixty-one Certified Performance Technologists completed a dynamic, Web-delivered questionnaire where they provided a general assessment of intervention success (Pr1), then responded to 12 performance improvement scenarios; by selecting an intervention, providing a prior probability (Pr2), receiving additional evidence, giving a posterior probability (Pr3), indicating whether the initial intervention was still preferred and making a subsequent choice if not. Results: Repeated measures ANOVA showed significant interaction between time and evidential agreement for probability assessments (p <.001). No effects were shown for scientific nature of evidence. Informed Bayesian analyses showed only main effects (evidential agreement and time). Factorial MANOVA found significant effects for evidential agreement on Pr3 and changes between prior and posterior probability (l) (p<.001). Marginally mixed effects were noted for scientific nature of evidence. Normal (Z) approximation revealed subjects tended to stick with their initial intervention choice (p<.0001) and only scientific evidence was associated with this action (r s=-0.3160, p<.0001); informed Bayesian analyses revealed contrary findings. Binary logistic regression illustrated Pr3(OR=1.085) and l (OR=3.792) are good models for changes of mind (p<.0001, Max-rescaled R 2=.75). When subjects did change their minds, no differences in self-reported familiarity on initial interventions existed (p = 0.085), but familiarity was significantly lower for subsequently preferred interventions (p = 0.003). Post hoc paired t-tests showed higher levels of familiarity with selected interventions than their non-selected counterparts (p <.0001). No significant correlations occurred between familiarity and Pr3, four analyses yielded correlations for general and prior assessments of likely interventions success. Informed Bayesian analyses illustrated dramatically different results, specifically, 15 of the 18 correlational analyses between self-reported familiarity and assessments of likely intervention success were significant. Repeated measures ANOVA showed no significant effects of practice on the repeated probability measures (p = 0.806) and post hoc ANOVA showed that randomized blocks were similar (p <.0001) and no differences between them (p =.201, p =.604, p =.072). Discussion: These findings bolster the long-standing concern about the technical nature of performance improvement and practitioners are strongly encouraged to approach intervention selection as a decision, where their intervention preferences, beliefs of likely success are carefully adjudicated on the basis of the evidence they obtain. Future research with other types of performance improvement practitioners, replication studies, longitudinal, structural equation modeling, externally verifiable probabilities, and natural environments are recommended.

Download

Share

COinS