Comments (1)

By Lynn McKee, April 8, 2011 1:25 pm

This year I decided to tackle presenting the dangers inherit with metrics and measurements. The topic has been well covered in books, articles, and blogs and I was fortunate to have such great material to learn from. I am passionate about encouraging people to understand the many ways we can be deceived.

I cover many concepts in my presentation including:

• What is Measurement?
• Why do we Measure?
• Measurement & Context
• The Qualitative vs Quantitative Debate
• Deductive & Inductive Reasoning
• Entomological & Ontological Assumptions
• The Positivism Viewpoint & Empiricism
• The Post Positivism Viewpoint Including Reality, Observations & Perceptions
• Orders of Measurement
• Measurement Models and Validity
• Construct Validity
• Abstraction & Aggregation
• Measurement Side Effects
• Inquiry vs Control

The emphasis is on asking “What is the question we are really trying to answer?”. It is important to understand that measurement is difficult as many things we seek to measure are subjective: complex, qualitative, non-repeatable, and involve human judgment or human performance. This is further complicated when we develop complex, scientific models in our attempt to measure more accurately. We need to question the validity of our models and understand concepts such as construct validity, abstraction and aggregation.

We must also understand that measurements do not have validity, only our inferences or conclusions can have validity. Be mindful of the notion of truth and the need to be critical of our ability to know reality with certainty. This combines the affects of epistemological and ontological assumptions, causal inference, confirmation bias and many more factors that result in ill-informed conclusions. Combine qualitative and quantitative measures and triangulate to get a better handle on reality.

Managers need to think carefully about the potential side effects of measures. People tailor their behaviour to things that they are measured against and behaviours change in predictable ways to provide the answers management is looking for. Beware of these effects. Better yet…stop measuring testers in terms of test case production and bug counts!

It is imperative that metrics and measurement are used for inquiry vs control. Remember to use numbers to illustrate stories; watch out for numbers becoming placeholders for stories. Stop providing metrics and measurements solely in the written form, delivered by email, with the hope your stakeholders are drawing meaningful conclusions from them. Establish regular opportunities to have conversations with your stakeholders about your testing efforts and the value of your team within the organization. Educate your stakeholders on the importance of discussion to provide the rich descriptive detail that sets quantitative results into their human context.

I am hopeful you will check out my presentation Deception Dangers of the Numbers Game. I encourage you to research these dangers and strengthen your understanding of how we deceive ourselves. At the end of my presentation you will find three pages of great references to use as a starting point. I also welcome the chance to chat on Skype @lmmckee or Twitter @lynn_mckee.