Making sense of business research

A lot of the research presented to us by the gurus, self-proclaimed or otherwise, is anecdotal in nature. Not that examples aren't useful to illustrate and clarify a point of view. They are. But, really, you can prove anything with anecdotes. Even if you attempt to apply some more rigorous research principles and methdologies to your study, all may not be what it seems. Let me explain.

Winners and losers

The business world represents a competitive and ever changing arena. Some company stocks are on their way up, some are on their way down. The winner one day could well be a loser the next, depending how you define "winning". In addition, stock values can fluctuate significantly in the short term while still displaying a different directional trend over the longer term. So what is the relevant time frame? The upshot is, the rules are not equivocal. They are made up. There is no "natural science of management". There are no immutable principles that define the "right" or "natural" way to run a business. It still more like an art. Or maybe not. We can't say. One thing is clear to me, though. You need to understand the context of the business being researched to decide if you believe the rules that are presented to you.

Any evaluator of business performance either has to define his own set of rules or find some created by others. Sometimes the latter is the case. Perhaps the business did it; for example in their strategy or in public statements. Or maybe the author is out to prove a point, to support his theory or whatever. Then he is likely to make up his own rules. That's like redefining the game, in a way. Did the players have a say in this? Probably not. This doesn't invalidate what the researcher is doing, but should be kept in mind when you evaluate his results.

At any given time, in any given industry

A business invariably changes over time. No-one can be a thought leader at all times. Someone better take time out for execution once in a while. So if we go looking, we'll find that at any given time in any given industry, some companies are crafting new strategies while others are busy executing theirs. Who will look the most "with it" if you shine the spotlight of the moment on these players? The crafters, not the executers. Does that make the executers bad? Not necessarily.

I also think you'll find that the same corporations are trotted out again and again as either heroes or villains. If you look closer, you'll also notice that these same - invariably large - corporations are taken as evidence of a successful application of almost any new hot management concept, theory or outright fad. So what is going on? Well, whatever else these corporations may be, they aren't usually stupid. They keep trying stuff. And they aren't homogenous organizations either. Different parts of the organizations are doing different things. There's more.

In any given country

Think about it. General Motors manages more money than some national economies. So what goes on in any given country? Pretty much anything you can think of. In principle, a large corporation is not much different. The bottom line here is that these biggies became biggies and stay that way because - like it or not - they have become pretty astute at executing whatever they set out to do. Don't you think it reasonable to assume that there is too much at stake to let a feeble concept or weak theory get in the way of meeting a budget, making a bonus or advancing a career? Of course it is. Stated differently, a feeble concept or weak theory will probably "succeed" despite itself. The stakeholders will certainly take the necessary steps to rescue a budget or plan and not worry too much about what label is stuck on the end result.

A messy laboratory

Did we shoot ourselves in the foot here? First we basically say that research is desirable to prove a point and then conclude that a real life business doesn't seem like being a very reliable laboratory. Like it or not, real life makes for a messy laboratory because we cannot change our variables one at a time while holding everything else constant.

What to trust and how

So, does this mean that all business research is just totally hopeless and not to be trusted? Of course not. That would be unfair and a gross over simplification. We have to approach the subject with a healthy dose of skepticism, though. Here's how I try to make sense of the research I come across.

Peer reviews

Scientists typically rely only on research published in peer reviewed journals. A peer reviewed journal has a panel of bona fide experts in the authors' fields of study who review the submitted papers More than one reviewer will review any one submitted paper. The wrinkle is that the authors are anonymous to the reviewers. While not perfect, this process ensures that a certain level quality control and scientific rigor is brought to bear on the research and its conclusions.

I am not suggesting that non-peer reviewed studies are bad. I am merely pointing out a distinction you should be aware of. For example, a news story on a remarkable new study, may be all you need. Then again, the newspapers often get it wrong. Going to the source is always best, but can be painful. The truth is that the peer reviewed journals may be all but incomprehensible to a lay person. Scientific writing can be outright turgid and the academic requirements hardly assist readability either. This opens up a role for editors and synthesizers. Important information would risk being forever hidden behind ivory tower ramparts unless someone sat down and made it legible for the rest of us.

In either case, please consider the source of the information you have in front of you.

Paradigms and mental models

I usually want to have an idea about what paradigm or mental model the guru/author/researcher is using. I firmly believe it is totally futile to try to understand anything if you don't have a theory, concept, paradigm, mental model or something like that to provide a context for your speculations.

Sometimes this is stated very clearly, sometimes it is not. Sometimes it is missing, other times you have to dig around for it.

Metaphors and examples

Metaphors and examples often give the game away. Personally, I tend to tune out at professional sports metaphors applied to business management. Why? Well, business may be a game to you, but even though I don't watch much American football, I am pretty sure that among all the line backers in the National Football League, there's not one single grandmother. I'm equally sure that more often than not, in any given business with a payroll the size of a professional football team, we should confidently expect to find at least one. (grandmother that is, not line backer.) So please don't waste my time with third downs on the assembly line and 100 yard rushes in Accounts Payable.

Fit the play to the players

If you insist on using the sports metaphor, at least try to fit the play to the players on your team and don't base it on the players on some other team in an entirely different game.

What's it all for anyway?

OK. Context well in hand, the next question is this: what is this research for? Is the researcher trying to develop a new theory? Is he applying an old theory in a new or old context? Or is he trying to explain or predict something? Perhaps the most critical of these is the explanation versus prediction issue. The two often get mixed up.

Explanation versus prediction

Most social science theories are good for either of two things: explanation or prediction. Seldom both. That's not necessarily a bad thing. We just need to know which beast is in front of us and act accordingly.

Let's look at a simple example. Knowing the biological mechanisms behind conception and birth (an explanatory model) does not by itself allow us to tell with much certainty when the next birth will occur (a predictive model). We'd need a whole set of different information to predict. And would you agree that we can easily make a predictive model to tell quite accurately when the next birth will occur without knowing anything at all about the biology of conception and birth itself? Certainly looks that way. Otherwise how could life have evolved at all?

Which is easier?

It also looks like it might be a lot easier to predict something than to explain it. But there's more. What we have here is a classic case of two variables, each with two states and a total of four possible outcomes. The variable states are: Valid prediction, invalid prediction, valid explanation and invalid explanation. The outcomes are the four possible combinations of these variable states taken two at a time.

Here's an example. Let's assume you are making gravy. For this you need a recipe. Depending upon how good the recipe is and how it is executed, you can explain why the gravy is good or bad. The recipe also has some predictive power for the outcome (or so we hope). We can put this in a table to make it clearer.

Hypothesis: a point to prove or disprove

Lets return to our review. Look at the formulation of the basic hypothesis or point the researcher is trying to prove. This is usually fairly straight forward to nail down. Watch out if it's not. Furthermore, is it reasonable to expect that the point can be proven? Here we're on thinner ice. Turn the researcher's problem around. Can his point be disproved? What would we learn if he were to be proven wrong? We shouldn't be asked to take too big a leap of faith here. Otherwise the local psychic hotline may be faster, cheaper and more entertaining for the same useless result.

Expect the predictable

Most published research produced the expected results. (Or at least it seems that way from what we get to read.) There is obviously little guru-mileage in announcing that things didn't go as you expected. Still, that's a poor excuse for formulating only provable hypotheses. It's all too easy to overlook how alternative explanations may account for the observed results.

Data collection

And we're not done yet. Let's ask how the data was collected. No need to get into the fine art of random sampling and statistical mechanics. Does the procedure seem reasonable? For example, did the researcher do a survey? Yes? Good. Who did he survey? How many responses did he collect? How many people did he ask in the first place? Do we believe the respondents are representative of the general population of whatever the researcher says he is interested in?

If he asked 1000 people and 50 replied, it's anybody's guess what we can really know about the general population or even just the sampled group. If 500 answered, you're probably a whole lot more ready to accept that you can extrapolate from the results to the general population being researched. But get this: If the researcher set out to study bigotry and selected his respondents solely from known bigots, getting 500 answers from a 1000 person survey gave him zero information about the other 5 billion people on this planet.

Sample size and the power of one

Besides worrying about how representative the respondents are, the researcher must consider his sample size. I promised I wouldn't get too deeply into this, but you should at least be aware of a couple of things.

Under the right experimental conditions a small sample (e.g., less than 100) can be valid and allow generalization to the total population. The key is "right experimental conditions". We have already suggested that a business is a messy laboratory so you need to watch this one for a really, really convincing research scenario.

Opinion polls normally operate with respondent numbers in the low thousands and publish results claimed to contain a margin of error of only a few percentage points. Thus, the sample size need not be enormous for the researcher to learn something conclusive and statistically valid about the general population. A statistically valid result can still be quite useless. But I think you already knew that.

An anecdote is a sample of one. It is one observation. From a statistical point of view, a sample of one tells you nothing about the population you sampled from. Other than the fact that you observed an occurrence of something that may in itself be significant to the observer. For example, observing a customer buying your product tells you nothing about the market. You need more information from more observations. It is probably significant to you if the observation included whether the customer smiled or threw the product back at you in disgust.

An anecdote may support an argument, but is hardly conclusive evidence of anything but the occurrence of the described events. Even that is sometimes disputed, though. Just ask any three spectators about what happened right in front of them. You'll likely get three different stories. I think the best way to look at this is that an anecdote is an example and can be very useful to clarify a point or illustrate a position. However, an anecdote is not necessarily "evidence". If a given occurrence can have more than one explanation, anecdotes can be used to "prove" or "disprove" anything a researcher wants. Hardly a comfortable thought for those who are asked to put money down on these same conclusions.

Consider the alternatives

Now let's try a different tack. What other way could the researcher find out what he wants to know? The answer may surprise you. Some researchers move around in wide circles about the subject when a simple question to the people closest to the real issue would suffice.

I want to extend this a little further. Given what the researcher has told us about what he did and how much we believe of all this, are we confident that his conclusions follow from his premises and the evidence he presents us? Are we asked to take a leap of faith? What I am getting at here is the distinction between necessary or sufficient conditions to produce an outcome. Let me define our terms before we go on.

Necessary conditions

A necessary condition is one that must always be present to produce the outcome in question. There could be more than one necessary condition required for any given outcome.

Sufficient conditions

A sufficient condition is a condition that will produce the outcome in question. It may not be necessary for all events producing the outcome. It's just that if a sufficient condition occurs, you may get the outcome. Sounds a little like "the exception that proves the rule".

Necessary and sufficient conditions

These are conditions that, when they occur, will invariably produce the outcome irrespective of what ever else occurs.

Conditions and their consequences

Now back to our discourse. Consider the conclusions you are asked to accept. If the researcher hasn't defined his model properly and tested for all the right conditions, we could be in trouble with our conclusions. Has the researcher presented you with conditions that are necessary for the observed results to occur? Yes? Good. Now, let's ask this: Were they sufficient for the observed results to occur? It doesn't take a rocket scientist to see how unclear thinking about conditions can produce some strange conclusions that could get the unwary in a lot of trouble. With this little tidbit of insight I predict you will have a whole lot more fun reading research. You'll be amused, surprised and chagrined at what you will find.

More on outcomes

The case we are building is that outcomes follow from conditions. Once again we have two variables, each with two possible states. Explicitly, the variable states for conditions are: sufficient, not sufficient, necessary and not necessary. This gives us four possible outcomes and another table.

No guarantees

OK. So we go through all this smart questioning. Will this guarantee that you will always be able to sort out the good stuff from the bad? Well, hardly. Social science research being what it is, you just can't get the precision and confidence levels you'd expect from research in many other areas. Sorry, that's just the way it is.

Judgment

Other people's research and opinions can help you learn, but it can only take you so far. An opinion is just an opinion no matter how well qualified the opinion giver. There really is no substitute for exercising good judgment. And that goes for both the researcher and the reader. To put it bluntly, "you pays your money and you takes our chances".

Warning signals of suspect research

Be suspicious of research that:

1. Gives recommendations based on studies published without peer review.
2. Offers recommendations from a single study.
3. Draws simplistic conclusions from a complex study.
4. Offers trivial or quick fix solutions to complex problems.
5. Makes claims that sound too good to be true.
6. Makes dramatic statements that are refuted by reputable research or scientific organizations.
7. Makes partisan statements like a) badmouthing of single products, individuals, companies, management practices or b) recommendations to help sell a product or service.
8. Gives recommendations from studies that ignore differences among individuals.

Post script

As a closing note, I'll share with you that I'm writing this from a hotel room in Toronto. Just now I put my breakfast tray outside the door. On the tray was a glass of water, untouched, with a loose fitting paper lid on it. As I lowered the tray to the floor, I tilted it past the critical angle and the water glass flipped off the tray and landed upside down with the cap on. And stayed there without the water running out. Go figure the odds on that one.

I conclude from this extraordinary and observed fact that the big R&D god in the sky must like this piece. Thus he smiled upon me this morning to signify his agreement with my most excellent arguments and knighted me into guru-hood by sharing with us this visible sign of his absolute powers over the utterly improbable. As an addition to the random luster, the high and mighty certainly chose a most unlikely chrysalis for this long overdue event.

I leave it as an exercise for the reader to verify why this is so. I've given you all the tools...

Home | BCVSP | Coaching Voyage | Musings | Recreation | Runes | Sailing | Stories | Viking Ships | About this site | Contact me | Links