At a recent A1 event I was asked about how academia can help Analytics. At another recent event, a discussion with a recruiting professional focused on the criteria employers in HR departments use in filtering and selecting Analytics candidates.
Analytics education is a hot topic. A growing, multidisciplinary and highly complex field is experiencing a shortage of suitably qualified people, and this is becoming a matter of concern for business.
There is a growing number of data mining/Analytics subjects, majors and even courses. I have been asked by a number of people in recent weeks what I think of particular courses, or what individuals should do to prepare themselves for a career in Business Analytics.
My opinion of the value of existing courses is such: the number of suitable people will increase as a result of these courses. But not by much.
As it stands, there are serious problems with what passes for “data mining” training, particularly at undergraduate level, particularly in computer science.
I single out computer science because it produces seemingly suitable candidates who are hired by HR, and fail in fundamental, but not immediately obvious ways. I also speak as an “insider”, with my entire academic training based in computer science. As with all useful generalizations, there are quite a few exceptions, and the problem outlined lives on a continuum of pathology, with a minority of extreme cases and many more less severe ones. Nevertheless, there is a real and consistent problem with computer science undergraduates (and many postgraduates!) moving into a career in Analytics. Given that these represent most of the new talent in the field, this is an issue to address ASAP.
The problem is not easy to detect by HR at interview time. A typical computer science graduate may well have one or more AI, machine learning, data mining and even statistics courses under their belt. Indeed, they are capable of writing from scratch some of the more sophisticated algorithms of machine learning.
And therein lies the problem. They may insist on writing algorithms when they should be extracting value from data. They will not appreciate the key differences and similarities of algorithms. The will insist on trying them all.
Worse, they may not even have the right basic categories in place. While they will eagerly deploy Boosting, Bagging, Support Vector Machines and Generalised Linear Models they might not do so with the appropriate pre-processing, error function selection and, worst- of all, suitability to the business problem. Worse still, some will not even appreciate that all four are fundamentally distinct from means clustering. To some, they are all just cool algorithms, fun to play with.
Worst of all, this is a cohort that sees Business Analytics as in IT job. Their main activity is tinkering with, writing, re-writing and deploying algorithms. Their computer science backgrounds provide them with an IT model of Analytics, where being able to deploy an algorithm from scratch is possible without understanding the statistical subtlety that gave rise to the algorithm in the first place, and distinguishes it from other methods. Not really understanding the theoretical basis, these candidates are inclined to try them all. Parameters are tinkered with based on “best practice” or voodoo rather than sound statistically trained intuition.
Worst of all, the final product is the code itself, or “specified” outputs, rather than a considered analysis.
There is a somewhat superior cohort which is more inclined to explore the data. This group suffers from a lack of training in this approach, and must rely on their natural curiosity alone, without the benefit of understating the multidimensional, correlated, uncertainty and information-rich nature of data.
The key problem here is that Business Analytics needs educated, curious “finders”, hunters of truth in data, who know their tools, and also their prey, and enjoy the uncertain, manual, iterative nature of the hunt. They also understand their clients, and their multiple, sometimes uncertain, under defined or conflicting objectives. They flourish in uncertainty, and the thrill of the hunt. Tools are important, and sometimes they can build their own, but the tools are far from the main thing, and the process is far less important than the finding.
The IT model of training instead creates “builders”, who see their role as creating, testing and comparing algorithms, which are implemented as black boxes, part of clearly specified processes. Either the process itself, or the data produced by it is the end result. This is what we refer to as the IT model of Analytics.
The most naturally curious and intelligent of these still find a way to become “finders”, but without the benefit of rigorous statistical training, a shortfall that they usually address in time, leading to their becoming competent Business Analytics professionals.
The rest tend to be an ongoing problem, particularly in the larger companies and government departments where they tend to accumulate. At best, they are naturally well suited to data-acquisition, warehousing and pre-processing tasks – supporting Business Analytics at a low level, but not taking part in the real thing. At worst, they suck in valuable time, money and the attention of management, while nothing substantial is produced in terms of insights, or even statistically rigorous, meaningful data processing. They can be particularly problematic in government departments such as those in Australia where it is virtually impossible to fire someone once hired, and difficult enough to direct, performance manage or criticise staff.
This is a fundamental problem. Happily, it can be addressed quite easily on the recruitment side. It should be simple enough to determine whether the candidate is at heart a builder or finder, and what level of statistical analytic, as opposed to computational training they have. All it really takes is to ask some key questions in the interview, and take a critical eye to the CV. Of course, it requires an appropriately educated interviewer.
On the educational side, the issue is a little more complex. Again, the solution begins with recognizing the key distinction between builders and finders. There is a professional track for both, and current computer science courses are better at preparing people for jobs in data warehousing, BI implementation, ETL and other tasks supporting Business Analytics. Perhaps this track should have its own name, to distinguish it from Business Analytics proper.
Having recgnised the key distinction, What can computer science undergrad courses do to produce more actual Analytics professionals ? Is there a computer science graduate “finder” ? Of course there is. But these tend to be the most gifted, curious and unusually quantitative in their training.
The obvious solution is to create a serious, multidisciplinary degree, one with the right amount of computer science, mathematics, statistics, psychology and business studies, ideally a four or five year course. Most importantly, there must be specialized subjects in Business Analytics, taught by competent practitioners. There would also be specialized subjects on data preparation, business tools, communication skills and other things that current undergrads lack.
Whether this course sits in computer science or elsewhere is less important than producing well educated, multidisciplinary finders if we are to meet the current and growing training shortfall.
In my experience working for software vendors the answer to this has always been ‘yes and no’, but the one sure thing is that everyone uses Excel. Spreadsheets are the most pervasive and effective decision support tools. No organisation doesn’t use them, and it’s a safe bet that this will always be the case. No amount of data warehousing will ever be able to provide decision makers with all the information they need. To the extent that it can, those decisions can be automated. Decisions invariably require new data. That new data will be either unanticipatable, or tacit, or both. Spreadsheets are unbeatable for ad hoc data analysis and turning tacit data into explicit data.
Evelson poses his questions in the context of (presumably Forrester) research into BI Pricing, which he says is:
[S]howing a broad range of transparency (or non transparency) from BI vendors themselves. Some vendors welcomed our research RFI and are happily providing all the info we requested. Some are less transparent and are insisting that we only publish price ranges or comparative analysis (who’s more/less expensive) without showing their exact quotes. Yet, some others have declined to participate.
That doesn’t surprise me. Wide price ranges are both inevitable and understandable. Software businesses, particularly in growth markets like BI, concentrate more on increasing revenue than on managing to the bottom line. Costs just don’t matter as much. They’re also indirect – software being an information product. Part of the software sales process is working out what the prospect is willing to pay, which is basically what they’ll end up paying, which will vary from customer to customer.
Related Analyst First posts:
Enough of Cato Unbound’s What’s Wrong With Expert Predictions debate has now unfolded that it makes sense for me to offer some commentary. The discussion encompasses many predictive and decision-making subject areas and institutions – politics, economics, business, media punditry – but for the purposes of Analyst First I’m primarily interested in prediction in the context of organisations.
All the discussants agree that expert predictive track records are terrible, but they diverge in the degree to which they see this as problematic and in their recommendations as to what to do about it. The debate so far:
In their Lead Essay, Dan Gardner and Philip Tetlock present a puzzle:
Every year, corporations and governments spend staggering amounts of money on forecasting and one might think they would be keenly interested in determining the worth of their purchases and ensuring they are the very best available. But most aren’t. They spend little or nothing analyzing the accuracy of forecasts and not much more on research to develop and compare forecasting methods.
They go on to provide an overview of Tetlock’s longitudinal study of experts, encompassing 28,000 predictions over a fifteen year period, which found that eclectic foxes outperform dogmatic hedgehogs, but that both are outperformed by extrapolation algorithms. They argue that we need to get better at accepting our limitations and to “give greater consideration to living with failure, uncertainty, and surprise”. They accordingly call for “decentralized decision-making and a proliferation of small-scale experimentation”.
In the Reaction Essay section, Robin Hanson addresses the puzzle of why forecasting remains so immune to accountability via – presumably easy to assemble – track records: “[s]urprising disinterest [he means uninterest] in forecasting accuracy could be explained either by its costs being higher, or its benefits being lower, than we expect.” His conclusion is that, even in profit and loss settings such as organisations, the signalling value of forecasting must compete with its information value:
Even in business, champions need to assemble supporting political coalitions to create and sustain large projects. As such coalitions are not lightly disbanded, they are reluctant to allow last minute forecast changes to threaten project support. It is often more important to assemble crowds of supporting “yes-men” to signal sufficient support, than it is to get accurate feedback and updates on project success. Also, since project failures are often followed by a search for scapegoats, project managers are reluctant to allow the creation
He points out that, while prediction markets are best able to incentivise information holders to provide accurate forecasts, institutional respect for accuracy is a necessary and thus far absent precondition to their widespread uptake.
John H. Cochrane turns the tables by arguing that unforecastability is a good sign as seen through the lens of economics:
In fact, many economic events should be unforecastable, and their unforecastability is a sign that the markets and our theories about them are working well.
This statement is clearest in the case of financial markets. If anyone could tell you with any sort of certainty that “the market will go up tomorrow,” you could use that information to buy today and make a fortune. So could everyone else. As we all try to buy, the market would go up today, right to the point that nobody can tell whether tomorrow’s value will be higher or lower.
An “efficient” market should be unpredictable. If markets went steadily up and delivered return without risk, then markets would not be working as they should.
Forecasting, in the sense of accurately trying to predict the future, is a “fool’s game”. But it does work as an input into risk management:
The good use of “forecasting” is to get a better handle on probabilities, so we focus our risk management resources on the most important events. But we must still pay attention to events, and buy insurance against them, based as much on the painfulness of the event as on its probability. (Note to economics techies: what matters is the risk-neutral probability, probability weighted by marginal utility.)
So it’s not really the forecast that’s wrong, it’s what people do with it. If we all understood the essential unpredictability of the world, especially of rare and very costly events, if we got rid of the habit of mind that asks for a forecast and then makes “plans” as if that were the only state of the world that could occur; if we instead focused on laying out all the bad things that could happen and made sure we had insurance or contingency plans, both personal and public policies might be a lot better.
Cochrane defends a hedgehog-like reversion to principles – basic economic principles like supply and demand – in order to build effective conditional forecasts which inform plans and provide decision support.
Bruce Bueno de Mesquita argues that expert prediction is, properly contextualised, a sideshow. Statistical methods are widely used, so much so that we’ve ceased to notice (e.g. in insurance pricing and political polling). Game theory is better still, and continues to make incremental progress:
Are these methods perfect or omniscient? Certainly not! Are the marginal returns to knowledge over naïve methods (expert opinion; predicting that tomorrow will be just like today) substantial? I believe the evidence warrants an enthusiastic “Yes!” Nevertheless, despite the numerous successes in designing predictive methods, we appropriately focus on failures. After all, by studying failure methodically we are likely to make progress in eliminating some errors in the future.
So why do we continue to focus on the poorly performing experts? De Mesquita’s view is that:
Unfortunately, government, business, and the media assume that expertise—knowing the history, culture, mores, and language of a place, for instance—is sufficient to anticipate the unfolding of events. Indeed, too often many of us dismiss approaches to prediction that require knowledge of statistical methods, mathematics, and systematic research design. We seem to prefer “wisdom” over science, even though the evidence shows that the application of the scientific method, with all of its demands, outperforms experts.
De Mesquita goes on to explain and advocate his own game theoretic (Expected Utility Model) approach:
Acting like a fox, I gather information from a wide variety of experts. They are asked only for specific current information (Who wants to influence a decision? What outcome do they currently advocate? How focused are they on the issue compared to other questions on their plate? How flexible are they about getting the outcome they advocate? And how much clout could they exert?). They are not asked to make judgments about what will happen. Then, acting as a hedgehog, I use that information as data with which to seed a dynamic applied game theory model. The model’s logic then produces not only specific predictions about the issues in question, but also a probability distribution around the predictions. The predictions are detailed and nuanced. They address not only what outcome is likely to arise, but also how each “player” will act, how they are likely to relate to other players over time, what they believe about each other, and much more.
In the Conversation section, Robin Hanson challenges Cochrane and de Mesquita to produce conditional forecasts and submit them to systematic public measurement and verification. He is doubtful, however, that they will assent:
The sad fact is that the many research patrons eager to fund hedgehoggy research by folks like Cochrane and De Mesquita show little interest in funding forecasting competitions at the scale required to get public participation by such prestigious folks.
Forecasting, he contends, is a domain in which the rewards to affiliation with prominent expertise trump accuracy.
Bruce Bueno de Mesquita replies that the acceptance of his methods in journals, via peer review, is evidence of their having been sufficiently scrutinised; furthermore that no one has been willing to publically compete with him; additionally that he has successfully beaten alternative approaches; and finally that he has made his methods available online.
Robin Hanson responds that more comprehensive standards of proof are required to settle the matter.
Gardner and Tetlock then provide an insightful running summary. In response to Hanson they speculate that the costs of admitting to poor forecasting performance would disenfranchise those currently enjoying their – unjustified in terms of performance – public and organisational reputations:
Open prediction contests will reveal how hard it is [for them] to outperform their junior assistants and secretaries. Insofar as technologies such as prediction markets make it easier to figure out who has better or worse performance over long stretches, prediction markets create exactly the sort of transparency that destabilizes status hierarchies… If these hypotheses are correct the prognosis for prediction markets—and transparent competitions of relative forecasting performance—is grim. Epistemic elites are smart enough to recognize a serious threat to their dominance.
In response to Cochrane they speak up for the value of hedgehogs – more compelling, more visionary, better at envisioning extreme events – but note that the cost of this is that they are more wrong, more often.
They close by welcoming de Mesquita’s willingness to be publically scrutinised, note that the jury is still out in terms of systematic and decomposed measurement of his methods, and caution that:
For many categories of forecasting problems, we are likely to bump into the optimal forecasting frontier quite quickly. There is an irreducible indeterminacy to history and no amount of ingenuity will allow us to predict beyond a certain point.
De Mesquita responds that he welcomes being assessed.
Although Cochrane comes close, none of the discussants explicitly recognises and makes central the difference between forecasting and other activities which organisations call forecasting (i.e. planning and goal setting). I explained this distinction in a previous post, namely:
- Forecasting means objectively estimating the most likely future outcome: “what’s going to happen?”
- Goal setting means putting a target in place, generally for motivational purposes: “what would we like to happen?”
- Planning means establishing an intended course of action, usually to direct the allocation of resources: “what are we going to do?”
This distinction is key because, while all three activities are based on prediction, only in the case of forecasting is predictive accuracy the primary purpose. Organisations can improve all of these, but to do so they need to address three tiers of potential failure:
All the Cato discussants take it as read that, in assessing predictions, they’re operating in an empirical paradigm. In organisations, however, this can’t be taken for granted. Many organisations place prediction either in the wrong paradigm, or no paradigm at all. It’s common for predictive activities and processes to be ritualised and adhered to, but without any systematic error measurement or validation. Gardner and Tetlock acknowledge the “widespread lack of curiosity—lack of interest in thinking about how we think about possible futures” as “a phenomenon worthy of investigation in its own right,” pointing out the wastefulness of remaining ignorant given the resources involved.
Systematic error measurement and validation can’t happen without the right categories being first recognised and agreed upon. Disambiguating forecasting from goal setting from planning is critical. Organisations don’t do this well. Loose language doesn’t help. The same Finance department will update a budget (a plan) and call it a “forecast”, oversee the revision of sales “forecasts” (goals), and publish revenue estimates for the scrutiny of stock market analysts (true forecasts). As an earlier Analyst First post pointed out, these activities, while all reliant on objective estimation, do not share the same benchmarks when it comes to assessing error and value. Forecast error makes sense for forecasting; execution error makes more sense for goal setting and planning.
The Cato discussants all tacitly acknowledge these distinctions, but none recognises its implications when it comes to understanding the way organisations do prediction.
Tetlock’s experiment required that pundits’ anonymity be protected. Participants knew to distance themselves from their projections when they were accountable for accuracy. The implication here is either that pundits are dishonest, or that they recognise that their projections serve a purpose other than informing people about the likelihood of future events. Gardner and Tetlock, and Hanson, acknowledge that punditry is a form of entertainment, has signalling value, and by virtue of this trades off accuracy for clarity and narrative value. As Hanson puts it:
Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions.
Hanson recognises that affiliation with sophistication has signalling value within organisations too. He notes the multiple roles played by managers, including the requirement that they appear impressive enough to attract affiliation and inspire their subordinates:
[C]onsider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness.
He goes on to describe the motivational impact of managerial ‘overconfidence’:
Often, managers can increase project effort by getting participants to see an intermediate chance of the project making important deadlines—the project is both likely to succeed, and to fail. Accurate estimates of the chances of making deadlines can undermine this impression management. Similarly, overconfident managers who promise more than they can deliver are often preferred, as they push teams harder when they fall behind and deliver more overall.
Incentivising workers to “deliver more overall” is precisely the purpose of goal setting. Consistently producing overshooting projections in this context isn’t necessarily “forecast hypocrisy,” as Hanson characterises it. It may be effective stretch targeting.
Many of the discussants also acknowledge that planning is a different activity from forecasting (and goal setting), but don’t pursue the full implications of this in terms of error and value measurement. The Kenneth Arrow anecdote relayed by Gardner and Tetlock, for example, illustrates that plans are reliant on, but different from, forecasts:
Some [corporations and governments] even persist in using forecasts that are manifestly unreliable, an attitude encountered by the future Nobel laureate Kenneth Arrow when he was a young statistician during the Second World War. When Arrow discovered that month-long weather forecasts used by the army were worthless, he warned his superiors against using them. He was rebuffed. “The Commanding General is well aware the forecasts are no good,” he was told. “However, he needs them for planning purposes.”
Gardner and Tetlock look also at the role of self-aware (i.e. of limitations) prediction in preparedness planning, comparing the effectiveness of the recent New Zealand and Haiti earthquake responses:
Designing for resiliency is essential, as New Zealanders discovered in February when a major earthquake struck Christchurch. 181 people were killed. When a somewhat larger earthquake struck Haiti in 2010, it killed hundreds of thousands. The difference? New Zealand’s infrastructure was designed and constructed to withstand an earthquake, whenever it might come. Haiti’s wasn’t.
Cochrane seconds this, adding that predictions have scenario generation utility regardless of their accuracy:
Once we recognize that uncertainty will always remain, risk management rather than forecasting is much wiser. Just the step of naming the events that could happen is useful.
In these and other ways, the discussants acknowledge that accuracy isn’t the only purpose of prediction. It should therefore follow that forecast error might not be the only relevant measure.
Much of the discussion contrasts different predictive tools, techniques and approaches: expert judgement, statistical algorithms, prediction markets, game theory. Methodologies and expectations both need to be appropriately calibrated: simple statistical extrapolation works well in some settings, but in complex systems environments the best we can hope for may be a better feel for the probabilities involved.
There are a range of insights here for organisations. Individual human judgement on its own, it is unanimously acknowledged, performs poorly. Statistical algorithms consistently beat the experts. There is general agreement among the discussants that eclecticism is desirable. The clear implication is that organisations should adopt collective intelligence methods.
Tetlock’s wider work on expert political judgement has implications for optimal forecasting team composition (use hedgehogs to generate possibilities and foxes to synthesise and calibrate probabilities). Gardner and Tetlock also call for what we term Decision Performance Management:
Imagine a system for recording and judging forecasts. Imagine running tallies of forecasters’ accuracy rates. Imagine advocates on either side of a policy debate specifying in advance precisely what outcomes their desired approach is expected to produce, the evidence that will settle whether it has done so, and the conditions under which participants would agree to say “I was wrong.” Imagine pundits being held to account.
It’s also worth imagining what sort of environment supports this, as Hanson does in his discussion of a different “social equilibrium”:
A track record tech must be combined with a social equilibrium that punishes those with poor records, and thus encourages rivals and victims to collect and report records. The lesson I take for forecast accuracy is that it isn’t enough to devise ways to record forecast accuracy—we also need a new matching social respect for such records.
He’s right. New ways to record accuracy aren’t enough. We also need to know whether accuracy is the real goal. On the subject of goals, when it comes to organisational planning and goal setting, it may well be that these are best understood in a game theoretic context.
Whatever the case, once they are disambiguated, and because they are all related, empiricism means doing forecasting and goal setting and planning better.
Related Analyst First posts:
- Hedgehogs are foxy when they’re right
- *What’s Wrong with Expert Predictions*
- *The Folly of Prediction*
- Forecast error versus execution error
- Forecasting, goal setting, planning
- Robin Hanson on Information Accounting
- Paying for software is buying insurance
That’s the title of today’s Harvard Business Review Management Tip of the Day, which is worth reproducing in full:
Differentiating your company based on products or cost is near impossible these days, especially in crowded industries. Instead, pull ahead of the pack by using data-collection technology and analysis to get value from all of your business processes. Analytics let you discern not only what your customers want, but how much they’re willing to pay and what keeps them loyal. It also arms your employees with the evidence and tools they need to make sound decisions. Start by championing analytics from the top. Acknowledge and endorse the changes in culture, process, and skills that analytics competition requires. Be sure that you understand the theory behind various quantitative methods so you can recognize their limitations. If necessary, bring in experts who can advise on how to best apply analytics to your business.
Related Analyst First posts:
Yesterday night saw two back-to-back events at Deloitte in Melbourne:
The A1 meeting went well, attended by approximately 20. Arranged by Yuval Marom, the founder and convener of MelbURN, and chaired by the dynamic Richard Fraccaro, head of Melbourne’s A1 chapter.
The presentation consisted of an update of what A1 has achieved since it was founded and revealed at MelbURN in October last year. The recent AIPIO presentation, the Canberra launch, the Sydney Chapter think tank and this website/knowledge repository all rated a mention.
A healthy discussion followed on the nature and needs of analytics education, prompted by a question regarding the role of academia.
But this was not the main event.
Following right after, and attended by 45 or so, came the MelbURN event, with the title “Experiences with using SAS and R in insurance and banking”, presented with human and unassuming polish by Hong Ooi, statistician at ANZ Bank.
This was, without exaggeration, one of the best R presentations I have ever seen.
Hong taught the audience some very important things about banking and finance, rigorous statistics, data representation and a masterful use of the R language, and key R packages such as plyr.
More importantly, he provided not one, but multiple case studies, and in each a comparison of R and SAS, as well as ways of combining the two together. This included calling R from SAS, and using R to generate SAS code.
Most striking for me was the comparison of SAS with R in a live, corporate financial context, and the presentation of R as a viable, robust, industrial strength option, with some unique advantages, and admitted weaknesses.
I hope that Hong can present this again to the Sydney Users of R Forum (SURF)
His presentation slides can be found here.
A video of the presentation will hopefully be available soon.
Stephen and I will be in Melbourne until Saturday if any A1 blog readers want to meet before then.
The difference, and relationship, between data and information is a common debate. Not only do these two terms have varying definitions, but they are often used interchangeably.
Just a few examples include comparing and contrasting data quality with information quality, data management with information management, and data governance with information governance.
That’s Jim Harris at Information Management. He cites the distinctions commonly made between data, information, knowledge, and wisdom, arguing that the term Knowledge Management makes a lot of sense as a way of describing the goals of business intelligence:
I can’t help but wonder if the debate about data and information obfuscates the fact that the organization’s appetite, its business hunger, is for knowledge.
He concludes with three insightful questions, designed to determine whether the distinctions are consequential or merely linguistic:
- Does your organization make a practical distinction between data and information?
- If so, how does this distinction affect your quality, management, and governance initiatives?
- What is the relationship between those initiatives and your business intelligence efforts?
The post is interesting to me because it catalogues various attempts to get away from the word “data”. In my experience, “data” is a spellword. Its invocation gives people permission to tune out and to dismiss what follows as technical, geeky, and irrelevant. Business Analytics practitioners themselves don’t do this, of course, but businesspeople often do. Status signals are important inside organisations, and by virtue of association, “data” is lower status than “information” or “knowledge”. All other things being equal, Information Management therefore carries greater business cachet than Data Management.
Knowledge is further up the value chain than information, and as a previous post has noted, linguistic slipperiness is common in business. So why not Knowledge Management? Harris notes that the term is no longer in vogue in the business world. The key reason, I suspect, is that enough initiatives coined ‘Knowledge Management’ when the term was still fresh went on to fall short of expectations.
The last fifteen years saw a lot of reporting get rebranded as Business Intelligence, and we are now seeing lots of Business Intelligence getting rebranded as Analytics. Lots of Business Intelligence efforts have disappointed their sponsors, so I wasn’t surprised when a Melbourne colleague told me yesterday that Business Intelligence was becoming a dirty word around town.
Related Analyst First posts:
What are the defining attributes of the ideal Business Analytics sponsor? Over the years we have distilled them down to three, the ‘holy trinity’:
- Understanding: The sponsor needs to be analytically literate. Not a practitioner necessarily, but knowledgeable enough to know how to manage in both directions. Managing a team of analysts requires knowing how to tell whether they’re pursing relevant questions in technically appropriate ways and knowing how to validate what they produce. This means understanding the organisation’s data and its analysis objectives, knowing what analytical techniques exist and where they fit, which error measures are appropriate to each, and how to interpret results. Managing up means knowing how to communicate analytical results to non-analysts, particularly to senior management, and critically, how to translate executive demands back into analytically tractable questions.
- Empowered: The sponsor also needs to be given the appropriate mandate. The Business Analytics function needs to be resourced with human and electronic infrastructure, of course, but it also needs to be protected politically. Doing analytics means measuring things. Sometimes those things are people, and none of us likes to look bad. Sometimes those things are business functions, and none of us likes to be part of an ineffective team. Then there are the various organisational and perceptual barriers which make it difficult for Business Analytics functions to learn from mistakes and adapt.
- Motivated: Finally, the ideal sponsor needs to be motivated and incentivised enough to stay the course. Business Analytics initiatives typically take time to get going, and often their analyses don’t turn up the results people were wanting or expecting. Sometimes the result is ‘no result’, and this can be dispiriting. Other times it’s a mythbusting or counterintuitive finding, and this can be challenging. Almost always, the organisation’s data is in a more of a mess than expected. Analytical blind alleys are also inevitable, and exploration and discovery aren’t assessed via a mature set of measures. Nor is Business Analytics an IT project that can be communicated and monitored using standard KPIs. The Business Analytics sponsor, simply put, needs to be on a perpetual ‘internal sales’ mission.
- Analytics is… A Literacy – Parts 1 and 2
- Barriers to entry and exit
- Analytics is… A Lean Startup Enterprise
- Assume bad data
- Forrester on the need for agility
- IT support
Welcome to A1′s very first podcast.
This is a relatively quick (less than 30 mins) overview of what Analyst First is all about, and why Human Infrastructure matters so much.
This is a recording of the presentation I gave to the Intelligence 2011 conference, which is the annual conference of the Australian Institute of Professional Intelligence Officers (AIPIO), as part of their very apt “The Analyst vs the IT” stream.
Tried and true best practices for enterprise software development and support just don’t work for business intelligence (BI). Earlier-generation BI support centers — organized along the same lines as support centers for all other enterprise software — fall short when it comes to taking BI’s peculiarities into account. These unique BI requirements include less reliance on the traditional software development life cycle (SDLC) and project planning and more emphasis on reacting to the constant change of business requirements.
That is from Forrester Research’s Agile Business Intelligence Solution Centers Are More Than Just Competency Centers report, just released. The full version of the report is paid (USD 499) but a free overview from its two lead authors, Boris Evelson and Rob Karel, is here. The case against the SDLC / project approach is summarised thus:
Earlier-generation BI support organizations are less than effective because they often:
- Put IT in charge
- Remain IT-centric
- Continue to be mostly project-based
- Focus too much on functional reporting capabilities but ignore the data
In response, Forrester advocates a a ‘flexible and agile’ approach to BI, and establishing “BI on BI” to explicitly learn from successes and failures.
This echoes much of what Analyst First advocates, namely that:
- Analytics is not IT
- IT risk management practices hamper Business Analytics initiatives
- It is prudent to assume bad data
- A Lean Startup approach makes more sense
- A good deal of Business Analytics is bespoke
Note that Forrester is making these recommendations at the Business Intelligence end of the Business Analytics spectrum. It’s arguing that, even where Business Analytics lives in an operational, repeatable, systematised, automated, decision automation context:
[No] repository can fully substitute for personal, qualitative knowledge; that’s often more art than science. Therefore, staff the BICC/COE [Business Intelligence Competency Centre / Centre Of Excellence] with individuals whose primary responsibility is to disseminate such knowledge above and beyond what’s available in the repository.
In other words, the human infrastructure is critical, and investments in electronic infrastructure which ignore it will be unsuccessful.
Related Analyst First posts:
IBM has released its latest biennial C-suite study of CIOs, The Essential CIO: Insights from the Global Chief Information Officer Study, summarising interviews with more than 3,000 CIOs. The study is available for download here (free, requires registration). It contains much of interest for Business Analytics practitioners and sponsors:
One of the most compelling findings in the study is that CIOs are now increasingly in step with CEOs’ top priorities. One priority they agree on is how critical it is for today’s public and private sector organizations to derive insight from the huge volumes of data being amassed across the enterprise, and turn those insights into competitive advantage with tangible business benefits.
CIOs increasingly help their public and private sector organizations cope with complexity by simplifying operations, business processes, products and services. To increase competitiveness, 83 percent of CIOs have visionary plans that include business intelligence and analytics.
Our research suggests that this new alignment [between CIOs and CEOs] comes as CEOs better understand the importance of technology. They increasingly rely on CIOs to turn data into usable information, information into intelligence and intelligence into better decisions.
Business intelligence and analytics ranked as the highest CIO priority across the board, ahead of Mobility solutions, Virtualization, Cloud computing, Business process management, Risk management and compliance, Self-service portals, and Collaboration and social networking. There was also “remarkable consensus” on how these priorities should be addressed:
CIOs identified the top three success factors for IT initiatives as putting in place the correct IT/ business talent, managing beyond line responsibilities and creating the right conditions before starting.
Read that as an advocation of the importance of human infrastructure.
The study segments organisations into four groups based on ‘CIO Mandate’. Each mandate reflects how the IT function is viewed by the rest of the organisation:
- Leverage: ”Provider of fundamental technology services”
- Expand: ”Facilitator of organizational process efficiency” (the most common CIO mandate)
- Transform: ”Provider of industry-specific solutions to support business”
- Pioneer: ”Critical enabler of business/organization vision”
These are presented as cumulative, but not as a progression path per se – rather as a reflection of the nature of different businesses. They also read as a continuum from operational to strategic. Some organisations see IT’s job as keeping the machines running (Leverage), and perhaps facilitating marginal efficiencies (Expand). The more strategic IT functions are seen by their organisations as enablers of competitive advantage (Transform) or drivers of change (Pioneer). Nonetheless, Business Analytics is on most mandates’ radar:
A full 95 percent [of Expand mandate CIOs] said they would lead or support efforts to drive better real-time decisions and take advantage of analytics.
Analytics and data management hold the key to extracting greater value from data. Over the next three to five years, the majority of Transform mandate CIOs across our sample will focus on customer analytics, product/service profitability analysis and master data management.
This means moving beyond traditional relational database management systems into the next generation of integrated data warehouses and analytical tools. A Consumer Products CIO in Australia said, “A master data management initiative will cleanse corporate data, facilitating our ability to deliver rich customer analytics for the business.”
IBM’s recommendations to Transform CIOs include:
Harness more real-time data Generate insights through feedback collection, sentiment analysis and connecting CRM to social networks. Use the data explosion to grow relationships with all key stakeholders.
Analyze! Dive deep into advanced analytics to develop insights into customer behavior, value chain relationships and competitive intelligence. Deploy text analysis to glean insights from structured and unstructured data, including blogs, customer service records and Web transactions.
On the Pioneers:
This group of CIOs ranked product/service profitability analysis and product/service utilization analysis as their top two priorities for turning data into usable intelligence.
[Pioneer] CIOs are in a unique position within an organization. They help generate and have access to customer preference data, supply chain patterns, emerging trends—both within their organizations and from competitors— Internet behavior and response patterns, and so much more. Combining this data with marketing analytics can reveal previously undiscovered and unmet needs. It can lead to product innovations, massive process changes, cross-industry value chain cooperation and other synergies across industries.
IBM’s recommendations to Pioneer CIOs include:
Develop a culture of analytics Build predictive intelligence capabilities that can fundamentally change the business. Encourage widespread application of analytics to fully leverage business intelligence. Take an advanced look at what drives profitability.
Add dials to your dashboards Offer dynamic dashboards using real-time data and use predictive analytics to provide situational metrics, including: formal business case monitoring; customer satisfaction; employee motivation; and social value and sustainability.
Pioneers are encouraged to ask themselves, among other questions:
How can you develop the talent to apply predictive intelligence to radically change your business model, products or industry?
How will you design dynamic dashboards that leverage real-time data and predictive analytics?
Among the concluding advice for CIOs across the board is:
Embrace the power of analytics Educate yourself, your team and your organization about extracting meaning from unstructured data sources, predictive intelligence, social network analysis and sentiment mining.
The study is a useful companion piece to the recent McKinsey report on big data. Both align Business Analytics with recent technological trends: big data, RFIDs and sensors, real-time Web transactions, and so on. Both also focus on operational analytics (e.g. “dynamic dashboards using real-time data”) and decision automation. I see this as a shortcoming of the McKinsey report and as more understandable in IBM’s case given their market and worldview.
There is a notable recognition of complexity as a growing problem and a desire to simplify (internally, for clients, for partners).
Also, reading between the lines, evidence that Business Analytics basics are still causing trouble for organisations. Transform CIOs are either planning to or advised to build dashboards, for example. I’ve learned to interpret this as “we’re having trouble getting to our data”. There is also a tendency to conflate data warehousing with data analysis. The quote above from the Australian Consumer Products CIO is a good example of this.
Finally, the study focuses on CIO plans and priorities, not on their successes and failures. I’d be fascinated to know what CIOs have been surprised by in the past. What’s exceeded and disappointed expectations?
Related Analyst First posts:
- McKinsey on big data
- The Business Analytics market should be much bigger
- Vendor worldviews
- Vendor worldviews evolve
- Decision support versus decision automation
About usAnalyst First is a new approach to analytics, where tools take a far less important place than the people who perform, manage, request and envision analytics, while analytics is seen as a non-repetitive, exploratory and creative process where the outcome is not known at the start, and only a fraction of efforts are expected to result in success. This is in contrast with a common perception of analytics as IT and process.
Tags in a CloudAIPIO analyst first Analyst First Chapters analytics analytics is not IT arms race environments big data business analytics business intelligence cargo cults collective forecasting commodity and open source tools complexity data decision automation decision support educated buyer EMC-greenplum forecasting HBR holy trinity human infrastructure incentives intelligence model of analytics investing in data lean startup literacy management culture MBAnalytics operational analytics organisational-political considerations Philip Russom Philip Tetlock prediction markets presales R Robin Hanson Strategic Analytics tacit data TDWI Tom Davenport uncertainty uneducated buyer vendors why analyst first