From the monthly archives: August 2011
Week 1, Day 5 of the CORTEX MBAnalytics program includes Tom Davenport’s ‘Rethinking Knowledge Work: A Strategic Approach’ from the McKinsey Quarterly of January 2011. In the essay, Davenport argues that productivity software hasn’t boosted the productivity of “knowledge workers” to the extent hoped for given the outlays of the last two decades. The primary method employed over this period has been what he calls ‘free-access’: providing knowledge workers with tools and information and leaving it to them to work out what to do with them:
In this model, knowledge workers define and integrate their own information environments. The free-access approach has been particularly common among autonomous knowledge workers with high expertise: attorneys, investment bankers, marketers, product designers, professors, scientists, and senior executives, for example. Their work activities are seen as too variable or even idiosyncratic to be modeled or structured with a defined process.
This approach suits when there is uncertainty, ambiguity, and contingency, each of which work against predictability. The upside is the ability of humans to adapt to these. The downside is that autonomy doesn’t come for free. Workers will execute variably, some poorly. The lack of standardisation leads to duplication and other kinds of inefficiency. Precise performance measurement and management is also a challenge. Typical productivity metrics in the free-access domain are rough and high level if present at all, and there is a trade-off between additional measurement and ease of information access.
The alternative model Davenport terms ‘structured-provisioning’, in which tasks and deliverables are defined and knowledge workers slotted in. Typical examples are workflow or ‘case management’ systems, which integrate decision automation, content management, document management, business process management, and collaboration technologies:
Case management can create value whenever some degree of structure or process can be imposed upon information-intensive work. Until recently, structured-provision approaches have been applied mostly to lower-level information tasks that are repetitive, predictable, and thus easier to automate.
The upside is efficiency. The downsides are worker alienation and resistance, and detrimental business outcomes resulting from complexity and poor specification—bad mortgages, for example.
Davenport believes that businesses should increasingly “structure previously unstructured processes”. That is, that the free-access domain should be progressively structure-provisioned. He uses a 2 x 2 matrix to frame his argument. On the x-axis is ‘Complexity of work’, ranging from Routine across to Intepretation/judgement. On the y-axis is ‘Level of interdependence’, ranging from Individual actors up to Collaborative groups. The resulting knowledge work quadrants are:
- Transaction model (Routine x Individual actors)
- Expert model (Interpretation/judgement x Individual actors)
- Integration model (Routine x Collaborative groups)
- Collaboration model (Interpretation/judgement x Collaborative groups)
The Transaction model contains most existing structure-provisioning, and the Collaboration model—consisting of “Improvisational work”, being “Highly reliant on deep expertise across multiple functions”, and “Dependent on fluid deployment of flexible teams”—is inherently free-access. Davenport sees the Expert and Integration models, however, as open to further structured-provisioning.
Martin Ford’s book, The Lights in the Tunnel: Automation, Accelerating Technology and the Economy of the Future, (free as a PDF download), further illuminates these trends. Ford identifies three categories of job vulnerable to displacement by technology:
- Hardware jobs, such as assembly line jobs, which become displaced by robotics—a process which is already well underway.
- Software jobs, such as radiology, which are first displaced by outsourcing, then by AI.
- Interface jobs, such as loan officers, which become displaced by telecommunications, digitisation, and data standardisation.
‘Rethinking Knowledge Work’ is an interesting change of direction for Davenport. His seminal ‘Competing on Analytics‘ essay, and the book that followed, profiled business effectiveness and adaptiveness powered by analytics. The arguments here, by contrast, are all about efficiencies.
[T]o date, high-end knowledge workers have largely remained free to use only the technology they personally find useful. It’s time to think about how to make them more productive by imposing a bit more structure. This combination of technology and structure, along with a bit of managerial discretion in applying them to knowledge work, may well produce a revolution in the jobs that cost and matter the most to contemporary organizations.
Given the vulnerability of so much knowledge work to displacement, it’s a good time to be an analyst. Business Analytics clearly lives in the “Expert model” quadrant. Further to that, Davenport sees it as playing a role in augmenting other expertise within that domain:
Expert jobs may also benefit from “guided” data-mining and decision analysis applications for work involving quantitative data: software leads the expert through the analysis and interpretation of data.
This further validates Analyst First principles, namely our insistence on the importance of human over electronic infrastructure, our conception of Business Analytics as an intelligence rather than IT function, and our focus on strategic in preference to operational analytics.
Related Analyst First posts:
Further to recent initiatives in free online learning covered here is David Friedman’s course on ‘Analytic Methods for Lawyers’ from Santa Clara University. As he describes it:
The idea is that there are a number of subjects, such as statistics, accounting, and economics, that lawyers cannot expect to be competent in but should be familiar with. We spend a week or two on each.
Other methods covered in the course include decision theory, game theory, and ‘back of the envelope’ calculation. In essence, the classes (which are available for download here as recordings and whiteboard snapshots) provide ‘literacy primers’ for business professionals on each of these forms of reasoning. Analyst First maintains that analytics—over and above being a discipline, a set of techniques, and a profession (all of which it is)—is a literacy. As such, does analytical literacy draw on, live in parallel with, or subsume these?
All of the above. Business Analytics at a minimum fuses statistics (probabilistic reasoning) with accounting (the language of business). Some analytical techniques additionally integrate game theory (e.g. agent-based modelling). Others such as prediction markets bring in price theory from economics. All involve the scientific method, and therefore require empirical literacy.
David Friedman is a brilliant communicator and his lecture audio files are highly recommended. There is something new for everyone in the Analytic Methods course.
Related Analyst First posts:
- Want an intro to AI from Stanford?
- CORTEX MBAnalytics – Week 1, Day 1
- Analytics is… A Literacy – Parts 1 and 2
BBC News reports that ‘black swans’ are busting IT budgets. One in six large IT projects go over budget by an average of 200%, according to a recent Oxford University and McKinsey study, ‘Why Your IT Project May Be Riskier Than You Think’, published in HBR. This comes as no surprise when paired with Gartner’s estimates that 70 to 80% of corporate business intelligence projects fail. It’s interesting from a Business Analytics perspective, both because analytics projects are themselves software dependent—particularly at the operational end—and because project risk analytics are part of the solution.
The study’s authors, Bent Flyvbjerg and Alexander Budzier, describe IT projects as generating “a disproportionate number of black swans”. But one in six is not a rare event—it’s a single roll of the die. Their underlying research shows that decision makers are working with poor initial estimates of probabilities and maintaining them in the face of persistent error. Such widespread failure to readjust projections in response to disconfirmatory data is a signal that accuracy may not be the goal. That is, IT project management straddles the planning and goal setting domains, not the forecasting one. Robin Hanson wrote about the perversities of project planning and management in the Cato Unbound forum on expert forecasting:
Even in business, champions need to assemble supporting political coalitions to create and sustain large projects. As such coalitions are not lightly disbanded, they are reluctant to allow last minute forecast changes to threaten project support. It is often more important to assemble crowds of supporting “yes-men” to signal sufficient support, than it is to get accurate feedback and updates on project success. Also, since project failures are often followed by a search for scapegoats, project managers are reluctant to allow the creation of records showing that respected sources seriously questioned their project.
Often, managers can increase project effort by getting participants to see an intermediate chance of the project making important deadlines—the project is both likely to succeed, and to fail. Accurate estimates of the chances of making deadlines can undermine this impression management. Similarly, overconfident managers who promise more than they can deliver are often preferred, as they push teams harder when they fall behind and deliver more overall.
The primary KPI for large projects, it appears, is simply “completion”. Completion on time, on budget, and sensitive to changes in specification and priority appear to be at best secondary considerations. Flyvbjerg and Budzier cite additional research showing that 67% of companies failed to terminate unsuccessful projects.
But the model failure chronicled by the study runs deeper still. Projects don’t live in political or economic isolation, but planners act as though they do (from the BBC report):
“Black swans often start as purely software issues. But then several things can happen at the same time – economic downturn, financial difficulties – which compound the risk,” explained Prof Flyvbjer
Projects are being approached as though they’re engineering problems when in fact they’re complex systems problems.
The study raised concerns about the adequacy of traditional risk-modelling systems to cope with IT projects, with large-scale computer spending found to be 20 times more likely to spiral out of control than expected.
Size and complexity play critical roles. Flyvbjerg dispells the notion that this is a public sector problem:
“People always thought that the public sector was doing worse in IT than private companies – our findings suggest they’re just as bad.
“We think government IT contracts get more attention, whereas the private sector can hide its details,” he said.
The study’s concluding advice, given both the frequency and magnitude of project failure, is more reality-based risk management:
Any company that is contemplating a large technology project should take a stress test designed to assess its readiness. Leaders should ask themselves two key questions as part of IT black swan management: First, is the company strong enough to absorb the hit if its biggest technology project goes over budget by 400% or more and if only 25% to 50% of the projected benefits are realized? Second, can the company take the hit if 15% of its medium-sized tech projects (not the ones that get all the executive attention but the secondary ones that are often overlooked) exceed cost estimates by 200%? These numbers may seem comfortably improbable, but, as our research shows, they apply with uncomfortable frequency.
In addition:
Even if their companies pass the stress test, smart managers take other steps to avoid IT black swans. They break big projects down into ones of limited size, complexity, and duration; recognize and make contingency plans to deal with unavoidable risks; and avail themselves of the best possible forecasting techniques—for example, “reference class forecasting,” a method based on the Nobel Prize–winning work of Daniel Kahneman and Amos Tversky. These techniques, which take into account the outcomes of similar projects conducted in other organizations, are now widely used in business, government, and consulting and have become mandatory for big public projects in the UK and Denmark.
In other words, take a more learning-oriented approach to project planning and execution, informed by simulations based on data from similar projects. Completion is by itself a dangerous goal. Risk-adjusted completion looks quite different, and like projects provide a better model than idealised and data free assumptions.
Related Analyst First posts:
Any analysis can be understood as the intersection of audience and subject. In the Business Analytics context, typical audiences are you, your customers, and your prospects. Typical subjects—for analyses that model human behaviour as opposed to other processes—include yourself, your customers, your competitors, and your adversaries. Some examples:
- Performance Management: for the organisation, about itself—e.g. employee scorecards, HR cubes, management reporting
- Most BI: for the organisation, about its customers—e.g. sales cubes
- Customer Intelligence: for the organisation, about its competitors and prospects
- Risk Intelligence: for the organisation, about its adversaries
- Most B2C Analytics: for customers and prospects, about customers—e.g. a commerce website’s recommendations engine
Most BI is employee-facing. Most analytics, as it gets operationalised, is aimed at customers and prospects in the form of surveys, experiments, recommendations, and targeted interactions and offers.
Day 3 Week 1 of the CORTEX MBAnalytics program covers ‘Competing on Talent Analytics‘, by Davenport, Harris, and Shapiro, from the October 2010 Harvard Business Review. The essay describes the application of advanced analytics methods more typically aimed at customers to employees. This might otherwise be termed ‘talent analytics’ or ‘HR analytics’ or ‘Performance Management analytics’.
The Analyst First view is that strategic analytics is in many respects easier than operational analytics. In part, operational analytics is hard because motivating and coordinating humans is hard. For typical operational analytics applications to consistently work end-to-end (e.g. driving up customer acquisition, retention and value via predictive modelling and campaign management) and to be able to prove and articulate their value-add, they require the coordination and cooperation of, at a minimum, people in each of the following organisational functions:
- IT
- Data Warehousing and BI
- Analytics
- Marketing
- Call Centre
- Sales
- Product Management
- Finance
The dependent set of business processes are difficult to execute. They are inherently brittle due to their many moving parts. But they’re also difficult to coordinate because they’re human-centric processes. The monitoring and performance management needs of humans are demanding and resistant to automation. The maintenance of human capital is far more mercurial and challenging than the maintenance of physical or information capital. This has implications for competition. It decreases the attractiveness of competing on analytics—particularly operational analytics—relative to alternative competitive frontiers.
Google is a competing on analytics business through and through. But its recent purchase of Motorola Mobile, according to many analyses, was about arming it to attack its competitors in the courtroom using its lawyers rather than in the marketplace using its engineers. Motorola Mobile’s thousands of patents provide it with new ammunition in its patent arms race with Apple, Microsoft, and others in the mobile telephony hardware sector. Its move into the political lobbying game was similarly explained five years ago.
What makes lawsuits and lobbying more attractive than analytics—to a company built on analytics? The law and the legislature are like analytics in many respects: complex domains, information based, and the province of highly qualified, experienced and intelligent specialists. However, far less of their complexity is contingent on the effective coordination and performance management of human activities inside an organisation. Operational analytics frequently fails because a business is divided against itself. Lawyers and lobbyists, on the other hand, can represent a large, complex, multinational business as a single, unified entity. The human coordination effort is far simpler and much less fragile.
Like any other competitive processes, lawsuits and lobbying campaigns can be more effective if they’re informed by analytics. But the sort of analytics that can do this will be of a more bespoke, tactical, or strategic nature, and less amenable to standard practices and operationalisation as IT processes.
Related Analyst First posts:
An explorer returning from new territory may be able to describe the path he took. He may also have drawn a map. A path is more prescriptive than a map. A map is more descriptive than a path—not in terms of detail, necessarily, but in terms of the flexibility it affords. A path presumes a more specific purpose than a map. A map provides more context than a path. To have a map when following a path is to take a more contextually rich journey, and to have options. But options are taxing and scenery distracting when the the goal is simply the most direct route.
A data analyst exploring new data, or newly exploring old data, can create similar objects. He may create a path, enabling others to take the same journey, or saving them from needing to. He may also create a new map, or enlarge, extend, enrich or correct an existing map. In doing so he may uncover new paths, or alternative paths, or territory through which no paths yet exist. Sensemaking and decision support are map-making activities. Decision automation is a path-taking activity. Simulation builds a richer map. Optimisation uses it to find the best path.
This is a short blog to extend our thanks to Richard Volpato for a terrific presentation given to A1 Sydney recently. For those who haven’t been exposed to what Richard’s work and approaches, I believe it is a fantastic example of what can be achieved adhering to A1 principles.
Some key take-out messages for me are the importance of:
Learn what matters before, not after an infrastructure build. Even if Richard’s current employer was convinced against all reason and empirical evidence to implement an alternate system today, the costs of doing so would be a fraction of the ‘blind build’ costs (we’re again reminded of black swan risks in a recent HBR article – http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/2), because they now have a set of very rich insights concerning their revenue profile, and importantly a ‘blue print’ for how to transform and process their data in order to operationalise these insights.
Manage Value exchange. Don’t assume users will give up their highly prized spreadsheets, just because you covet the rich sematic content therein. Richard showed us how offering to assist users to repair, document and update their prized digital assets works well for both parties.
Visualisation. Richard and his team used visualisation to help his management (most of whom I understand have a legal background ) to understand the impact on individual cases of imposing multiple, interacting weighting factors (which while leading to entirely logical outcomes, may nevertheless appear to be counter-intuitive).
Most importantly, this is a success story about what can be achieved through assiduous and open minded interrogation of data, using a ‘fit for purpose’ mix of open source solutions.
Continuing the recent theme of online self-education, Steve Bennett of Oz Analytics and CORTEX is generously developing and sharing an ‘MBAnalytics’ program, consisting of a daily diet of between 5 and 30 minutes of material on Business Analytics. This is a fantastic idea, which kicked off yesterday in fine style:
Week 1, Day 1’s ‘Big idea’ essay is “What People Want (and How to Predict It)” by Thomas H. Davenport and Jeanne G. Harris, from MIT Sloan Management Review—a 2009 survey of the use of prediction and recommendation systems for cultural products. The premiss is that:
[T]he balance between art and science is shifting. Today companies have unprecedented access to data and sophisticated technology that allows even the best-known experts to weigh factors and consider evidence that was unobtainable just a few years ago.
In this context, Davenport and Harris present a series of success stories from the book, music and movie industries. Collaborative filtering, predictive modelling and prediction markets are contrasted with expert judgement, gut feel, and rules of thumb. On the flip side, the article is realistic about data challenges (particularly pre-production) and model decay (given the ephemerality of cultural products). It also finds that, in terms of the uptake of analytics, the “primary obstacles appear to be cultural rather than analytical or technological.”
A financial executive at a major studio confirmed in an interview that so far all prediction models have made relatively little headway with executives who make film production decisions, though he is hoping that they will be applied more frequently in the future.
On business models, the article notes that all the big success stories it identifies (Apple, Netflix, Amazon) are in the distribution of cultural products rather than their production. By contrast:
Most of the companies we encountered that provide only recommendation or prediction capabilities are relatively small.
The article is a good introduction to Business Analytics for executives and non-analysts. Cultural products are familiar. Everyone consumes them. But being subjective, creative, artistic, non-commodity goods, and sensitive to fads, they intuitively seem difficult to predict using ‘sterile’ data-driven methods. Davenport and Harris challenge that intuition.
The ‘extra credit’ material is a video interview of Michele Chambers, GM & VP Analytics Solutions, by Michael Kearney, Director of Product Marketing, both of IBM Netezza.
There are two gold nuggets in the interview, titled ‘Transitioning from Reporting to Predicting’. The first is Michele Chambers’ presentation of advanced analytics in terms of a progression across a spectrum, from low to high value decision support:
- SQL based analytics: simple query and reporting
- Descriptive statistics on historical data: the focus of Business Intelligence; explaining past occurrences
- Data mining / machine learning / predictive analytics: predictive modelling; predicting the future from the past; predicting the most likely outcome with no contingencies
- Simulation: conditional prediction; evaluating multiple scenarios generated by a predictive model given a range of preconditions
- Optimisation: simulation applied to trade-off decisions; selecting the best actions given preconditions and/or the best preconditions for action
The second is her solid advice to organisations on how to get started with analytics, which is to begin with a high business value and impact project, which can deliver in stages, and to iterate through full cycle prototypes:
- Selecting and cleaning a subset of data
- Transforming that data
- Mining for the discovery of insights
- Productionising those insights sufficiently to realise some business value
- Repeating the cycle: embellishing models, enriching and adding data, and building incrementally on each success
This is the ‘Strategic First‘ approach. It focuses on delivering ‘low footprint, high impact’ business value and avoids many of the pitfalls of monolithic electronic infrastructure construction projects.
Related Analyst First posts:
We’ve spoken much within our Analyst First sessions about the transformational power of open source tools like R and RapidMiner, which effectively harness the collective intelligence of the statistical development community, within a system that is largely self-regulating. I can only imagine open source tools must be creating a vexing challenge for commercial analytics software venders and their R&D Depts, who are having to compete with their rich functionality and enthusiastic support, extended through an intelligent and energised network.
Far from being fringe, we hear the power of tools like R are being effectively exploited by large organisations with highly sophisticated Analytics capabilities ranging from Google to the ATO.
Logically, this must take the focus of Analytics away from data ‘crunching’ and reporting to one which enables Strategic Agility. In short, and consistent with A1 principles, Analytics is there to inform how to ‘do better things’, rather than just ‘doing things better’. Unencumbered by the weight of expensive Enterprise application licensing fees and high data processing costs, companies can now single-mindedly focus their budgets and resources on sourcing smart analysts to answer key strategic questions – ‘why are we….?’, ‘should we be …?’, ‘what if we were to…?’.
So much for the theory, what’s the practical reality of this new era of Analytics? Let me share with you why NTF is so committed to A1 principles, and why I believe Eugene and Stephen are truly two of the most astute thinkers on Analytics I’ve encountered.
Take the last couple of weeks at NTF. One of our clients recently engaged us to undertake a project requiring the analysis and modelling of over 10 million customer records. We read the data into Microsoft Powerpivot (which is free – www.powerpivot.com) using a garden variety desktop (8GB RAM; i7 processor; retail cost about $700). We undertook the usual normalisation and transformation steps. Fast forward through our learning curve and we now load Powerpivot data via MySQL and use the 64-bit version of Excel (yes, sorry you’ve got to jettison the trusty old VB scripts, Outlook compatibility, etc on 1 box – but it is worth it!). We can now interrogate 10 million records, in a fraction of a second, on a standard desktop. We are yet to fully exploit Microsoft’s DAX language capabilities, but we were able to learn much about this (vast and rich ) dataset by conducting basic exploratory data analyses in Powerpivot. Currently because of our ‘history’ with it, we clean and transform data in Python (which again is free), but I can imagine a time when we’re cleaning and transforming data mostly using DAX. To be clear, we’re not confident at this point the above process can scale beyond 25 million records, but recognise we’re currently processing 10 million customer records in a fraction of a second, on a $700 desktop box. I’m not for one minute suggesting this is a ‘big data’ solution, but there is huge scope for companies to transform their businesses through analyses conducted on datasets up to 20 million records, without any material CAPEX or OPEX outlay.
Enter R and the modelling component, and you’ve all seen this film before. Most of you from A1 Sydney have seen the truly amazing AFL modelling system our Head of Analytics, Tony Corke, has created on his own (http://maflonline.squarespace.com/; the A1 presentation will be replayed at SURF next Wed – http://www.meetup.com/R-Users-Sydney/) . I really encourage you to read Tony’s post (http://maflonline.squarespace.com/mafl-stats-journal/2011/8/3/predicting-the-home-teams-final-margin-a-competition-amongst.html) where he uses R’s carat package to test the out of sample fitting performance of 54 different algorithms. Ponder for a couple of seconds what we might be able to learn from a table like this …
The end result is that our client enjoys the ability to predict weekly volumes for their most profitable product to within +/-3% (in a marketplace where weekly volume fluctuated +/- 11% over the past 3 years). Expressed another way, our client has a highly commercially exploitable model which explains 85% of the weekly variability in their most important product (which makes a profit contribution in the hundreds of millions pa), without over-fitting. All models were estimated in R; all exploratory and explanatory data analyses were undertaken using a plurality of open source software and Powerpivot.
What level of CAPEX and risk exposure was incurred? Our total hardware costs were $700; our software costs were $0; but consistent with all the A1 presentations I’ve been privileged to see, we invested all of our client’s budget in tirelessly trying to understand the data – it’s semantic structure and patterns. Like the vast majority of what we do at NTF, and what I see in A1 practitioner presentations: this problem yielded to perspiration, not inspiration.
So to the central questions posed at the outset- if Microsoft are serious about Powerpivot, yes it is absolutely a game changer (please excuse my timidity, it comes from the remembered pain of having the Google Wave hook and sinker deeply embedded in my oesophageal tract). Powerpivot allows corporate decision makers immediate data access, without waiting in queues for SQL programming resources. Smart, numerate people without programming skills (e.g Finance grads) now don’t need to know about indexing or how to code ‘joins’, they just link fields from different datasets via a mouse click. Can Powerpivot change the dynamics of information accessibility? Absolutely – I have no doubt based upon our experience to date. Powerpivot is not there yet – it is still prone to crashing (particularly before you’ve learned some basic tricks – two of which are pointing to a MySQL database and using the 64-bit version Excel [a free but painful download process]). However, for a first release, it is an impressive execution of an even more impressive vision.
I just hope Microsoft, with all its competing priorities, appreciate what they have.
Most of the literature on why BI projects fail tends to put some variant of ‘a lack of communication between IT and the business’ near the top of the list of culprits. This implies a relationship between two parties prone to friction. Either IT or ‘the business’ can own the BI initiative, but each needs the other in order for it to progress. In crude terms, the business has information needs; IT has data and the electronic infrastructure required to access it, transform it, and publish it as information. The two parties don’t speak the same language so each has to second-guess the other. Each is also required to generalise the multiple views of its members into a unified position for representation and negotiation purposes. The result is an equilibrium conception of the information needs of business consumers which may only poorly approximate each individual user’s actual needs.
The data needs of analysts are, by comparison, easier to serve than the information needs of business consumers. Analysts want data in closer to its raw form, and are less reliant on others for its transformation into packaged information. There are fewer representation and negotiation steps between demand and fulfilment.
Related Analyst First posts:
Follow Us!
