Currently viewing the tag: "big data challenges"
Technology Spectator recently published an article highlighting the need for big data speed …
This article highlights the communication challenge for accredited A1 professionals.
We all recognise Analytics is about using information better than competitors, so we are: 1. doing things better, and 2. doing better things than competitors/relevant comparators. But like so much of the coverage of our sector, the article focusses solely on Operational Analytics, not the latter area of Strategic Analytics.
Secondly, the article fails to recognise speed is only one part of the equation.
Taking the author’s example of the retail sector, sure real time analytics can detect an early decline in sales for a particular product, controlling for some extraneous factors. But a retailer’s promotional response (who they target and how) doesn’t necessarily require real time analytics (they can apply in real time outputs from models created last week, with little risk of degradation).
The most important questions for shareholders of the retailer require Strategic Analytic capability: how should pricing across the entire product portfolio be optimised?, what products should we be ordering now for next season (or the season after)?, how to optimise the physical network and supply chain? These strategic questions demand the right answer, not necessarily the fastest answer.
Any experienced industry professional gets that making sense of data is our primary role. But clearly interpreting data to the best of our ability flies in the face of throwing away information (e.g. because inconsistencies in the available data makes the task more cognitively complex). No one would advocate storing and processing data which possesses no incremental information value, but information value can be measured, so that shouldn’t be an issue.
Critically this article fails to recognise many of the barriers for Australian companies in effectively using their data relate to data quality, not their data storage and processing capacity.
Finally, there is no explicit recognition of the talent required to use data more effectively than your competitors.
From an A1 perspective we should welcome the growing focus on our sector, but we need to better articulate the more nuanced (and interesting) story of Analytics in an A1 Practice. It would be easy to criticise the journalist for being naive in swallowing the line of vendors and other vested interests, but the responsibility is ours to better explain the reality.
Eugene is totally right that we need to stand with a united voice. From today, NTF with publicly back A1 in all our proposals and marketing collateral. I regret not taking this action sooner.
I’ve argued before that most businesses continue to struggle with query and reporting, and that the increasingly common invocation of big data and social media in business cases for Business Analytics is a way of leveraging hype in order to fund second and third chances at getting these basics right. Beth Stackpole, writing at SearchBusinessAnalytics.com, quotes Brian Hopkins of Forrester on a related aspect of this phenomenon:
“Over the last 20 to 25 years, companies have been focused on leveraging maybe up to 5% of the information available to them,” said Brian Hopkins, a principal analyst at Forrester Research Inc. in Cambridge, Mass. “Everything we didn’t know what to do with hit the floor and fell through the cracks. In order to compete well, companies are looking to dip into the rest of the 95% of the data swimming around them that can make them better than anyone else.”
Stackpole’s thrust is that big data presents fundamental technical challenges to established data management practices:
This whole notion of extreme data management has put a strain on traditional data warehouse and BI systems, which are not well-suited to handle the massive volume and velocity requirements of so-called big data applications, both economically and in terms of performance.
Sources of big data identified in the piece include:
[T]he constant stream of chatter on social media venues like Facebook and Twitter, daily Web log activity, Global Positioning System location data and machine-generated data produced by barcode readers, radio frequency identification scans and sensors.
Social media, the Web, and GPS are data sources we’re conscious of interacting with everyday. It’s transparent to us as users of social media sites and smartphones that we’re consuming and generating digital information. The social processes which generate big data are more obvious and more novel to us than the business processes which generate ‘small data’.
Part of the significance of social media from a Business Analytics perspective, then, is that it has made business consumers of information far more aware of how much data they’re not able to get their hands on. The 95% that Hopkins characterises as having fallen through the cracks in the past did so less visibly. In the age of the smartphone, however, it’s much clearer how little data makes it into warehouses relative to what’s out there.
Seen in this way, big data is a two-edged sword for BI managers. It increases awareness and demand for information, and can breath new life into data management initiatives, but it also increases the dissatisfaction of business consumers because they’re now more conscious, and more frequently reminded, of what they’re missing.
Related Analyst First posts:
The McKinsey Global Institute has recently (May 2011) released a comprehensive report (156 pages) entitled “Big Data: The next frontier for innovation, competition, and productivity”. It contains good news for Business Analytics practitioners, analytically literate managers, and proponents of the Analyst First approach:
A significant constraint on realizing value from big data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from big data… Furthermore, this type of talent is difficult to produce, taking years of training in the case of someone with intrinsic mathematical abilities. (p.10)
That said, the report is best summarised as a restatement of the standard business case for Business Analytics, but using the phenomenon of big data as an organising principle. So much so that if you replaced “big data” with “Business Analytics” throughout you would end up with something very similar to Tom Davenport’s ‘Competing on Analytics’ thesis from 2006. Take, for example, the penultimate paragraph from the Executive Summary:
The effective use of big data has the potential to transform economies, delivering a new wave of productivity growth and consumer surplus. Using big data will become a key basis of competition for existing companies, and will create new competitors who are able to attract employees that have the critical skills for a big data world. Leaders of organizations need to recognize the potential opportunity as well as the strategic threats that big data represent and should assess and then close any gap between their current IT capabilities and their data strategy and what is necessary to capture big data opportunities relevant to their enterprise. They will need to be creative and proactive in determining which pools of data they can combine to create value and how to gain access to those pools, as well as addressing security and privacy issues. On the topic of privacy and security, part of big the task could include helping consumers to understand what benefits the use of big data offers, along with the risks. In parallel, companies need to recruit and retain deep analytical talent and retrain their analyst and management ranks to become more data savvy, establishing a culture that values and rewards the use of big data in decision making. (p.13)
A number of challenges to realising value through big data are identified in the report. It does not, however, note the critical fact that most organisations continue to struggle with small data.
This omission has implications. For example, the first chapter, ‘Mapping global data: Growth and value creation’, estimates the data generated by various business sectors by way of storage aggregates. It goes on to estimate different sectors’ ‘intensity’ as a factor of the concentration of this data in order to argue that the greater the number of firms in a sector, the more dispersed the big data, and therefore the fewer the competitive spoils on offer. It is here that the focus on big data gets in the way of a deeper point: organisations have been struggling with Business Analytics for many years and for many reasons, none of which have historically included big data. Moving into a big data world is only going to exacerbate already existing challenges, and if they are perceived to be ‘big data challenges’ they will not be addressed effectively.
This isn’t to say that big data doesn’t present any new challenges – it certainly does – rather, that their novelty may unhelpfully mask longer-standing and more fundamental ones. There is no reason to think that sectors in which there are many players are poorly positioned to take advantage of Business Analytics (discrete and process manufacturing are offered as ‘low intensity’ examples in the report). To the contrary, basic economics would predict that they would be more competitive and therefore have greater incentive.
The opening chapter also duly notes the trends driving big data: the Internet, multimedia, sensors, RFIDs, mobile phones, social media, and so on.
Chapter 2, ‘Big data techniques and technologies’ provides a useful non-exhaustive glossary. It includes definitions of some technologies specific to big data (Cassandra, Hadoop, MapReduce). Most of the technologies and none of the long list of techniques, however, are big data specific nor dependent. One or two are contemporaneous (crowdsourcing). The report does note that “not all” techniques require big data, and that bigger data sets are, for analytical purposes, generally better than smaller ones. But again, the importance of ‘big’ relative to ‘data’ is overstated.
Chapter 3, ‘The transformative potential of big data in five domains’ makes the case for Business Analytics (under the guise of big data) as it is being – and could further be – applied to US health care, EU public sector administration, US retail, global manufacturing, and global personal location data. Each section looks at available data, industry composition, economic and competitive factors, and then presents a range of viable analytical applications ranging from nascent to common practice in terms of maturity. Notably absent here – unsurprisingly – are arms race sectors: those for whom Business Analytics is central to competitive advantage (financial trading, e-commerce, the Internet more generally, and Intelligence, for example). Algorithmic trading is mentioned in the context of stream processing in Chapter 2 (p.33), but Business Analytics is not much explored as a core business function.
The sector-specific applications presented in the report are, furthermore, generally operational and IT-intensive in nature. The potential for Business Analytics to be a primary lever of strategic and tactical decision-support, and its key function as an exploratory, sense-making activity, are not given the attention they deserve. These possibilities are implicit in the report’s comprehensive analysis, however they are systematically obscured by a pervasive bias: existing business models are pictured being made more efficient at the margins through operational analytics being grafted on to existing processes (cross-selling, various kinds of optimization, supply chain management, leaner manufacturing, sales support, and so on), and startups based on new business models made possible by big data are envisaged. What is not envisaged is the opportunity – and the potential – for existing businesses to strategically adapt using Business Analytics.
Chapter 4, ‘Key findings that apply across sectors’, summarises both the sources of value on offer from big data (read: Business Analytics) and various impediments to its realisation (skills shortage, data and technology access, data policy inadequacies). Towards the end it makes a critical point regarding industry structure which begins to get at some of the core challenges to Business Analytics. Unfortunately the insight limits itself to sector level generalisations:
Sectors with a relative lack of competitive intensity and performance transparency and industries with highly concentrated profit pools are likely to be slow to fully leverage the benefits of big data. The public sector, for example, tends to have limited competitive pressure, which limits efficiency and productivity and puts a higher barrier up against the capture of value from using big data. US health care not only has a lack of transparency in terms of the cost and quality of treatment but also an industry structure in which payors gain from the use of clinical data… but at the expense of the providers… from whom they would have to obtain those clinical data. (p.108)
Principal-agent problems and various sources of inertia (commercial, bureaucratic, cognitive, regulatory) are in reality common features of any sizable organisation, public or private; thus the unforgiving measurement and transparency that Business Analytics can’t help but bring are so often resisted. The difficulty of building the necessary ‘human infrastructure’ within this context – combining roles, skills, relationships, trust and culture with supporting electronic infrastructure – should not be understated.
Rounding the report off, the final chapters, ’5. Implications for organization leaders’ and ’6. Implications for policy makers’, are a series of action pitches to prospective decision makers reflecting the SWOT analyses detailed in preceding sections.
I am indebted to an earlier and excellent summary of the McKinsey report by Steve Miller at Information Management – which again, if you substitute “Business Analytics” for “big data”, reads like Davenport.
Follow Us!
