The perennial problem

Respondents to the survey indicated that they want BI applications that can easily integrate with major enterprise applications from the likes of Oracle and SAP. Do smaller or niche BI software vendors have a track record of problems when it comes to integrating with such applications?

[Rick] Sherman: You know, it’s funny if you think about 10 years ago versus now. Ten years ago, the smaller vendors didn’t have access to — and there wasn’t as much knowledge as to what was in — SAP or Oracle apps. But, especially with services and SOA and everything else coming out, the ability to access the enterprise applications has gotten easier and easier. So I really don’t think that’s as big an issue today.

That’s from SearchBusinessAnalytics.com on the results of their March 2011 survey. The interview is with Rick Sherman, “the founder of Athena IT Solutions, a Stow, Mass.-based firm that provides data warehouse and business intelligence consulting, training and vendor services.” Sherman speculates on what might have caused the increase in concerns about data integration problems which the survey brought to light:

I think what’s new is the fact that the [data integration] issues are more visible and more people have access to BI or are trying to do reporting, analytics and BI than ever before. It isn’t that the problems are new. It’s that they’re more visible because more people are encountering them. The other point [from] a business user context [is that they are] initially trying to do reporting and analysis from an existing operational application [and it's just one] source. [There] might be data quality issues, but they do not have to integrate data, because they’re getting it from one source. As soon as you start doing that, you start needing to get data from other applications, and that’s when you start encountering more data quality issues and then more data integration issues.

The distinction between data access, data integration, and data quality is a useful one. Sherman is arguing that data access is a problem of the past, and that the real contemporary challenges are in data integration, not access.

I think that what happens [is that at] the larger firms you have SAP and then you have all the enterprise apps that Oracle has acquired over the years. You’ve got two major spheres of application knowledge that you have to have. But when you get down to the SMBs [small and medium-sized businesses], there are hundreds of enterprise apps, such as financial apps geared toward smaller firms and, more importantly, you start getting into industry-focused applications. [The challenge for SMBs is that they] have a lot more BI apps that might not have as much knowledge as to how to access their [enterprise apps] or those enterprise apps might not be as open as SAP and Oracle are. SMBs also have the issue of figuring out how to integrate with a lot more sources than if you’re talking about larger firms.

But, seemingly at odds with this, survey respondents indicated “that they want BI applications that can easily integrate with major enterprise applications from the likes of Oracle and SAP.” And reading that closely, what they mean by “integrate with” is “access”. So what’s going on?

Sherman addresses the problem in terms of technology (tools) and know-how (knowledge). In fact, there are at least two more dimensions to it. One is complexity and the other is politics.

Yesterday’s post challenged the assumption that the market growth of emerging BI vendors must be coming from small to medium businesses, pointing out that it might also be coming from autonomous business units within larger organisations. Sherman appears to be making a similar conflation – implying, because larger organisations run enterprise systems from Oracle and SAP, that they don’t have anything else. It’s certainly the case that, as he says, smaller businesses run niche applications geared towards their needs (and budgets), but it simply doesn’t follow that large businesses only run enterprise applications. I would expect the opposite to be true. Large organisations become large in part through acquisition, and large organisations have more moving parts. Large organisations are more complex. I would expect them to be running more systems, not less, many of them niche, heavily customised, and developed in-house. Sherman’s larger point is well taken: less common systems are harder to access, and the more systems in place, the harder the integration effort. But there’s no reason to think this problem has gone away for large organisations because they’ve purchased enterprise apps.

Enterprise systems are more IT-reliant, meaning more organisational functions between users and data, increased layers of internal policy compliance, additional bureaucracy, depersonalised communication channels, more dispersed knowledge, and as two previous posts have argued, divergent incentives.

The common thread running through all of this is that business systems have been consistently poor at making their data available to analysts in manipulable form. Routine access to verbose data – native, unsummarised and readily tractable – is a perennial problem. Verbose data at its most basic doesn’t need to be clean or integrated, just available. That is all many analysts need.

In technical terms this translates into a query and reporting deficit. Over the last decade or so I’ve watched as the same elemental query and reporting needs have piggybacked on a succession of ‘sexier’ requirement sets, such as:

• Business Intelligence
• KPIs, Dashboards and Scorecards
• Performance Management
• Analytics
• Business Analytics

Also attaching to various parallel technological trends:

• Enterprise Search
• Web 2.0
• Mobile Platforms
• Cloud Computing
• Social Media
• Big Data

It’s furthermore the case that query and reporting needs repeatedly merge themselves with the related but different objectives of various data management and infrastructural projects, for example:

• Data Warehousing
• Master Data Management
• Data Quality
• Data Governance

None of this takes anything away from any of the above disciplines, each of which tackles real and distinct problems. The point is simply that basic query and reporting remains a problem.

Related Analyst First posts:

• *Snapshot on the Business Intelligence Market*
• IBM CIO Survey 2011
• IT support
• Data is a spellword