Guide To Implementing Business Intelligence - 2. Merging And Managing Data

It is often assumed that a data warehouse is imperative to any business, normally because IT departments don't want to report directly off their source systems and they decide they need a data warehouse.

Building a data warehouse can be a large and costly project so you need to consider what real benefits it could bring to your business, you need to think carefully about why you're doing it and what the purpose is.

Maybe you can report directly off your source system by improving the capacity of that server or that hardware you're using. By duplicating and just reporting off it, you'll very quickly get management information out to people and it might be adequate, further more, by doing this you'll learn your data, and understand the pitfalls in it.

This means that you've already solved a lot of the issues and problems if you do decide to build a data warehouse in the future, so when you out grow reporting directly off your operation systems you can then evolve quite easily because you now have more knowledge into building a data warehouse.

One of the biggest challenges you're likely to face with business intelligence when it comes to ensuring you merge prospective customers is making sure your data is merged and collated to give you the right information at the right time. Your customer relationship management (CRM) system may know who your customers are, but it might not know when one of your customers transacts with you.

This can become even more problematic if you have several transactional systems and channels to market. For example, if you were to take a football club that may have it's season ticket holders, then turn style ticket holders and a shop. You may not know that a customer brought something from the shop even though they've got a season ticket, so you'd use a data warehouse to pull all that data together giving you a single view of the customer that can be sent to the CRM solution and the marketing campaign can be targeted accordingly.

The problem often lies in ensuring the CRM doesn't already have that information and duplicating it, as most CRM systems will contain duplicate data anyway (with different titles, names, addresses etc), you need to ensure the tools you are using can integrate that data effectively from the data warehouse into the CRM solution and vise versa as the data comes back.

Another major obstacle your business may encounter when implementing a business intelligence solution is user adoption - getting your staff to actively analyse the data you're producing and using the data gathered.

Just because the IT department has brought some new tools, doesn't mean the rest of your staff are going to use them, they might not have the time, they might not see the benefit to them or they simply may not understand it. This means as part of implementing a new BI solution, you need to understand your users and deliver the information to them in a way that's going to make them want to use it too. This needs to be looked at on an ongoing basis, you need to always be aware of what's being used and what isn't, if it's not being used you need to know why, if it's simply a case of staff not having the time or understanding it, you need to invest the resources to rectify this, or if it's genuinely not working you'll need to find an alternative solution. Once your users see this technology can make their job easier, you'll start to see the value in your investment.

This always works from the opposite perspective too, if one of your users are told any requests will take several weeks or months to action, they will soon stop asking and the technology wont be being used to it's full potential.

Data quality will always cause risk because you're pulling data from several systems and from several years and this means the chances are the data has been captured in different ways.

There are several steps you can take to ensure the risk is as minimal as possible starting with exposing the data to the end user as soon as possible because it's their data and it's them who are going to be able to resolve whether the data is right or wrong.

There are plenty of data quality tools that can help when it comes to automating and cleansing the data that's been gathered. These tools can help merge and de-dupe the data and they can use fuzzy look up logic and look up to catalogues. If you have units of measurement across your product range, these units can be standardized and converted to clean your data automatically.

The main challenge here is making sure all the data is standardized and documented as soon as possible. The different business rules are documented and shared and this all falls under the umbrella of a data governance programme to be managed, coordinated, and governed.

In most companies it will be the finance department that looks after the planning and budgeting using Excel, meaning it's done outside the core IT systems. The planning and budgeting is a key part of your information management through monthly or quarterly forecasting which is often feed into the management information systems.

These two systems ideally should both be incorporated into the business intelligence solution because it's a key input. This means the data warehousing, business intelligence reporting process, month end reporting and planning and budgeting are actually interlinked and as such you need to structure your programme and you business intelligence deployments taking them into account.

When we're doing data warehousing and pulling data from many sources, you often have the same data coming from multiple systems causing issues with duplication and inconsistencies. The data quality, customer data or product data often has to be merged into a master set, often the data warehouse becomes the master data management product. It's often ideal if you can manage you master data, and put it in as part of your data governance programme early on in the project then that will help your BI deployment later on. This means the your master data will be known, you'll know who controls it and the procedures and principles around managing it.