I've had a few quiet days with the onset of the holidays and that has given me time to read and write. It's given me time to think. About what? Well, a colleague of mine asked me to think about ways to evolve our approach to capturing requirements and doing analysis for our business intelligence projects. We have a dedicated group that delivers outstanding EDW and BI initiatives and this colleague wanted me to see what tools/techniques from my advisory services practice could be leveraged in our BI work. I've been thinking about this since he mentioned it to me late last week and wanted to get some thoughts down to help evolve my thinking.
To do so, let's look at something unrelated to technology. Building a house. When building a house, you spend a considerable amount of time capturing requirements. Once you are done, you had those requirements over to the various trades people and they go off and build your house. This makes sense to us and it lends itself into the tech world to application development or web development. When I am engaged by clients to build websites or web-based applications, we often spend a lot of time capturing the requirements. It is absolutely critical. Like building a house, once you've captured those requirements though, they are handed off to graphic designers, developers, testers and away they go. They build what was needed based on the requirements.
This approach however doesn't apply to business intelligence initiatives. I've witnessed this first hand on projects I've observed. It is often complex to capture requirements, and that can be true of web development projects as well. The difference, however, is that BI requirements seem to be a constantly moving target throughout the project. That makes it challenging. It would be like handing blueprints to the trades people building a home, and then, once they have the framing done the unexpectedly find out that the physical properties of lumber have changed on them and they now need to reconsider things. That of course doesn't happen when building a home, but in BI projects, this kind of shifting does happen. The question is why?
As I puzzled on this I recalled a wonderful article that Peter Thomas wrote where he used an excellent metaphor for BI projects, equating them more to an archaeological dig than building houses. The point of his article is that as you begin an archaeological dig you find unexpected things under the ground so to speak. No amount of requirements gathering is going to tell you what you will find once you start digging down into the data.
I liked Peter's analogy when I first read it and I've seen this happen repeatedly over the past 4-6 months on BI projects. As I sit here this evening as ponder the question that my colleague asked me to assist with, a few things need to be written down as guiding principles to start off on a broader discussion (for later posts and ultimately, for the evolution of our own methodology).
Firstly, requirements on BI projects will change. Go back and read the underlined, bold statement again. I'm not saying that the requirements might change. They absolutely will change. It is the very nature of BI projects that requirements will change. Knowing this, two things should be done. 1) Plan for that change upfront and 2) use a just-in-time (JIT) approach to capturing requirements. By planning for the change, I mean, make it a part of your project estimating and planning process. Once you are done your traditional project planning process, look at your plan and identify milestones in the project where you think new information will be revealed to the team. There are some things you just won't know until you are into various parts of the project. Plan to add time and resources to the project at these junctures so you can re-visit and re-capture requirements. If you don't build this iterative process into your requirements gathering, your project will undoubtedly go off the rails. Point number two, use a JIT approach, is important if you are going to iterate quickly to keep pace with the nature of BI projects. Due to the exploratory nature of BI projects, it makes little sense to gather requirements and model the details at the start of the project. As I said above, the requirements are going to change, so why model things knowing they are going to change? If you wait until later in the project, you and your team will have more information that is revealed to you during the course of the project. For example, if you aren't going to be implementing a certain requirement till you are several months into the project, then you will have several more months of information if you brainstorm the requirements at that point in time rather than doing so at the onset of the project. Having several more months of information will allow you to ask better questions based on what you know. If you don't follow this approach, anything you've spent time on modeling at the start of the project will have to be re-visited and that will result in wasted time & project resources. A good article I read seems to support this idea of wasted effort in BI projects.
Secondly, insist on delivering smaller chunks of functionality in shorter periods of delivery time. Things always seem to work perfectly when you are doing them in Powerpoint, or sketching them on a whiteboard or modeling them in ERWIN, but as it is said in one of my favorite books, "The proof of the pudding is in the eating." In other words, the proof is in the code. You won't know what you don't know until you start delivering and running code. This of course isn't a new idea. If you take the guiding principles of something like theAgile Unified Process (AUP), it drives toward building a complete, end-to-end "skeleton" of your working system. To do this for EDW / BI projects, that means chunking down the scope. It is along the lines of our own Rapid Results approach that we embrace at Project X and would allow us to learn things along the way that would iteratively feed into the process of modelstorming requirements in a JIT fashion.
Thirdly, and lastly for this post, I believe that these types of projects need a centralized repository for project knowledge and documentation. A tool must be used to capture requirements to make for a more effective analysis of those requirements. I have too often seen information for the project living in disparate documentation: design specification documents, inside comments of store procedures, within emails, within production guides, within deployment plans. The list goes on and on. Much of this information is duplicated over and over again. Change it in one spot and it must be tracked down and changed in other spots. This is hugely inefficient and introduces multiple opportunities to make mistakes. I think that a centralized database needs to be used right at the start of the project for capturing and documenting requirements. This database should continue to be used throughout the project to capture other project details. Of course, the other documents, such as production guides, deployment plans, etc., can all exist. They are documents that are required by different stakeholders. They can and must exist, but they should be generated based on info from within the centralized knowledge repository. Without this type of approach, it becomes hard to iterate quickly and in a just-in-time fashion because previous project knowledge is not easily accessible, easily analyzed and easily updated.
Based on these initial musings, there are several things I'm setting out to develop in the next short while.
1) Refined way of scoping, estimating and planning BI projects to ensure we build into our approach an iterative cycle for capturing requirements
2) A JIT approach for requirements capture and developing model details that allows for a team to modelstorm when the need arises, rather than forcing that to be at the start of the project
3) A centralized tool for capturing requirements and ongoing systems documentation.
Stay tuned....as I develop these intellectual assets, I will blog my thoughts. As always, I would welcome any input you might have.
Thanks for the link and the kind words - I agree with the points you go on to make, particularly the inevitability of some requirements only manifesting themselves once a prototype has been developed, or once a chunk of the system has gone live, or even after it has been in use for six months. People charged with managing BI projects should expect this type of thing and make allowances. One such allowance is not spending aeons trying to get every i dotted and t crossed before starting to cut code.
Peter
Posted by: Peter Thomas | January 19, 2010 at 08:01 PM
Thanks Peter for the comment/feedback. I really support your approach / idea in terms of cutting code sooner rather than later and learning from the process as opposed to waiting till things are "perfect". I'm working on some other posts and thoughts around this as I help shape things for our Business Intelligence practice. As always, I'd welcome your continued feedback on this.
Posted by: mip | January 20, 2010 at 11:09 AM