What are the successful processes that are followed in the industry for task estimation of a data warehousing project ?

Danette McGilvray's Answer: The goal of the task estimation process is to determine the tasks needed to complete the project, estimate effort, and create a draft schedule for completion. Note: The task list is also known as a project plan, a work breakdown structure, or a workplan. When developing the initial task list and timeline with a project team, it is helpful to use a facilitator. The facilitator expedites the process so the project manager and team members are free to focus on determining the actual tasks and effort needed. Depending on the size and scope of the project, you may have more than one project team creating their own project plan for their specific deliverables. The overall project manager then brings together the plans into a master plan. The approach below works whether you have one team or several teams.

Caution: Do not expect this plan to be set in stone. The plan will continue to change as the project progresses; new information is available; scope, resources, and priorities change; deliverables are (or are not) completed on time, etc. The process of estimating and modifying your plan should be repeated many times throughout the project. Even the initial plan could take several iterations before you have gathered enough information to create a useable plan.

I will briefly explain a process I have used successfully for many projects.

Determine the general approach, high-level phases, and the methodology to be used in the project.

• This is usually done by the project manager and team leads. You may be using a methodology provided by a vendor, other experts, or creating it yourself.
• For example, high-level phases might include Plan Project; Gather Data Requirements; Develop Data Model; Design and Develop Physical Database; Source, Profile, and Map Data; Extract-Transform-Load Data; etc.
  

Gather the project team.

Manual technique:

• For each high-level phase in the project, follow the steps below to develop your plan.
• One technique is to use large sticky notes or index cards and masking tape. Have team members write tasks on the sticky notes and place on the wall or white board. Use one sticky note per task. It is easy to move the notes as you organize the order. Each sticky note will include as much information as you have at the time: task, owner, time estimates, and dependencies.
• Once the manual work is done, transfer the information to your project tracking software (such as Microsoft Project).
  Automated technique:
• If you already have a fairly detailed methodology, you can enter the tasks into your project tracking software prior to the planning meeting. Breakdown or create additional tasks by entering the information directly into the software.
• Caution: It is easier for the team to understand the project as they see it develop visually using the manual process. It is more difficult when creating tasks and estimating times using only the software.
  Benefits:
• Benefits from creating the timeline manually first with the project team include:
  1) tasks, effort, and dependencies are visible to all team members, 2) team has a greater understanding of and commitment to the project, 3) team members have an opportunity to get to know each other and set the foundation for working together. This is particularly important if the team is dispersed geographically and will not be face-to-face throughout much of the project.

Develop task list and description.

• Review the project and business objectives, constraints, and the high-level phases.
• Determine tasks needed to complete each phase.
• Keep the description simple. A verb-noun form works well, e.g. interview users, document requirements.
• Continue to decompose the tasks until they have rough durations of 2-20 days.
• Break down the tasks only to the level of detail that you are willing to track.
• Include key checkpoints or milestones as tasks to be completed. A noun-verb form works well for milestones, e.g. requirements completed, data model completed.

Identify owner for each task.

• For each task, identify a single owner who will be responsible for the task deliverable.
• Other people may help with the task, but the owner is the one accountable for ensuring the task is completed on time.

Estimate times.

• Time can be estimated as effort and/or duration. Effort is the time needed to complete the task. Duration is the elapsed time in number of work days before the task will be completed. For example, it may take 20 hours of effort over a duration of two weeks to complete a task. Determine which type of time estimate to use. It may be useful to capture what you know about both types while building the initial timeline. Some techniques for estimating time include using historical data, actual experience, experts, rules of thumb, known formulas, segment tasks into more detail, or actually do part of the task, keep track of the time, and extrapolate your estimate.

Note any dependencies.

• Describe any constraints or dependencies that impact starting or completing a task, e.g. data model must be completed before physical database is developed.

Organize the tasks.

• Place the tasks on the wall into a rough order of dependence. Group associated tasks into major categories of work.
• Filter the tasks for any duplicates.
• Number the tasks. The high-level phases could be 1, 2, 3, etc. Tasks under each phase would be sequenced as 1.2, 1.3, or 2.1, 2.2, 2.3, etc.

Create a timeline.
Manual:

• Write time periods across the white board.
• Use the task time estimates to determine the earliest start and finish dates for each project task.
• Arrange tasks according to time needed and dependencies.
• Link the sticky notes on the board by drawing in the dependencies.
• Transfer the information into your project software.
• Automated:
• Transfer tasks, time, and dependencies into your project planning tool. The software will help you create a timeline with due dates factoring in resource availability.
• Use your project plan to track progress. Review and modify your estimates and keep your project plan updated throughout the project.