Portals eNewsletters Web Seminars dataWarehouse.com DM Review Magazine
DM Review | Covering Business Intelligence, Integration & Analytics
   Covering Business Intelligence, Integration & Analytics Advanced Search

View all Portals

Scheduled Events

White Paper Library
Research Papers

View Job Listings
Post a job


DM Review Home
Current Magazine Issue
Magazine Archives
Online Columnists
Ask the Experts
Industry News
Search DM Review

Buyer's Guide
Industry Events Calendar
Monthly Product Guides
Software Demo Lab
Vendor Listings

About Us
Press Releases
Advertising/Media Kit
Magazine Subscriptions
Editorial Calendar
Contact Us
Customer Service

Enterprise Content Management:
Vendor Evaluations, Part 2: Designing an In-House Evaluation

  Column published in DM Review Magazine
March 2003 Issue
  By Dan Sullivan

This column continues last month's discussion of evaluating enterprise content management, "Vendor Evaluations Part 1: Functional Requirements." At the end of the column, I recommended an in-house evaluation as the best method to understand how well an enterprise content management (ECM) application would work in your organization. Now I will outline how to design and implement such an evaluation.

The evaluation plan should include at least six core areas: vendor assessment, product functionality, architecture, security, integration and ease of use. The importance of each of these will vary with your requirements and should be weighted accordingly.

Vendor assessment should include checking customer references with similar functional requirements. Do not assume others in your industry will have the same needs - they may be tackling different problems. Similarly, it is not necessary to require a reference from your industry if you are working on basic content and document management issues. Do get industry references if you are looking into categorization, information extraction or other language-dependent operations. Performance of these tools can vary widely depending on the nature of the content. For example, identifying protein names is a particular challenge in the pharmaceutical industry.

In addition to the usual financial considerations (initial license cost, maintenance fees and consulting rates), assess the vendor's viability. There are many start-ups in the enterprise search and categorization markets, and not all will survive. We saw significant consolidation last year with well-known and respected companies such as Quiver, Inktomi and Semio being acquired. Other big names in the market have scaled back or continue to lose money. Picking a long-term success is not a day at the races, but we should understand a vendor's position in the market and past performance to minimize the chance of surprises.

Test product functionality against your specific requirements. This means installing the product at your site if you will host the system yourself or running it off site if you will use an application service provider. Define a test plan that measures the effectiveness of the key features you expect. We all understand that search and categorization tools are not mind readers, and they never give us exactly what we want. Thus, it is important to determine how close they come to the ideal situation. Will results in hand press the vendor's technical staff to describe what is required to improve results? For example, can you specify that particular links appear in the search results list for specified terms? What is required to develop a gateway to your document management system to improve performance? Compile a test data set that reflects the full range of content you will use in production. If you will be indexing content in document management systems, network files systems, customer relationship management (CRM) applications or external Web sites, be sure they are included in the test.

The test plan should be divided into categories, such as content process, search, categorization, personalization and security. Within each category, list individual test elements, such as ability to index PDF files and effectiveness of access controls in a content repository. Weight each test element according to its relative importance, and use the combined score to calculate the category score. In turn, weight each category and combine the results to derive the final score for a product. This approach allows flexibility in developing test criteria without skewing the results to categories with many test elements.

During the evaluation, assess architectural compatibility and security factors. Key considerations are how the application will integrate with your portal and whether it will work with an existing or planned single sign-on system. Because most enterprise content management systems require access to distributed content, evaluating security mechanisms can be the most time-consuming part of the evaluation. Be sure to understand how distributed security mechanisms work. For example, are access control lists synchronized between repositories and a centralized search system or does one system delegate security services to another? One approach is not necessarily better than the other, but the implications should be understood. When synchronized access control lists (ACLs) are used, for example, changes in one system are not reflected in the other until they are synchronized again. This will influence the frequency of those operations.

Integration and ease of use are especially important when integrating an enterprise content management system in a portal. Often ECM systems provide services across a range of applications. During evaluation, determine the level of effort required to integrate these services. Does the vendor provide an application programming interface (API) such as a JSP or ASP interface suitable for your environment? Include both end users and developer needs in ease-of-use considerations.


For more information on related topics visit the following related portals...
Enterprise Intelligence and Content Management.

Dan Sullivan is president of the Ballston Group and author of Proven Portals: Best Practices in Enterprise Portals (Addison Wesley, 2003). Sullivan may be reached at dsullivan@ballstongroup.com.

Solutions Marketplace
Provided by IndustryBrains

Autotask: The IT Business Solution
Run your tech support, IT projects and more with our web-based business management. Optimizes resources and tracks billable project and service work. Get a demo via the web, then try it free with sample data. Click here for your FREE WHITE PAPER!

Revenue Maximization Software
Maximize, protect, produce and analyze revenue by consolidating customer data from multiple disparate sources across the enterprise and associating it with real-time operational data for a complete and unique view of your customers and business.

Life Cycle Management - Share Cad With Everyone
Share Cad projects with non-tech users. Get everyone on the same page. Request free information.

Design Databases with ER/Studio: Free Trial
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.

Grid Computing: Cut through the hype
Free Forrester Report discusses developing trends in grid technologies, breaks down the three major ways grid is being used and provides survey results from organizations that are piloting, rolling out, or currently leveraging grid technology.

Click here to advertise in this space

View Full Issue View Full Magazine Issue
E-mail This Column E-Mail This Column
Printer Friendly Version Printer-Friendly Version
Related Content Related Content
Request Reprints Request Reprints
Site Map Terms of Use Privacy Policy
SourceMedia (c) 2006 DM Review and SourceMedia, Inc. All rights reserved.
SourceMedia is an Investcorp company.
Use, duplication, or sale of this service, or data contained herein, is strictly prohibited.