Enterprise Content Management:
Vendor Evaluations, Part 2: Designing an In-House Evaluation
This column continues last month's discussion of evaluating enterprise content management, "Vendor Evaluations Part 1: Functional Requirements." At the end of the column, I recommended an in-house evaluation as the best method to understand how well an enterprise content management (ECM) application would work in your organization. Now I will outline how to design and implement such an evaluation.
The evaluation plan should include at least six core areas: vendor assessment, product functionality, architecture, security, integration and ease of use. The importance of each of these will vary with your requirements and should be weighted accordingly.
Vendor assessment should include checking customer references with similar functional requirements. Do not assume others in your industry will have the same needs - they may be tackling different problems. Similarly, it is not necessary to require a reference from your industry if you are working on basic content and document management issues. Do get industry references if you are looking into categorization, information extraction or other language-dependent operations. Performance of these tools can vary widely depending on the nature of the content. For example, identifying protein names is a particular challenge in the pharmaceutical industry.
In addition to the usual financial considerations (initial license cost, maintenance fees and consulting rates), assess the vendor's viability. There are many start-ups in the enterprise search and categorization markets, and not all will survive. We saw significant consolidation last year with well-known and respected companies such as Quiver, Inktomi and Semio being acquired. Other big names in the market have scaled back or continue to lose money. Picking a long-term success is not a day at the races, but we should understand a vendor's position in the market and past performance to minimize the chance of surprises.
Test product functionality against your specific requirements. This means installing the product at your site if you will host the system yourself or running it off site if you will use an application service provider. Define a test plan that measures the effectiveness of the key features you expect. We all understand that search and categorization tools are not mind readers, and they never give us exactly what we want. Thus, it is important to determine how close they come to the ideal situation. Will results in hand press the vendor's technical staff to describe what is required to improve results? For example, can you specify that particular links appear in the search results list for specified terms? What is required to develop a gateway to your document management system to improve performance? Compile a test data set that reflects the full range of content you will use in production. If you will be indexing content in document management systems, network files systems, customer relationship management (CRM) applications or external Web sites, be sure they are included in the test.
The test plan should be divided into categories, such as content process, search, categorization, personalization and security. Within each category, list individual test elements, such as ability to index PDF files and effectiveness of access controls in a content repository. Weight each test element according to its relative importance, and use the combined score to calculate the category score. In turn, weight each category and combine the results to derive the final score for a product. This approach allows flexibility in developing test criteria without skewing the results to categories with many test elements.
During the evaluation, assess architectural compatibility and security factors. Key considerations are how the application will integrate with your portal and whether it will work with an existing or planned single sign-on system. Because most enterprise content management systems require access to distributed content, evaluating security mechanisms can be the most time-consuming part of the evaluation. Be sure to understand how distributed security mechanisms work. For example, are access control lists synchronized between repositories and a centralized search system or does one system delegate security services to another? One approach is not necessarily better than the other, but the implications should be understood. When synchronized access control lists (ACLs) are used, for example, changes in one system are not reflected in the other until they are synchronized again. This will influence the frequency of those operations.
Integration and ease of use are especially important when integrating an enterprise content management system in a portal. Often ECM systems provide services across a range of applications. During evaluation, determine the level of effort required to integrate these services. Does the vendor provide an application programming interface (API) such as a JSP or ASP interface suitable for your environment? Include both end users and developer needs in ease-of-use considerations.
For more information on related topics visit the following related portals...
Enterprise Intelligence and
Dan Sullivan is president of the Ballston Group and author of Proven Portals: Best Practices in Enterprise Portals (Addison Wesley, 2003). Sullivan may be reached at email@example.com.
Provided by IndustryBrains
|Autotask: The IT Business Solution|
Run your tech support, IT projects and more with our web-based business management. Optimizes resources and tracks billable project and service work. Get a demo via the web, then try it free with sample data. Click here for your FREE WHITE PAPER!
|Revenue Maximization Software|
Maximize, protect, produce and analyze revenue by consolidating customer data from multiple disparate sources across the enterprise and associating it with real-time operational data for a complete and unique view of your customers and business.
|Life Cycle Management - Share Cad With Everyone|
Share Cad projects with non-tech users. Get everyone on the same page. Request free information.
|Design Databases with ER/Studio: Free Trial|
ER/Studio delivers next-generation data modeling. Multiple, distinct physical models based on a single logical model give you the tools you need to manage complex database environments and critical metadata in an intuitive user interface.
|Grid Computing: Cut through the hype|
Free Forrester Report discusses developing trends in grid technologies, breaks down the three major ways grid is being used and provides survey results from organizations that are piloting, rolling out, or currently leveraging grid technology.
|Click here to advertise in this space|