http://msdn.microsoft.com/en-us/library/bb220797.aspx

Alchemy Business Overview

3 out of 4 rated this helpful - Rate this topic

Mark Baciak
Architecture Strategy
Microsoft Corporation

December 2006

Applies to:
Microsoft .NET Framework

Summary: Microsoft's Service-Oriented Architecture (SOA) for line-of-business applications, Alchemy, integrating line-of-business applications with a production-quality enterprise-service environment. (10 printed pages)

Overview
About This Paper
Introduction
Situation
Solution
Alchemy Common Platform System Overview
Key Features and Capabilities
Benefits
Lessons Learned
Conclusion
More Information

Overview

Problem Statement

Line-of-business applications have often been designed and built in an isolated and monolithic manner, making it difficult to share information between applications. Over a period of more than four years, Microsoft Information Technology (Microsoft IT) has addressed this problem, using Web services to integrate line-of-business application services and data. This paper provides an overview of the Microsoft IT solution.

Solution

Microsoft recognized the need for a consistent approach to connect this fragmented information, and developed a flexible, service-oriented architecture to address that need: Alchemy.

Alchemy is a Service-Oriented Architecture (SOA) for Web service management, developed for line-of-business use by Microsoft IT. Alchemy Web services were released in 2003, followed by the Alchemy Common Platform (ACP) in 2004. ACP is a protocol-agnostic framework for service development and management across the enterprise.

Benefits

Seamless and easy integration of applications
Reduced server-deployment costs
Accelerated adoption and reuse of Web services
Proactive management of service-level agreements

Lessons Learned

Adopt a service-oriented architecture.
Implement a governance body for architectural standards and strategy.
Use readily available tools and frameworks.
Line-of-business application owners should own their services.

About This Paper

This white paper is written for IT architects, developers, and support professionals who are interested in the design, deployment, and management of an enterprise SOA. This white paper will be of interest also to line-of-business owners and analysts who need to understand the issues involved in adopting an SOA in their organization. This paper assumes that readers are technical decision makers and are already familiar with the value and challenge of enterprise-wide service solutions.

Microsoft IT developed its ACP implementation as an internal solution. This white paper presents the business case of that solution, as a case study for customers who need similar solutions in their enterprise environments. This internal solution was implemented through Web services; and, although the Web service programming model was optimal at the time of development, customers should be aware that the Windows Communication Foundation (WCF) will offer additional programming models as part of Microsoft .NET Framework 3.0. Support for MSMQ, COM+, and Web Services Enhancements (WSE) will be included.

Note For security reasons, the sample names of forests, domains, internal resources, organizations, and internally developed security file names used in this paper do not represent real resource names used within Microsoft, and are for illustration purposes only.

Introduction

Prior to the development of the Alchemy solution, Microsoft began developing Alchemy services, an internal use of services to combine data from multiple line-of-business applications. The initial goal was to enable easy integration of business data from four core line-of-business applications at Microsoft:

Siebel for customer relationship management
Clarify for product service request management
MS Sales for sales revenue reporting
World-Wide Sales and Marketing Database for managing e-mail newsletter subscriptions, event invitations, and diverse marketing campaigns

Alchemy services supported read and update access to several additional business applications. Microsoft client- and server-based applications used Alchemy services as the Microsoft IT standard for accessing line-of-business application services and data.

Situation

The basic Web services standards (SOAP, WSDL, XSD, and UDDI) provide the functionality necessary to build and connect distributed applications. However, by themselves, the basic standards are not sufficient to support a production-quality enterprise environment for the development, deployment, and management of services. The second release of Alchemy services, the Alchemy Common Platform (ACP), addressed this situation, in addition to the following deficiencies:

Lack of common service-development standards
No common service-deployment and change-management processes
Need for a services operations management infrastructure

Microsoft IT decided that an enterprise-class Service-Oriented Architecture (SOA) based on Web services would address these issues. ACP added the deployment, management, and support features needed by line-of-business developers, operations staff, and line-of-business application owners.

Lack of Common Development Standards

Microsoft IT's early experience developing and deploying services was a common one: Each business unit application support group created and deployed services for its own particular application. Microsoft IT found, when they relied only on the basic services standards, that there were many different options for developers to implement:

User- and process-authentication
Digital signing and encryption of service requests and responses
Formatting of parameter data
Handling of error conditions

This broad range of implementation options resulted in incompatible security solutions, duplicated development effort, inconsistent management processes, and unpredictable service levels.

No Common Deployment and Change-Management Processes

The early deployments of services at Microsoft were made on an application-by-application basis; as early adopters, there was no enterprise plan yet for incorporating services into the production environment. This created a services environment that was difficult to manage from both operations and change-management perspectives.

It was not easy for a group of developers to take a service from development through testing to production. The development, testing, and deployment processes placed a heavy burden on the server infrastructure and personnel who manually deployed and configured the services. Additional servers were required, which resulted in increased hardware and operations-management costs. Furthermore, manual installation and configuration of application software produced inconsistencies that were difficult to detect, correct, and manage.

No Infrastructure for Operations Management

It is essential that an enterprise computing environment include an integrated approach for operations management and enforcement of service-level agreements (SLAs), including the need to support monitoring, logging, analyzing, and reporting on all service transactions (service requests and responses), and issuing alerts for out-of-bound SLA conditions.

Without an adequate infrastructure for service-operations management, Microsoft IT found that it was impossible for service providers to offer predictable service levels to their customers.

Lack of Automation for Service Subscriptions and Service-Level Agreements

Microsoft IT's experience in deploying services with multiple application consumers turned up a number of additional requirements. One of these was the need for a formal process by which service providers are notified of an application developer's intent to subscribe to a service. Subscribing to a service establishes the relationship between service consumer and service provider. The subscription process enables providers to receive notification of pending changes, outages, and out-of-range SLA alerts, and for the appropriate SLA parameters to be established between the two entities. For example, SLA parameters for runtime operations guarantee maximum time-outs and service-availability times (with daily, weekly, and monthly schedules).

Solution

To satisfy these enterprise requirements, Microsoft IT developed and deployed the Alchemy Common Platform (ACP). ACP provided a secure, consistent, enterprise-wide infrastructure for developing, deploying, and managing services across a diverse range of Microsoft business applications.

The goals for the first release of ACP were to:

Create a comprehensive services-management solution using Microsoft technologies.
Provide a reusable, agile, and global services framework that supported all Microsoft line-of-business applications.
Make it easier for development teams to build and consume services in a line-of-business application environment.
Replace stand-alone services that use different technology solutions for authentication, digital signing and encryption, formatting of parameter data, and handling of error conditions.
Establish a single comprehensive repository for services in Microsoft that enabled the categorization, discovery, and reuse of those services.
Reduce development and deployment complexities, as well as overhead costs, when providing operational support for services.
Understand the Service-Oriented Architecture (SOA) model.

Technically, ACP was implemented as a protocol-agnostic framework for services development and management across the enterprise. ACP secures XML SOAP services using a common platform, supports .NET Remoting, establishes a single comprehensive repository of services, enables discovery and categorization of services, and abstracts complexity from development and operational support.

In an SOA model, applications are structured as a fabric of interacting and cooperating services. Each service is designed to be discoverable and accessible in a standard manner. Microsoft IT found that software designers who use an SOA model are more able to solve business-system development problems in a cost-effective, standards-based way. Developers were able to address system interoperability and data-integration requirements more quickly. Microsoft IT wanted to move beyond traditional object-oriented, component-based designs to a model based on common business services that can be easily combined to create new services and end-user applications.

Microsoft observed that applications are expanding beyond the scope of a single system running on a single computer bounded by a single organization. Services were developed and deployed as discrete units of application logic exposing message-based interfaces that can be accessed across a network. As a result, Microsoft IT determined that an SOA built on industry standards and specifications would enable applications to interoperate more easily and new solutions to be created more quickly.

Alchemy Common Platform (ACP) System Overview

The Alchemy Common Platform (ACP) simplifies the discovery, security, management, notification, and analysis tasks required when using services in a production environment. ACP incorporates a distributed design philosophy that works in concert with a centralized management system. The ACP design includes two main subsystems: the ACP System and the ACP Runtime.

The ACP System provides services for data management, transaction-history storage, service registration, and execution verification. It also handles the interaction and storage of services configuration data, including system users, roles, and operational metrics. The ACP Runtime runs in-process on the consuming application, as well as on the service provider. It handles all incoming and outgoing SOAP request/response transactions.

ACP, detailed in Figure 1, comprises the following two subsystems:

The single ACP System, which provides management services for all ACP service calls
An instance of the ACP Runtime, which runs in-process on each service consumer and provider

Figure 1. ACP architecture (Click on the picture for a larger image)

Illustrated components of the ACP Runtime and ACP System are listed in Tables 1 and 2.

Table 1. ACP Runtime

Component	Purpose
Service	Managed service provider, enrolled into ACP
Proxy	Consumer of services, invoked by client application process (for example, Web form application)
Local Transaction Store (LTS)	Holds transactions from ACP-enabled applications before shipping them to the Global Transaction Store (GTS) for archival

Table 2. ACP System

Component	Purpose
Configuration Service	Provides effective metadata for each ACP-enabled service to the ACP Runtime
Security Token Service (STS)	Provides a central mechanism for principal verification, local metadata version verification, and controller registration
Global Transaction Store (GTS)	Receives transactions from ACP Runtime, inspects them, and archives to the ACP data warehouse

By default, interaction with ACP Runtime is abstracted entirely from the host application. Microsoft IT's implementation of the ACP Runtime was based on Microsoft Web Services Enhancements (WSE) 2.0 for Microsoft .NET.

ACP Message Processing

ACP uses a pipeline of filters to process inbound and outbound SOAP messages. Some filters add headers to outgoing SOAP messages, while other filters read and check the validity of headers on inbound messages. Filters can be used also to transform the body of a SOAP message.

The active message processing components in the ACP Runtime are implemented as WSE input and output filters. The processing of an ACP message is depicted in Figure 2.

Figure 2. ACP message processing (Click on the picture for a larger image)

The ACP Runtime filter handles all interactions with the ACP System. This includes communication with the ACP System Configuration Service, Security Token Service (STS), and Global Transaction Store (GTS). These services provide SLA information and tracking mechanisms, centralized authentication and metadata services, and global logging of transactions.

To optimize performance, each ACP service consumer and provider logs transactions to a Local Transaction Store (LTS). The LTS batches transactions for periodic shipment to the GTS for final archiving, SLA monitoring, analysis, reporting, and the processing of alerts. ACP uses SQL Server 2000 Notification Services and Operations Manager to trigger the creation of alerts.

In addition, the ACP Runtime issues queries to the ACP UDDI service, to retrieve service entity and category information. The ACP UDDI service uses an internally developed Microsoft taxonomy to organize, categorize, and support searches for ACP services. Each business service is also broken down into subcategories based on service priority, operating environment (development, test, and production), geographic location of the service, required response time, and other keyword and SLA-related parameters. This unique taxonomy classification speeds search and helps developers locate the best service to meet their needs.

Key Features and Capabilities

The Alchemy platform provides the following key capabilities:

A standard development framework across different applications
Access-rights management
Service-Level Agreement (SLA) management
Dynamic service selection (DSS)
Centralized management
Monitoring and reporting systems
Subscription automation and workflow capabilities

Standard Development Framework

ACP provided solution developers and operations staff with a consistent framework for building, deploying, maintaining, and managing services. Service developers use a consistent framework and API for creating and maintaining services. The platform abstracts away most configuration settings as administrative parameters that can be configured centrally. These include settings for authentication, digital signing and encryption, and messages. A common set of ACP administrative tools are used to deploy services in development, test, and production environments.

ACP reduces the time and effort required to develop, test, and deploy new services. Microsoft IT has found that a service implementation that would typically require four to six months of effort was reduced to four to six weeks, when developed with ACP. Using ACP, Microsoft IT avoided the creation of service "silos" that would otherwise require individual development, deployment, and management infrastructures. Microsoft IT also reduced the server hardware and software resource requirements, as well as the amount of manual effort necessary to deploy and manage services at Microsoft.

Access-Rights Management

ACP was designed with a specific focus on security. ACP incorporates access-rights management mechanisms to control call-level access to particular resources. This is accomplished through policies configured on a central server and distributed to clients configured to access these resources. Checks are made by both the service consumer and provider, to ensure that all principals involved in the transaction have the required access rights and adhere to the policy established for the specific service.

Multiple authentication types are validated locally and verified centrally. The ACP Runtime provides support for Active Directory/NTLM, Microsoft Passport, X.509 certificate-based authentication, and custom identity providers. The Alchemy Repository provides user roles, transport options, and SLA parameters once the ACP Runtime has successfully authenticated the service consumer and provider.

SLA Management

Every ACP resource has a service-level agreement (SLA) associated with it. An SLA comprises specific values for service-level parameters, such as response time, time-out, priority, keywords, and execution environment (for example, development, test, and production stages). These service-level parameters enable a single application to serve data to many consumers in varying ways without the need for additional software configuration or hardware. For example, ACP can provide one application with a specific maximum response time for its requests, while providing another application at a specific priority level for its own requests.

Dynamic Service Selection

Dynamic Service Selection (DSS) determines the specific provider to which a service request will be directed. This is accomplished without requiring any host application involvement. In order to service the client's request most appropriately, SLAs are queried and matched against service-provider capabilities. These selection options ensure that the most appropriate service will be used to respond to service requests from each service consumer.

For example, in a "follow-the-sun" scenario, multiple call centers might be geographically dispersed around the world. One call center might service calls from noon to midnight GMT, and another call center might service calls from midnight to noon GMT. In this scenario, DSS would route application calls to the appropriate call center at the appropriate time, according to SLA service-availability schedules.

Centralized Management

The ACP development team designed the configuration service and services filter to support centralized configuration of ACP services. This provides a single point of control for making changes to an ACP service configuration. The use of a centralized configuration service results in improved accuracy, decreased operations staff, and increased uptime. Within Microsoft IT, ACP reduced administrative staff requirements by a factor of eight. This helped Microsoft IT reduce the system support costs of the services environment.

Monitoring and Reporting

The ACP Runtime automatically logs transactions to the Local Transaction Store (a temporary transaction store). It periodically sends these transactions in batches to a central Global Transaction Store (GTS), which stores and archives all transactions managed by ACP within the enterprise. This enables the monitoring, analysis, reporting, and alerting capabilities required for operations management and enforcement of SLAs.

ACP sends service providers automatic e-mail notifications that enable them to respond to out-of-range SLA conditions, such as poor response times and "service-down" situations.

Subscription Automation and Workflow

ACP gives Microsoft IT an automated subscription workflow process, which allows developers to register their intent to use an existing service. The subscription process sets up an exchange of contact information, whereby an ACP services owner can be informed about the consumers of its services and the applicable set of service-level parameters. Both providers and consumers have the option of receiving an e-mail alert or Instant Message when an ACP service operates outside its SLA. ACP's proactive subscription automation "pushes" relevant service discovery information to developers, freeing them from the need to "pull" that information through time-consuming manual procedures.

Microsoft IT found that reducing the complexity that developers experienced in discovering and subscribing to services was a key factor in encouraging broad adoption of enterprise-wide services.

Development and Deployment

During fiscal year 2004, Microsoft's ACP development team utilized one program manager (who contributed 80 percent of their time), four testers (100 percent), and three software developers (two at 100 percent and one at 50 percent) working for four months to build, test, and deploy ACP.

ACP was deployed in support of most services at Microsoft. This included the services published by eight line-of-business applications. These, in turn, were consumed by over 30 other applications.

When fully deployed, ACP processed and logged up to 1 million transactions per day. The CPU and memory utilization across all of the four ACP System servers was quite low, running between 7 percent and 18 percent.

Benefits

ACP provided Microsoft IT and Microsoft business divisions with significant benefits, based on the deployment of a consistent services development and management framework. Benefits included:

Seamless and easy integration of services from existing line-of-business applications.
Cost savings from deploying fewer server hardware and software resources.
Accelerated adoption and reuse of services.
Easier management of service-level agreements for services.

Seamless and Easy Integration

Organizations must be more agile and responsive than ever before, and Microsoft is no exception. Companies require an application architecture that enables the rapid implementation of services that can be quickly recombined to create new customer management, sales, marketing, and product-development solutions.

With ACP, a typical service can be built and deployed in four to six weeks, compared to the four to six months required to deliver equivalent capabilities using only basic services standards. In one specific project, Microsoft IT was able to reduce its custom development costs by almost 20 percent.

The reduction in development cost derived primarily from ACP's standardization of common service-management coding tasks.

For example, prior to ACP, Microsoft IT exposed services to public customers by implementing a customer security solution for each service. Developers typically required two months to build a customer security solution. This was the average time required for designing, coding, testing, deploying, and documenting the solution.

With ACP, developers were now able to implement customer security using standardized ACP objects and coding conventions. The net result was implementation of the customer security solution in six weeks, roughly 20 percent faster than would have been possible developing the solution using only basic services standards. Similar time savings were reported for development of routing, service discovery, service-level agreements, non-repudiation, and other service-management development tasks.

Reduced Server-Deployment and Associated Costs

Microsoft IT's early deployments of services required a separate server environment in support of services for each line-of-business application. ACP improved upon this by making it possible for Microsoft IT to deploy and manage several services on a single server, in small, medium, and large server-farm configurations. Reducing the need for server hardware and software also reduced the overall number of staff and cost required to manage server operations. Ultimately, only two full-time operations staff members were required for the ACP-enabled services, where previously one person was required, on average, to support the services for each of 10 line-of-business applications. The net result: a reduction from 10 staff members to 2.

From a services-management perspective, all services deployed at Microsoft were supported by a single network of four ACP System servers. To provide the same level of management, an application-by-application basis would have required 80 servers.

Accelerated Adoption and Reuse of Services

Automated subscription processes for managing services reduced costs, and encouraged fast adoption and reuse of common business services.

Proactive Management of Service-Level Agreements

ACP enabled proactive operations management and practical enforcement of service-level agreements by providing standard monitoring, logging, analysis, reporting, and alerting functions. These are the tools that developers of services and consumers needed in order to monitor, measure, and manage out-of-bounds service levels, such as poor response times or "server-down" situations. These capabilities are mandatory for the deployment and management of a production-quality, enterprise-wide platform for services.

Lessons Learned

Adopt a Services-Oriented Architecture (SOA)

In a Service-Oriented Architecture (SOA), applications are structured as interacting and cooperating services. Microsoft IT found, by adopting an SOA model, that it was able to respond to business changes more quickly and with more predictable results.

The ACP SOA helped Microsoft IT steer clear of monolithic applications that are resistant to change.

Value of a Governance Body

It is necessary to adhere to service standards and specifications, when building a robust yet flexible SOA. To this end, Microsoft IT established an internal services governance body that included representatives from the central IT group, as well as key line-of-business application providers. The role of this body was to act as liaison between business groups and the central IT organization. It also took responsibility for IT-related business strategy, technical strategy, and architecture as it relates to the design, deployment, and management of services.

Use Readily Available Tools and Frameworks

Readily available tools and frameworks enable rapid development of standards-based services. These include Visual Studio.NET, Microsoft .NET Framework 2.0, Web Services Enhancements 2.0 for Microsoft .NET, and Windows Server 2003.

Microsoft IT ACP-enabled its enterprise in a short four-month time frame, using the development technologies that are available to Microsoft's customers and partners.

Line-of-Business Application Providers Should Own Their Services

Within the context of the strategy, architecture and guidelines established by the services governance body enable every line-of-business application provider to build and own services for their own application. In most situations, providers are the most knowledgeable of specific line-of-business application functionality, and ready to reuse line-of-business data through services.

Deploy and manage all services using a common, production-quality services-management solution.

Conclusion

The Alchemy Common Platform (ACP) provided Microsoft IT and Microsoft business divisions with significant benefits, based on the deployment of a consistent services development and management framework.

ACP enabled Microsoft IT to deliver a more agile SOA. Fifteen groups have leveraged ACP. Business capabilities increased as development, integration, and operations-management costs decreased.

ACP adheres to the services standards and specifications implemented in the Microsoft .NET Framework and the Web Services Enhancements for Microsoft .NET. For this reason, Microsoft IT was able to build and deploy ACP with current technologies and, at the same time, position itself to interoperate with future services standards-based environments.

Thứ Sáu, 5 tháng 7, 2013