Tutorial-Pipelines

Module 5: Creating Pipelines

Introduction to Pipelines
Receive Pipeline
Send pipeline
Built-in pipelines
Built-in pipeline components
Building Custom Pipelines

Introduction to Pipelines:

Pipelines are probably the least properly utilized tools in the BizTalk toolbox.

Pipeline is important part of BizTalk server in message processing. It is a component of Microsoft BizTalk Server that provides an implementation of the Pipes and Filters integration pattern. As per BizTalk runtime architecture, pipeline comes just after receive message on receive port/receive location. And messages in BizTalk are immutable once they have entered the Messagebox.

In pipeline level we can do several activities before sending to Messagebox.

Following activity can be applied in Pipeline:

Conversion from another format to XML vice-versa
Process encoded document (MIME/SMIME)
Breaking incoming document into separate individual documents
Modify message
Verifying or signing document with digital signature

Pipelines are divided into two part: Receive & Send pipelines. Receive pipeline specify on receive location, and send pipeline specify on send port.

It processes messages in a predefined sequence. it divides processing into categories of work called stages and determines the sequence in which the stages are performed. Each stage defines logical work groups, determines which components can go in that stage, and specifies how the pipeline components in the stage are run.

Within each stage, pipeline components perform specific tasks. For example, components within stages of a receive pipeline may decode, disassemble, and then convert documents from other formats to XML. Send pipelines do essentially the opposite: convert documents from XML to other formats, assemble, and encrypt, with each pipeline component performing a portion of the entire process. Although a stage is a container of components, each stage is itself a component with metadata. Stages have no execution code, as opposed to pipeline components, which do have execution code.

Receive Pipeline:

Receive pipeline is specified on receive location. A receive pipeline operates on a message after it is received by the receive adapter. The receive pipeline takes the initial message, performs some transformations, and disassembles the raw data into zero, one, or multiple messages. These individual messages can then be processed by BizTalk Server.

Receive Pipeline Stages

Stage	Purpose
Decode	This stage can contain the functionality to decode or decrypt the message.The MIME/SMIME decoder, JSON decoder pipeline component or a custom decoding component should be placed in this stage if the incoming messages need to be decoded from one format to another.This stage takes one message and produces one message.This stage can contain between zero and 255 components.This stage is used to change the message format e.g. from excel to xml to process inside BizTalk via custom pipeline
Disassemble	This stage is used for components that parse or disassemble the message. Disassembler are designed to produce zero or multiple messages based on the input messageThe components within this stage probe the message to see if the format of the message is recognized. Based on the recognition of the format, one of the components disassembles the message.If this stage contains more than one component, only the first component that recognizes the message format is run. If none of the components within the stage recognize the message format, the message processing fails.This stage should include any custom components that implement special behavior to disassemble the message contents.The disassembler is responsible for promoting the message type of the messageAny custom property promotion occurs in disassembler stage. De-batching the enveloped message into several individual message. BizTalk provides out of box support for xml & flat file de-batchingThis stage can contain between zero and 255 components. If there are no components in the stage, the message is passed through.
Validate	A pipeline component processes only messages that conform to the schemas specified in that component. If a pipeline receives a message whose schema is not associated with any component in the pipeline, that message is not processed. Depending on the adapter that submits the message, the message is either suspended or an error is issued to the sender. Components in this stage are used to validate the XML messages produced by the Disassemble stage. Components in this stage specify schemas to perform the XML validation. This stage can contain between zero and 255 components.This stage may be run more than once. It runs once per message created by the Disassemble stage.
Resolve Party	This stage is used often in trading partner exchange scenarios. This stage is usually used to examine the digital certificates of a signed message to ensure the validity of the sending party.This stage is for components that determine the party that has sent the incoming message. Determining this is needed for routing based on sender rather than content of message.

Send Pipeline:

Send pipeline is specified on send port. this have specific stages that are related to preparing the message to be sent out of BizTalk A send pipeline is responsible for processing documents before sending them to their final destinations. The send pipeline takes one message and produces one message to send.

You can create a new send pipeline, or you can use one of the two default send pipelines included in BizTalk Server—the pass-through send pipeline and the XML send pipeline.

By default, the send pipeline consists of three empty stages: Pre-assemble, Assemble and Encode. This topic contains design considerations for populating these stages.

Send Pipeline Stages

Stage	Purpose
Pre-assemble	Performs any message processing necessary before assembling the message This stage is a placeholder for custom components that should perform some action on the message before the message is serialized.Manipulate the message before it is converted the format needed by the assembler componentThis stage is run once per message.All components in this stage are run.
Assemble	Assembles the message and prepares it to be transmitted by taking steps such as adding envelopes, converting XML to flat files, or other tasks complementary to the disassemble stage in a receive pipeline Components in this stage are responsible for assembling or serializing the message and converting it to or from XML.This stage assemblers that has the responsibility to convert the xml into the format that should be sent out e.g. xml, flat file, excel, pdf Assemblers are responsible for combining multiple smaller messages into one large message. This is often the case when you are aggregating several messages into a batch. But this will develop when send port call inside the orchestration. You can get this scenario from BizTalk design pattern. For more details click here.The assembler has the responsibility to demote to copy of values from context of message into the message bodyThis stage accepts zero components or one component.
Encode	Encodes or encrypts the message before delivery Encoders are responsible for writing the message in a fashion so that it can be read by the downstream system.This stage is used for components that encode or encrypt the message.Place the MIME/SMIME Encoder component or a custom encoding component in this stage if message signing is required.This stage is run once per message.This stage can contain between zero and 255 components.All components in this stage are executed.

Built-In Pipeline:

When you create a new application, the default pipelines are created and deployed by default and appear in the Microsoft.BizTalk.DefaultPipelines assembly in the \References folder for every BizTalk project. The default pipelines cannot be modified in Pipeline Designer. These pipelines can be selected when configuring a send port or receive location in BizTalk Explorer.

Note: The XML receive, and XML send pipelines do not support XML documents larger than 4 gigabytes.

PassThruReceive pipeline

The pass-through receive pipeline has no components. It is used for simple pass-through scenarios when no message payload processing is necessary. This pipeline is generally used when the source and the destination of the message are known, and the message requires no validation, encoding, or disassembling. This pipeline is commonly used in conjunction with the pass-through send pipeline.

Because it does not contain a disassembler, the pass-through receive pipeline cannot be used to route messages to orchestrations. There is no file format matter in this case. The pass-through receive pipeline does not support property promotion. BizTalk send port receive the message based on BTS.ReceivePortName in filter section. And Also, you need to use PassThruTransmit pipeline at send port.

PassThruTransmit pipeline

The pass-through-send pipeline has no components. This pipeline is generally used when no document processing is necessary before sending the message to a destination.

XMLReceive pipeline

XML Receive pipeline commonly used when receive xml files. The XML receive pipeline consists of the following stages:

Decode. Empty
Disassemble. Contains the XML Disassembler component
Validate. Empty
ResolveParty. Runs the Party Resolution component, which resolves the certificate subject or the source security ID to the party ID.

XML Receive pipeline does not validate the input message. There are some below scenario where you can use XML Receive pipeline:

To promote properties on message
1. Route message to orchestration
1. Disassemble the envelop message into individual document messages

XMLTransmit pipeline

The XML send pipeline consists of the following stages:

Pre-assemble. Empty
Assemble. Contains the XML Assembler component
Encode. Empty

This pipeline is used for below case. And this case applied when use send pipeline in orchestration to assemble multiple message into single message.

XML Assembler Builds envelopes as needed and appends XML messages within the envelope.
Populates content properties on the message instance and envelopes.

Built-In Pipeline Components:

Microsoft provides built in pipeline components to achieve desire result from pipeline. Before moving to custom pipelines, you need to understand what is currently have in pipeline bucket and may be not require for custom pipeline development. Below is the list of built in pipeline components:

Custom Pipelines:

Microsoft provides lots of feature in BizTalk Pipeline with having inbuild pipeline components. E.g. JSON Encoder, Decoder, MIME/SMIME Decoder etc.

But sometimes we need add something in context of message then we need to do customization in pipeline, BizTalk provides the facilities to customize with help of Custom pipeline. C# and VB.Net languages are basically used to write code for this.

Below are few scenarios where you need to apply Custom Pipeline

Conversion from another format to XML vice-versa:
Add namespace (this facility is existed in BizTalk pipeline, when use ESB pipelines)
Promote message context property on custom data
Set dynamic send port
Message archiving
Handling large message

Please click here for more details of custom pipelines

Below is reference for custom pipelines:

BizTalk World