BAM Alert Email From Address

Recently I was working on  setting up BAM Alerts for a client on BizTalk Server 2013.
It turned out that emails that were being sent out had "From Address" set as "BAM@microsoft.com".

I tried to change the address on the BAM profile configured in Database Mail, however, it didn't change anything.

After googling a bit I found an old post:
https://social.msdn.microsoft.com/Forums/en-US/ef339925-ad6b-46b9-9f89-29c4dc35dce8/reset-bam-alerts-email-from-or-return-address?forum=biztalkgeneral

This suggested executing ProcessBamNSFiles.vbs file using Notification Service Command Prompt. I couldn't find this command prompt with SQL 2012, so I could not run this.

After some further digging, I found that all this vbs is doing is updating a property in table "bam_Metadata_Proporties" in BAMPrimaryImport database.

I updated this value using SQL Query:

UPDATE dbo.bam_Metadata_Properties 
SET PropertyValue = 'Your email address' 
WHERE (Scope = 'Alert' AND PropertyName = 'AlertMailFrom')

After restarting BAMAlerts service, from address was changed.

Debatching Large Flat File using Custom Disassembler to produce custom sized messages

Requirement

Recently, I had a requirement where a huge flat file having around 20K records was supposed to be debatched and processed. This file is received on a file location and an orchestration starts for each debatched message later in the processing.

I used a flat file disassembler to debacth the file and it produced 20K messages and 20 K orchestratoin started and obviously the server was struggling with CPU usage of 100%.

This was not acceptable and hence I thought we need some mechanism so that only defined number of orchestrations should start at a particular time.

Solution

I created a custom pipeline component(code at end) to debatch messages in a manner so that it produces a batch of messages which has defined number of records.

So, for example, incoming file has 20K records and we define batch size of 50 in pipeline properties.

The custom pipeline component will create 400 messages each having 50 records.

Next, I created a staging database, where I dump these 400 messages with status "Ready". Then a WCF-SQL receive location can poll for this message one at a time at some defined polling interval.

So, this polled message is then published to BizTalk which starts 50 orchestrations at once.

Depending on polling interval next 50 orchestrations will not start until polling interval is elapsed.

We can increase/decrease batch size and polling interval according to the resources we have available on BizTalk environment.For e.g. if we increase batch size to 100 and keep polling interval to 60 seconds, 100 orchestrations will run for 1 minute.So, if say 100 orchestrations do not finish processing in 1 minute you can increase polling interval. And if 100 orchestration are causing high CPU usage, decrease batch size and so on.

This is like a funnel, which controls the outflow of messages to BizTalk:

The complete flow look like below:

1. Receive a large flat file in File receive location

2. Debatch using custom pipeline component to produce multiple messages

3. A WCF-SQL send port subscribes to these messages and saves them to database.

4. A WCF-SQL receive location polls messages from this database at a defined interval.

5. A send port subscribes to this message and sends to a file location where we can use normal flat file disassemble which will debatch each record into individual messages and start further processing.

One advantage of this approach is that we can use the File receive location of the 5th step for small files. So you can skip pre-processing done on large files.

The custom disassembler is very generic which accepts any flat file, you need to specify the document schema and header schema like we do for general Flat File Dis assembler.

Code:

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [System.Runtime.InteropServices.Guid("57D51828-C973-4a62-A534-6DB3CB3CB251")] 

    public class LargeFlatfileSplitter : FFDasmComp,IBaseComponent, IDisassemblerComponent, IPersistPropertyBag

    {

        private string systemPropertiesNamespace = @"http://schemas.microsoft.com/BizTalk/2003/system-properties";

        private int _BatchSize;

        public int BatchSize

        {

            get { return _BatchSize; }

            set { _BatchSize = value; }

        }

        List outboundMessages = new List();

        void IPersistPropertyBag.GetClassID(out Guid classID)

        {

            classID = new Guid("57D51828-C973-4a62-A534-6DB3CB3CB251");

        }

        

        void IPersistPropertyBag.Save

            (IPropertyBag propertyBag, bool clearDirty, bool saveAllProperties)

        {

            base.Save(propertyBag, clearDirty, saveAllProperties);

            object val = (object)BatchSize;

            propertyBag.Write("BatchSize", ref val);

        } 

                public LargeFlatfileSplitter()

                {

                }               

         

            public string Description

            {

                get

                {

                return "Component to batch (break) large message into multiple small messages";

                }

            }

          

            public string Name

            {

                get

                {

                    return "LargeFlatfileSplitter";

                }

            }

            

            public string Version

            {

                get

                {

                return "1.0.0.0";

                }

            }

            public System.IntPtr Icon

            {

                get

                {

                    return new System.IntPtr();

                }

            }

            public new void InitNew()

            {

                base.InitNew(); 

            }

            public new void Load(IPropertyBag propertyBag, int errorLog)

            {

                

                object val = null;

                try

                {

                    propertyBag.Read("BatchSize", out val, 0);

                }

                catch (Exception ex)

                {

                    throw new ApplicationException("Error reading propertybag: " + ex.Message);

                }

                if (val != null)

                    _BatchSize = (int)val;

                else

                    _BatchSize = 1;

                base.Load(propertyBag, errorLog); 

            }

            public System.Collections.IEnumerator Validate(object projectSystem)

            {

                return null;

            }

           

            public new void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)

            {

                

                base.Disassemble(pContext, pInMsg);              

            }

            public new IBaseMessage GetNext(IPipelineContext pContext)

            {                

                IBaseMessage disassembledMessage = null;

                IBaseMessage outMsg = null;

                int counter = 1;

                StringBuilder messageString;

                messageString = new StringBuilder();

                string namespaceURI = "";

                string rootElement = "Root";

                string receivePortName = "";

                while (counter <= BatchSize)

                {

                    disassembledMessage = base.GetNext(pContext);

                    if (disassembledMessage == null)

                        break;

                    //string originalDataString;

                    XmlDocument originalMessageDoc = new XmlDocument();

                    

                    

                    try

                    {

                        Stream originalMessageStream = disassembledMessage.BodyPart.GetOriginalDataStream();

                        originalMessageDoc.Load(originalMessageStream);

                        pContext.ResourceTracker.AddResource(originalMessageStream);

                    }

                    catch (Exception ex)

                    {

                        throw new ApplicationException("Error in reading original message: " + ex.Message);

                    }

                    try

                    {                        

                        if (counter == 1)

                        {

                            namespaceURI = originalMessageDoc.DocumentElement.NamespaceURI;

                            rootElement = originalMessageDoc.DocumentElement.Name;

                            receivePortName= System.Convert.ToString(disassembledMessage.Context.Read("ReceivePortName", "http://schemas.microsoft.com/BizTalk/2003/system-properties"));

                            messageString.Append("<" + rootElement + " xmlns='" + namespaceURI + "'>");

                        }                        

                       

                        foreach (XmlNode childNode in originalMessageDoc.DocumentElement.ChildNodes)

                        {

                            

                            if (counter == BatchSize)

                            {

                                messageString.Append(childNode.OuterXml);

                                messageString.Append("</" + rootElement + ">");

                                //Queue message

                                outMsg = CreateOutgoingMessage(pContext, messageString.ToString(), namespaceURI, rootElement, receivePortName);

                               // counter = 1;

                                messageString.Remove(0, messageString.Length);

                                messageString.Append("<" + rootElement + " xmlns='" + namespaceURI + "'>");

                                messageString.Append(childNode.OuterXml);

                            }

                            else

                            {

                                messageString.Append(childNode.OuterXml);

                            }

                        }

                    }

                    catch (Exception ex)

                    {

                        throw new ApplicationException("Error in creating output message: " + ex.Message);

                    }

                    counter = counter + 1;

                }

                if (counter < BatchSize && messageString.Length !=0)

                {

                    messageString.Append("</" + rootElement + ">");

                    //Queue message

                    outMsg = CreateOutgoingMessage(pContext, messageString.ToString(), namespaceURI, rootElement, receivePortName);                  

                    messageString.Remove(0, messageString.Length);

                }

               

                return outMsg;

            }

            private IBaseMessage CreateOutgoingMessage(IPipelineContext pContext, String messageString, string namespaceURI, string rootElement, string receivePortName)

            {

                IBaseMessage outMsg;

                try

                {

                    //create outgoing message

                    outMsg = pContext.GetMessageFactory().CreateMessage();

                    outMsg.AddPart("Body", pContext.GetMessageFactory().CreateMessagePart(), true);

                    outMsg.Context.Promote("MessageType", systemPropertiesNamespace, namespaceURI + "#" + rootElement.Replace("ns0:", ""));

                    outMsg.Context.Promote("ReceivePortName", systemPropertiesNamespace, receivePortName);

                    byte[] bufferOoutgoingMessage = System.Text.ASCIIEncoding.ASCII.GetBytes(messageString);

                    outMsg.BodyPart.Data = new MemoryStream(bufferOoutgoingMessage);

                    return outMsg;               

                }

                catch (Exception ex)

                {

                    throw new ApplicationException("Error in queueing outgoing messages: " + ex.Message);

                }

            }

    }

PowerShell Script to Export All BizTalk Applications from Server (using Powershell Provider for BizTalk)

I recently wrote a PowerShell script to export all the applications from a BizTalk server as MSIs and xml files for Bindings.

I have used "Powershell provider for BizTalk" for this. I found this tool quite useful, it really reduces the line of code you have to write.

I thought I will share it, may be someone finds it useful:

The first 5 lines are to load Powershell Provider.You will need to provide the SQL Server Name to connect to.

Set-ExecutionPolicy –ExecutionPolicy RemoteSigned

Write-Host "Loading Powershell"

$InitializeDefaultBTSDrive = $false

Add-PSSnapin BizTalkFactory.Powershell.Extensions

New-PSDrive -Name BizTalk -PSProvider BizTalk -Root BizTalk:\ -Instance SQLServerName -Database BizTalkMgmtDb

Write-Host "Creating MSI"


Now we will get all applications from Biztalk, excluding System ones.
$Applications = @(Get-ChildItem -Path Biztalk:\Applications |  Where-Object {$_.IsSystem -eq $false})

Set-Location –Path BizTalk:\Applications


Loop through applications to export, I have used Resource specification to exclude IIS web directories and bindings, if you want to export everything , just skip the portion which removes nodes from ResSpec xml file :

while($Applications.length -ne 0 -and $Applications -ne $null)
{
Write-Host $Applications.length
Write-Host $Applications[0].Name

$ApplPath = "C:\MSI\" + $Applications[0].Name + ".msi"
$BindPath = "C:\MSI\" + $Applications[0].Name + ".xml"
$ResPath = "C:\MSI\" + $Applications[0].Name + "ResSpec.xml"

Write-Host $ApplPath

(Get-ApplicationResourceSpec -Path $Applications[0].Name).OuterXml | Out-File $ResPath 

$xmlResource = [xml] (Get-Content $ResPath )

$delnodes = $xmlResource.SelectNodes("/*[local-name()='ResourceSpec' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resources' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resource' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12'][@Type='System.BizTalk:WebDirectory']")

    ForEach($delnode in $delnodes)
    {
        [void]$xmlResource.ResourceSpec.Resources.RemoveChild($delnode)
    }

 $delnodes = $xmlResource.SelectNodes("/*[local-name()='ResourceSpec' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resources' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resource' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12'][@Type='System.BizTalk:BizTalkBinding'][@Luid='Application/$appName']")

    ForEach($delnode in $delnodes)
    {
        [void]$xmlResource.ResourceSpec.Resources.RemoveChild($delnode)
    }

$xmlResource.Save($ResPath )

Export-Application -Path $Applications[0].Name -Package $ApplPath –ResourceSpec $ResPath

Export-Bindings -Path $Applications[0].Name -Destination $BindPath

if($Applications.length -eq 1)
{
$Applications = $null
}
else
{
$Applications = $Applications[1..($Applications.length -1)]
}
}

Write-Host "MSI and Bindings exported"

When you run the script (using Powershell (x86)) it will export all the applications as MSI and binding files in a file location(C:\MSI).

Custom File Adapter not releasing file lock

Recently, I got a chance to customize the file adapter to be able to access a remote location, providing UserName and Password.

I started with the File Adapter sample provided with BizTalk. The changes were easy , modifying the ReceiveLocation.xsd schema allowed to me to add properties like Domain, UserName and Password to the adapter properties. And some code changes to access remote location after providing username and password to receive files.

Deploying the new changes I faced one issue, my new adapter properties were not visible on BizTalk Admin Console.
I had correctly deployed my dlls to GAC, my registry settings were correct, pointing to correct dlls. So changes should have been there. After some investigation, I found that we need to close and reopen BizTalk Admin Console in order for the changes to appear. Ha... so
-> Add the adapter under Platform settings. Restart Host Instance.
-> Close Admin Console and then reopen, you will have your new changes.

So, now I could access the files on remote location after providing correct username and password. However, I was facing a strange issue.
My files were being renamed to .BTS_WIP and were not being deleted. In event log I had errors saying, "Cannot delete file because it is being used by another process".
Now, what was using my file. I could not manually delete files as well. Only after I stopped Host Instance, I was able to delete the file.

After debugging the code, it turned out that there is some cleanup code which calls GC.Collect() to release resources in the sample. Before the GC.Collect() is completed, File.Delete() was being called in a separate thread ofcourse.
So , I thought of placing my File.Delete() with in a try catch block and in the catch block, I called :
System.GC.Collect();
System.GC.WaitForPendingFinalizers();
File.Delete()

It did the trick!!!

Adding BasicHttp Binding OnRamps to ESB Toolkit

ESB Toolkit comes with few default OnRamps which use WSHttp bindings.
If you want to use BasicHttp binding, you need to do following:

1. Run the Biztalk WCF Service Publishing Wizard to create a new Service

a. Choose option Publish Schemas as WCF Service.



b. Give the Service your desired name. Keep Operation name as "SubmitRequestResponse".

c. For Request and Response choose schema: Microsoft.XLANGs.BaseTypes.dll#Any



2. Modify the Web.config file of the new Service.

a. Under System.ServiceModel element add:

<extensions><behaviorExtensions><add name="soapHeaderMetadata"
type="Microsoft.Practices.ESB.ServiceModel.Helpers.SoapHeaderMetadataExtensionElement
, Microsoft.Practices.ESB.ServiceModel.Helpers, Version=2.0.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35"/></behaviorExtensions></extensions>

b. Under behaviors add:

<soapHeaderMetadata enabled="true">
<soapHeaders>
<add headerTypeName="ItineraryDescription" operation="SubmitRequestResponse" message="ItineraryDescription"
operationType="Input"
targetNamespace="http://schemas.microsoft.biztalk.practices.esb.com/itinerary"
xsdFileName="ItineraryDescription.xsd"/>
</soapHeaders>
</soapHeaderMetadata>

3. Once the Wizard completes, it will create a Service in IIS and a Receive Location(On Ramp). 

Make sure to create a seperate App Pool for this new service. You cannot have both the services(ESB default and new one) runnning under same App Pool. 

4. On Ramp needs to be modified now. 

a. Modify the pipelines. Choose the same pipelines as in default ESB On Ramp(ItinerarySelectReceiveXml). 

b. Configure the adapter. You can keep Security settings according to your need(None, Transport etc.). Message tab needs to be modified same as WSHttp adapter on default On Ramp. 

This basicHttp On Ramp is now ready.

Itinerary Broker Service - Routing on Context Property

Recently, I worked on a Biztalk project which used ESB toolkit 2.0 for integration. I was working on a scenario where I had to make routing decisions, based on some property within the message, in Itineraries.

I knew that Itinerary Broker service shape should be used to make routing decisions in Itineraries. Itinerary broker services provides Message Context resolver which can be use to resolve destination on the basis of message context property.

So the first thing I did is, used property promotion in schema, to promote the message property on which routing had to be done. Then I created my itinerary using On Ramp shape to receive the message, broker service to route the message and finally the Off Ramp shape to send the message to destination.

However, after completing the itinerary, I realized that none of my filter condition was being set as true and I was getting below error:

There was a failure executing the receive pipeline: "Microsoft.Practices.ESB.Itinerary.Pipelines.ItinerarySelectReceiveXml, Microsoft.Practices.ESB.Itinerary.Pipelines, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Source: "ESB Dispatcher" Receive Port: "OnRamp.Itinerary.Response" URI: "/ESB.ItineraryServices.Response.WCF/ProcessItinerary.svc" Reason: Error 197501: Cannot determine next service, none of the filter conditions returned true

On analysing the message in tracked message instances, I realized that the property on which routing was being done, was not at all promoted in the message.

Reason:
Since itinerary is executed in the pipeline, the property promotion has not occurred at the time when we are calling the broker service and hence we get this error.

Workaround:
The workaround is to include a map shape in itinerary before broker service. This map should do nothing substantial, it should map input schema to input schema itself.
Because of this map, the property gets promoted and broker service shape can now use this promoted property.




Another important fact:
As soon as you use the Broker Service shape in Itinerary, the Export Mode of the Itinerary changes to "Strict".

In strict mode, you cannot use the Off Ramp Extender shape to resolve the destination. You need to use a Messaging Extender in Itinerary to resolve the destination:



I hope this helps someone.