WCF receive location and “Enabled routing for failed messages”
If a message failed schema validation on a WCF web http port, the system sending the message will get a 500 error but there won’t be any suspended messages in BizTalk. That means, even if you have checked “Enable routing for failed messages” on the receive port, the message won’t be routed because there is no message suspended.
After some googling, I found a post originally answered by my ex-colleague Colin:
http://stackoverflow.com/questions/22513010/creating-failed-message-on-wcf-customisolated-receive-location
Basically for WCF receive locations, you will need to configure the location to generate a suspended message in the case of a failure in the pipeline. You can do this by going into the WCF adapter configuration in the receive location, click the “Messages” tab and check “Suspend request message on failure”. Combining this option and “Enabled routing for failed messages”, the failed message should now be generated and routed correctly.
SQL connection failure, DTC issues
A SQL send port started to fail with the following error after a SQL server VM migration.
The adapter “SQL” raised an error message. Details “Connection failure”.”
Not very informative.
Some observations:
– Tested connection using SSMS running as the host instance’s login and the connection to SQL works fine
– Nothing is changed on the SQL server VM except that it’s migrated to a different physical host
– DTCPing errors: “Error(0x6D9) at dtcping.cpp @303”, “1753(There are no more endpoints available from the endpoint mapper.)”
Since the connection from SSMS works fine, this must be a RPC/DTC issue. This is a very good link on troubleshooting DTC issues:
However, according to the link above, my DTCPING error code means that I have a firewall issue but as it turned out, it’s not a firewall issue.
After a day playing with the network firewall settings with no joy, the SQL error log on the sql server gave a very good clue in the form of the error message below:
“Enlist operation failed: 0x8004d01c(failed to retrieve text for this error. Reason: 15105). SQL Server could not register with Microsoft Distributed Transaction Coordinator (MS DTC) as a resource manager for this transaction. The transaction may have been stopped by the client or the resource manager.”
Which leads me to this article below:
http://nearshore-mexico.blogspot.com.au/2011/05/quirks-of-distributed-transactions.html
Resolution:
The root cause of my issue is that the DTC service on the SQL server is set to be “Automatic (Delayed start)” and the SQL server services are set to “Automatice”. Which means after the SQL server migration, when the SQL server was started, the DTC service was started after the SQL server. According to the article, this may cause issues as the SQL service won’t be able to register in the DTC service.
So the resolution for my issue is to stop both DTC and SQL Server services. Start DTC service first, and then start SQL Server. To make sure this issue doesn’t happen again when the server is rebooted next time, I’ve also changed the DTC service’s startup type to “Automatic” and SQL Server to “Manual”.
BizTalk zip multiple large files in a pipeline component
Recently I used the Ionic.zip dll (aka DotNetZip) library in one of my projects to zip multiple incoming files and send out the single zip file.
The files will all be coming from a single location and I simply created an orchestration with the a loop and a listen shape. The custom pipeline with the zip component is being executed inside the orchestration because I can then send multiple files to the pipeline (Using the statement below in an expression shape) and zip them all in a single zip file using the code in this link (http://www.codeproject.com/Articles/152405/Zip-Files-in-a-Custom-Pipeline-Component)
Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline(typeof (PIPELINE.FULLNAME),PipelineInputMessages,OutputFile);
However, the files inside the outgoing zip file get corrupted if they are large files (More than 1mb or something).
Solution:
The issue is fixed when I change “.GetBuffer()” to “.ToArray()” in the Execute method of the pipeline component.
You can read more about the difference between the 2 methods here:
http://krishnabhargav.blogspot.com.au/2009/06/net-funda-memorystream-toarray-vs.html
FTP Adapter causing Host instances to crash
Recently I encountered a strange issue where some FTP receive locations were crashing the host instances and everything that runs under that host instance stopped processing and nothing was going through BizTalk.
Observations:
Nothing was changed before the issue started occurring.
I was able to locate the receive locations that are failing by disabling all FTP locations and start them up one by one.
When the failing receive location is started, the following entries are written into the event log (In this order):
Warning:
The adapter “FTP” raised an error message. Details “A failure occurred when parsing the remote folder listing. Inner Exception details: “The data connection could not be established. “. “.
Error – Application Error:
Faulting application name: BTSNTSvc.exe, version: 3.10.229.0, time stamp: 0x50fe4120
Faulting module name: ntdll.dll, version: 6.1.7601.18229, time stamp: 0x51fb1072
Exception code: 0xc0000005
Fault offset: 0x000222d2
Faulting process id: 0x968
Faulting application start time: 0x01cee4067888505b
Faulting application path: C:\Program Files (x86)\Microsoft BizTalk Server 2013\BTSNTSvc.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: db1a69bb-4ff9-11e3-9c2e-005056ad7e86
Warning:
The adapter “FTP” raised an error message. Details “The adapter recovered from recent failures. The Event Log Error Threshold count reset. “.
Windows Error Reporting:
P1: BTSNTSvc.exe
P2: 3.10.229.0
P3: 50fe4120
P4: ntdll.dll
P5: 6.1.7601.18229
P6: 51fb1072
P7: c0000005
P8: 000222d2
P9:
P10:
Attached files:
C:\Users\bt_user\AppData\Local\Temp\BizTalkTraceLog.bin
C:\Users\bt_user\AppData\Local\Temp\WER340A.tmp.appcompat.txt
C:\Users\bt_user\AppData\Local\Temp\WER3449.tmp.WERInternalMetadata.xml
C:\Users\bt_user\AppData\Local\Temp\WER346A.tmp.hdmp
C:\Users\bt_user\AppData\Local\Temp\WER4FD7.tmp.mdmp
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_BTSNTSvc.exe_d30c826784ca37da1143b5a5f4a3cd0d3a777_cab_03d15198
Analysis symbol:
Rechecking for solution: 0
Report Id: db1a69bb-4ff9-11e3-9c2e-005056ad7e86
Report Status: 4
Warning: User Profile Service:
Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards.
DETAIL –
2 user registry handles leaked from \Registry\User\S-1-5-21-1176131002-1001592366-623647154-72751:
Process 3304 (\Device\HarddiskVolume2\Windows\SysWOW64\DWWIN.EXE) has opened key \REGISTRY\USER\S-1-5-21-1176131002-1001592366-623647154-72751
Process 3304 (\Device\HarddiskVolume2\Windows\SysWOW64\DWWIN.EXE) has opened key \REGISTRY\USER\S-1-5-21-1176131002-1001592366-623647154-72751\Control Panel\International
Output from FTP adapter logging on the receive location:
< 220 Microsoft FTP Service
> USER USER_FTP
< 331 Password required for USER_FTP.
> PASS xxxx
< 230 User logged in.
> PWD
< 257 “/” is current directory.
> PWD
< 257 “/” is current directory.
> PWD
< 257 “/” is current directory.
> SYST
< 215 Windows_NT
> PWD
< 257 “/” is current directory.
> CWD ftp/blah/blah/blah/blah
< 250 CWD command successful.
> PWD
< 257 “ftp/blah/blah/blah/blah” is current directory.
> TYPE A
< 200 Type set to A.
> PORT 127,0,0,1,205,99
< 200 PORT command successful.
> LIST *.*
< 125 Data connection already open; Transfer starting.
< 226 Transfer complete.
> QUIT
< 221 Goodbye.
Output from IIS log:
2013-11-18 02:14:53 127.0.0.1 – 127.0.0.1 21 ControlChannelOpened – – 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 – 127.0.0.1 21 USER USER_FTP 331 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PASS *** 230 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 /
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PWD – 257 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PWD – 257 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PWD – 257 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 SYST – 215 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PWD – 257 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 CWD ftp/blah/blah/blah/blah 250 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 ftp/blah/blah/blah/blah
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PWD – 257 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 TYPE A 200 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 PORT 127,0,0,1,244,102 200 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 20 DataChannelOpened – – 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 20 DataChannelClosed – – 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 LIST *.* 226 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 ftp/blah/blah/blah/blah/*.*
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 QUIT – 221 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
2013-11-18 02:14:53 127.0.0.1 USER_FTP 127.0.0.1 21 ControlChannelClosed – – 0 0 9d3e6ecc-cb5e-4bc0-89d9-68e7abba5c37 –
More observations:
* As you can see, the FTP server is hosted locally
* There are other receive locations monitoring other folders on the same server with the exact same adapter settings, but they are working fine.
* If I enable one of the failing receive locations and restart the host instance, the host instance will crash and recover and repeat. Nothing will be picked up by adapters running under the host instance that crashed.
Work around:
I have no idea what caused the crash, but after changing the FTP mode to PASSIVE on the receive location, the issue went away.
Issue with Bm.exe
You get the following error message when configuring BAM or deploying Bam activity/view:
Microsoft.BizTalk.Bam.Management.BamManagerException: Encountered error while executing command on SQL Server “”. —> System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server) —> System.ComponentModel.Win32Exception: The system cannot find the file specified
Issue: Either SQL server is not installed on the server or the service is not started. Even in a multi server deployment, You will still need to install SQL server native client on the BizTalk server for Bam to work properly.
Silverlight WCF RIA services deployment and the 404 error
For the whole day yesterday I’ve been struggling with deploying a silverlight website with RIA services to a test environment.
The website worked fine in my development environment when it’s published from VS2012, the only problem I encountered in the dev environment was having to re-add (remove and add) some .net 4.0 svc extensions and making sure the correct .net 4.0 svc extension is being used (Running “aspnet_regiis.exe -i” in C:\Windows\Microsoft.NET\Framework\v4.0.30319).
However, once I copied the whole web application folder onto my test environment. All svc pages started to give me 404 errors. Only information in fiddler was an “EndpointNotFoundException”. I think I’ve gone through every single blog,forum post/comments but I was still getting nowhere with this issue. I’ve checked my config files, IIS settings, bin folders and everything seems to be correct.
The steps I used to deploy the application are in
http://msdn.microsoft.com/en-us/library/ff426912(v=vs.91).aspx
And finally, after a day and a half, I realized what I was missing. The guide only tells me to include the following 2 domain services assemblies in the bin folder.
System.ServiceModel.DomainServices.Server.dll
System.ServiceModel.DomainServices.Hosting.dll
But in fact, for the runtime generated svc to work, we need a 3rd dll which is
System.ServiceModel.DomainServices.EntityFramework.dll
Once I put this dll into my bin folder, the svc pages started to work and everything came together!
Then I went back to some of the blogs I’ve read, I noticed that some of them actually mentioned about this dll and it’s simply my attention to detail that’s been failing me..
http://blogs.msdn.com/b/saurabh/archive/2010/03/16/ria-services-application-deployment.aspx
Installing ESB Toolkit 2.2
The ESB Toolkit 2.2 installation can be started from the biztalk 2013 beta setup wizard.
The installation is a lot more straight forward and easy to follow than the previous ESBT versions.
The only issue I’ve encountered is when I tried to run the ESBConfigurationTool.exe, I get a popup window:
“Object reference not set to an instance of an object”
And then I found this post:
http://social.msdn.microsoft.com/Forums/en-US/biztalkesb/thread/9894fc8b-bca3-4a62-a2d0-c1d2df9a8086
To get rid of this error, go to the server manager, run the “Add Roles and Features wizard” and add all the roles under Web Server (IIS)/Management Tools/IIS 6 Management Compatibility
The new configuration tool also lets you create the ESB biztalk application from the “ESB BizTalk Applications” tab.
Bits and pieces about DLLs in BizTalk
Here’s some of the things to watch out for when GACing DLLs
Gacing BizTalk DLLs
When an dll file is GAC’d, the changes won’t come into effect until the host instances are restarted.
Although it’s not good practise, the dll gacing and host restarting can be done when there are existing instances that use the updated dll.
Example: There are existing orchestration instances and an updated orchestration dll is gac’d and host instances are restarted. The orchestration has 5 shapes in a sequential order A,B,C,D,E and changes have been made on shape C. All shapes are persistence points.
When the host instance is stopped, all the running orchestration instances will be saved to the database at their last persistence point (dehydrated) and the dll will be unloaded.
Persistent point: http://msdn.microsoft.com/en-us/library/aa547090(v=bts.20).aspx
As the host instance is started again, the saved orchestration instances will be rehydrated and the updated dll will be loaded.
For the saved instances, only the ones that were saved into the database before reaching shape C will pick up the changes.
Gacing DLLs and export to msi
When an dll is GAC’d and not property stored into the BizTalk management database, the exported msi file will not contain the latest dll but the dll which has been stored into the BizTalk management database.
To store a dll into the management database, it has to be imported from a msi or refreshed on the resources tab.
Same schema in different DLLs
The combination of namespace and root element name must be unique across all schemas in the same BizTalk group. It may not be obvious because biztalk allows you to deploy multiple DLLs with the same schemas and it doesn’t complain about it until the schema is used. When a non-unique schema is used, the following error will be raised.
“This Assembler cannot retrieve a document specification using this type:namespace#root”
Test map on multipart source document
Just a quick one on using test map on multipart source documents.
Simply set the Testmap input to “Generate instance” and run test map. The test will generate an input file for you which contains all the message parts in the source document. You can then open that file and put in your own messages and set the modified file as the Testmap input :).
BizTalk SMTP adapter – Sending emails with attachment
I’m sure every BizTalk developer has created many applications where one or more smtp send ports are required.
Sometimes you need to send emails with requirements like: dynamic subjects, dynamic attachments names, flat file attachments. Here are 3 common scenarios. There are many approaches for each of them but I will try to provide the simplest solution that I can think of. Hence, if an orchestration is not required, don’t use one.
Everything is tested in BizTalk 2006 R2 because that’s what I have access to at the time of writing this blog
1. Send an incoming xml file as an attachment with attachment name doofer.xml, subject “Doofer is here” and email body text “Here’s a doofer”.
– Create a custom pipeline component and put the following code into the execute method:
inmsg.BodyPart.PartProperties.Write(“FileName”, “http://schemas.microsoft.com/BizTalk/2003/mime-properties”, “Doofer.xml”);
inmsg.Context.Write(“Subject”, “http://schemas.microsoft.com/BizTalk/2003/smtp-properties”, “Doofer is here”);
– Add the custom pipeline component to a send pipeline
– Configure the send pipeline on the SMTP send port
– Go to SMTP Transport Properties -> Compose -> Text and enter “Here’s a doofer” in the textbox
– Go to the Attachments tab and select Attach only body part
2. Send an incoming flat file as an attachment with the original file name
– Create a Flat file schema
– Create a custom receive pipeline with a flat file disassembler
– Create a custom pipeline component which has the following code:
string fileName = (string)inmsg.Context.Read(“ReceivedFileName”,”http://schemas.microsoft.com/BizTalk/2003/file-properties”);
fileName = fileName.Substring(fileName.LastIndexOf(“\\”) + 1);
inmsg.BodyPart.PartProperties.Write(“FileName”, “http://schemas.microsoft.com/BizTalk/2003/mime-properties”, fileName);
– Create a custom send pipeline with a flat file assembler and the custom pipeline component
– Use the custom receive pipeline on the receive location
– Use the custom send pipeline on the send port
– In SMTP configuration, enter some text into the textbox in the compose tab (Note an error will occur if no text is entered)
– On attachment tab, select “Attach only body part”
Note: If dynamic email body text is required then we will need to use the rawstring class. A common solution which involves an orchestration is shown below.
3. Send the incoming xml file as an attachment with attachment name doofer.xml and email body text is set dynamically in orchestration.
– Create a rawstring class: http://msdn.microsoft.com/en-us/library/ee253435(v=bts.10).aspx
– In the orchestration, create a multipart message Multipart_Msg.
– Add a message of RawString type to Multipart_Msg
– Add a message of the incoming xml message type to Multipart_Msg
– Add an construct message shape in the orchestration with a message assignment shape.
– Add the following code into the message assignment shape
Multipart_Msg.MessagePart_RawString = new Microsoft.Samples.BizTalk.XlangCustomFormatters.RawString(“This is the rawstring email body”);
Multipart_Msg.MessagePart_Incoming = IncomingMsg;
Multipart_Msg.MessagePart_2(MIME.FileName) = “doofer.xml”;
– Deploy the solution
– Add a SMTP send port to the application.
– Go to the SMTP configurations. Go to the Attachments and select “Attach all parts”. Leave everything else default.
Nick's Blog 