Using a post receive exit script to copy or rename a file after the initial transfer

[MarkV]

We're using MIM to transfer a file to a Linux file system- the receiving/consuming application is grabbing and consuming the file before MIM has finished writing the file, so they are receiving partial files.  It was recommended to us that we set the dpath to a different directory or filename and use a post receive process exit to move/rename the file.

Has anyone used this solution, and if so, how did you get it to work?  Has anyone encountered this problem and used an alternate solution?  If so, what did you use? 

[mpolicarpio]

You can use xmosys.xms as the post receive exit to call the unix mv command to rename the file specified as the destination file name (dpath) to the true desired file name.  I have included a sample request on how to do this.  In the sample, the dpath specified is /tmp/destfile.tmp.  In the exitdata tag of the post receive exit, the unix mv command is called to rename /tmp/destfile.tmp to /tmp/destfile.out.  The post receive exit will run only after the File Transfer Service is done writing the file /tmp/destfile.tmp.

<XMOFTSReq>
      <label>TEST_RENAME</label>
      <lqm>SUPVALHALLA</lqm>
      <sqm>SUPVALHALLA</sqm>
      <dqm>SUPVALHALLA</dqm>
      <oqm>SUPVALHALLA</oqm>
      <spath>/tmp/sourcefile.txt</spath>
      <dpath>/tmp/destfile.tmp</dpath>
      <compressionLevel>5</compressionLevel>
   <ControlCenter>
         <doStatus>1</doStatus>
   </ControlCenter>
   <XMOFTSPostRecList>
      <XMOFTSExit>
            <exit>8</exit>
            <exitdll>xmosys</exitdll>
            <exitentry>xmoSysCall</exitentry>
            <exitqm>SUPVALHALLA</exitqm>
            <exitdata>mv  /tmp/destfile.tmp  /tmp/destfile.out</exitdata>
      </XMOFTSExit>
   </XMOFTSPostRecList>
      <username>admin</username>
</XMOFTSReq>

This is a solution that will work out of the box.  If you wish, you can write a custom exit which may better suit your needs.  The xmosys.xms should provide a good starting point/template on how to accomplish this. 

[proth]

You likely know this already but just to cover off the options...

There are generally two approaches when sending data to a target application.

In the first scenario the target application is running and waiting for the file to arrive (or show up) and in the second the target application is typically not running and is triggered or directly initiated.

Frequently customers implementing MIM don't have the luxury to choose which option and are forced into the approach you are currently using because the focus is on a) replacing an existing file transfer based solution and b) minimizing change to the existing environment.  

In scenario 1 and the example being discussed the target application is already watching (polling) for the arrival of a file.  If the operating system does not provide the facilities to 'lock' the file then the most popular approach is to send the file into a temporary directory or use a temporary name that the target application is not watching for.  Once the file transfer has completed successfully we can utilize a receiver post process to either move and or rename the file making it available to the target application.

When considering option #2 there may be advantages and disadvantages.  In option #2 one of the advantages is that the target application is not running and not polling and therefore not consuming system resources.  When the file arrives a post receive exit could trigger or initiate the target application.  While generally this is a good approach in situations where there are many file transfers and the resource costs associated with frequently starting the target application are significant one would be better served to have the target application configured as a long running process that is either waiting for the arrival of files.  Another alternative may be to 'signal' the long running application notifying it that a file is available for processing.

In most cases I would say that our MIM customers are using Option #1 where an existing application is waiting for the arrival of files.  In these cases the transfer is configured to send to an alternative directory or use an alternative file name and then using the post receive exit point either move the file or rename it making it visible and available to the target application.

[MarkV]

We are indeed using option 1 where the consuming application is watching for the arrival of the file and we don't have the ability to lock the file while it is being written to. 

Using the above solution, we've encountered a problem.  For unit testing I'm trying the rename method in a Windows file system using the ren or rename command.  Here's my request:

<?xml version="1.0" encoding="UTF-8"?>

<!--PM version='8.1.3.0' vendor='CommerceQuest, Inc.'-->

<XMOFTSReq>

  <label>MVH TESTING FOR POST-RECEIVER EXIT</label>

  <lqm>EIDTHDQ01WT_QM</lqm>

  <sqm>EIDTHDQ01WT_QM</sqm>

  <dqm>EIDTHDQ01WT_QM</dqm> <oqm>EIDTHDQ01WT_QM</oqm>

  <spath>D:\MVH\postreceiver.txt</spath>

  <dpath>\\cycihdq01wt\d$\MVH\TEMP\POSTRECEIVER.TXT</dpath>

  <delsrc>FALSE</delsrc>

  <ttype>TEXT</ttype>

  <compressionLevel>5</compressionLevel>

  <mode>REPLACE</mode>

  <ControlCenter>

    <doStatus>1</doStatus>

  </ControlCenter>

  <XMOFTSPostRecList>

    <XMOFTSExit>

      <exit>8</exit>

      <exitdll>xmosys</exitdll>

      <exitentry>xmoSysCall</exitentry>

      <exitqm>EIDTHDQ01WT_QM</exitqm>

      <exitdata>rename \\cycihdq01wt\d$\MVH\TEMP\POSTRECEIVER.TXT \\cycihdq01wt\d$\MVH\TEMP\POSTRECEIVERFILE.TXT</exitdata>

    </XMOFTSExit>

  </XMOFTSPostRecList>

  <username>mqsid</username>

</XMOFTSReq>

Problem is we're receiving the following error from the exit service:

2010-05-28 12:58:57 100528 12:58:57 XMOE000IMQS00000 Received 'Workflow' on queue 'CQ.EXIT' and qmgr 'EIDTHDQ01WT_QM'
2010-05-28 12:58:57 LOG: Fri May 28 12:58:57 2010 ...command is: rename \\cycihdq01wt\d$\MVH\TEMP\POSTRECEIVER.TXT \\cycihdq01wt\d$\MVH\TEMP\POSTRECEIVERFILE.TXT
2010-05-28 12:58:57 LOG: Fri May 28 12:58:57 2010 Exception XMESYSTEMCALL:
2010-05-28 12:58:57 <?xml version='1.0'?>
2010-05-28 12:58:57 <!--Metastorm V8.1.86.14-->
2010-05-28 12:58:57 <XMOException>
2010-05-28 12:58:57    <exceptionTextCount>1</exceptionTextCount>
2010-05-28 12:58:57    <locationCount>0</locationCount>
2010-05-28 12:58:57    <errorCode>24</errorCode>
2010-05-28 12:58:57    <exceptionCode>2010</exceptionCode>
2010-05-28 12:58:57    <exceptionText>XMUtilities is not a registered COM component. rc = -2147221005</exceptionText>
2010-05-28 12:58:57    <errorCodeGroup>XMO Class</errorCodeGroup>
2010-05-28 12:58:57    <scriptName />  
2010-05-28 12:58:57 </XMOException>

XMUtilities is found in the XMCOM.xml member in our registry.

[Ethan Beisher]

If possible, I recommend upgrading to 8.5.1. There are several new enhancements and bug fixes that are currently available in this release.

However, you should not see this error. I recommend you open a trouble ticket (see Support button at top of this page).

Ethan

[mpolicarpio]

The error you are encountering occurs when the XMUtilities tag is missing from XMCOM.xml.  The entry is shown below:

 <XMUtilities>
   <module>xmoplat</module>
</XMUtilities>

The problem can also occur in mutiple registry environments where an older copy of XMCOM.xml which does not contain the XMUtilities tag is found.  Check either your server.xml and/or client.xml files to see the search order for multiple registries.  The registry location listed first under the RegistryList tag will be searched first.  Verify the copy of XMCOM.xml that is found is valid and up to date.

[MarkV]

I made sure there is only one copy of XMCOM.xml running in the current registry, and inside XMCOM.xml, I verified the following is present.

<XMUtilities> 

     <module>xmoplat</module>

</XMUtilities>

On the last run I received the same error:

<XMOException>
   <exceptionTextCount>1</exceptionTextCount>
   <locationCount>0</locationCount>
   <errorCode>24</errorCode>
   <exceptionCode>2010</exceptionCode>
   <exceptionText>XMUtilities is not a registered COM component. rc = -2147221005</exceptionText>
   <errorCodeGroup>XMO Class</errorCodeGroup>
   <scriptName />  
</XMOException>

We're really not in a position to upgrade to 8.5 for this immediate need.  Should I open a trouble ticket?

[mpolicarpio]

I believe there is something in your environment that may be causing this issue.  I suggest contacting customer support for further assistance.  

[MarkV]

I think I have it working now.  First I restarted MIM services on the queue manager, just to be sure.  The test process then actually got into the post receive exit and gave a syntax error on the <exitdata> line.  So I played with the actual DOS command and ended up with the following that worked (specify the full path only once):

<exitdata>rename \\cycihdq01wt\d$\MVH\TEMP\POSTRECEIVER.TXT POSTRECEIVERFILE.TXT</exitdata>

Thanks for everyone's help on this.

[MarkV]

Problem with the mv command. I finally got around to testing in the actual Linux file system and it appears that the Linux command set is not recognized in the post receive exit. The command being executed is:

mv TEMP_HOSTDN00000220100616025426.DAT /appl/softgrocer/data/aplhst/HOSTDN00000220100616025426.DAT

The error thrown by the FTS Exit service is 'mv' is not recognized as an internal or external command .

As a reminder, the coordinator for this process is running in a Windows queue manager and this file is being sent to a Linux client. The file is successfully written to the Linux file system, but the post-receive fails when executing the Linux move command.

I attempted the echo command in the post-receive exit which was recognized, but other Linux commands 'cp' and 'ls' were not recognized. And remember, this worked in the Windows environment. So it likes Windows command sets but not Linux. Is there some kind of pre-qualifier or notation required to invoke the Linux command set?