Wednesday, August 27, 2014

FTP i-to-i part 2

ftp physical file from one IBM i to another

Having gone through a lot of the basics of using FTP, File Transfer Protocol, on an IBM i in the post FTP i-to-i part 1, in this post I will describe how I can transfer physical files, using FTP, to another instance of IBM i, whether it be another partition or another server.

In this scenario I have a job on QSRV1 that generates physical files that need to be transferred to QSRV2 to be processed. If this was a real job I would automate the FTP transfer, I will explain how to do it in Part 3. But in this post I am going describe how I would do it manually, one FTP subcommand at a time.

The physical files I am going to transfer from QSRV1 all start with "FTP" followed by a unique sequence number, for example: FTP00129, and are found in the library FTPOUT. I will be copying these files to the library FTPIN on QSRV2, and I will be calling a program to process the on QSRV2.

As I could be transferring more than one file I am going to use two FTP subcommands that I did not mention in my previous post:

  • mput - Mass copy of file(s) from the FTP client to the FTP host.
  • mget - Mass copy of file(s) from the FTP host to the FTP client.

The first thing I need to do is start FTP on QSRV1 using the FTP command:

  ftp qsrv2

When the FTP entry screen opens it looks like:

                  File Transfer Protocol

 Previous FTP subcommands and messages:
   Connecting to remote host 173.194.117.192 using port 21.
   220-QTCP at qsrv2.
   220 Connection will close if idle more than 5 minutes.






 Enter login ID (simonh):
 ===> _____________________________________________________
___________________________________________________________
___________________________________________________________

 F3=Exit     F6=Print       F9=Retrieve
 F17=Top    F18=Bottom    F21=CL command line

I enter my login id, user profile on QSRV2, and password. I am not going to bother to show you the message for those as you have seen them in Part 1.

I use the lcd to change the default directory/library FTP will use on QSRV1.

  lcd ftpout

Which produces the following in the message section of the screen:

> lcd ftpout
  Local working directory is FTPOUT

Then I change the default directory/library FTP will using on QSRV2 using cd.

  cd ftpin

I can see if this was successful by viewing the message section of the screen:

> cd ftpin
  250 "FTPIN" is current library.

Now comes the new FTP subcommand mput to copy the files from the FTP client, QSRV1, to the FTP host, QSRV2. The mput allows me to copy with a wildcard, therefore to copy all the files all I need to enter is:

  mput ftp*

As the files are copied the message section is updated with each file copied:

227 Entering Passive Mode (173,194,117,192).
150 Sending file to member FTP00129 in file FTP00129 in library FTPOUT.
226 File transfer completed successfully.
    37 bytes transferred in 0.053 seconds. Transfer rate 0.702 KB/sec.
227 Entering Passive Mode (173,194,117,192).
150 Sending file to member FTP00130 in file FTP00130 in library FTPOUT.
226 File transfer completed successfully.
    82 bytes transferred in 0.020 seconds. Transfer rate 4.198 KB/sec.
227 Entering Passive Mode (173,194,117,192).
150 Sending file to member FTP00131 in file FTP00131 in library FTPOUT.
226 File transfer completed successfully.
  1280 bytes transferred in 0.016 seconds. Transfer rate 81.920 KB/sec.

I can see that all three of the files were copied to QSRV2. This does not remove them from FTPOUT, from which they will need to be deleted so that they will not be copied again.

When the files are copied they become "flat files", files that do not contain the fields they did on the FTP client. For example, FTP0129 on QSRV1 looks like this when I use the RUNQRY to view its contents:

Line   ....+....1....+....2....+....
        First     Second   Third
        field     field    field
 000001   1            1   one
 000002   2            2   two
 000003   3            3   three---->
 ****** ********  End of report  ********

The copied version of the file on FTP host, QSRV2, does not have individual fields. This is the same file on QSRV2 when I use RUNQRY:>

Line   ....+....1....+
        FTP00129
 000001 1   one
 000002 2      
 000003 3   three---->
 ****** ********  End of report  ********

This is easily solved by using the CPYF command to copy the "flat file" to another with the desired fields. The FMTOPT(*NCOCHK) means that the data in each record is copied from left to right into the to file regardless of field type. Therefore, the to file should be identical to the file that the data was copied from on QSRV1.

 CPYF FROMFILE(FTPIN/FTP0129) TOFILE(MYLIB/FTPFILE1) +
        MBROPT(*ADD) FMTOPT(*NOCHK)

After going off on the tangent about the format of the copied file on QSRV2 let me return to the FTP. Having copied the files now I need to call the program to process them. This I can do using the quote FTP subcommand.

  quote rcmd CALL MYLIB/PGM1

When the program finishes you will see the following in the message section:

> quote rcmd call mylib/pgm1
  250 Command call mylib/pgm1 successful.

Personally I would not recommend calling a program, I would submit a job so that the FTP can be ended without having to wait for the called program to end.

In Part 3 I will discuss how to automate this.

 

Other posts in this series:

 

You can learn more about this on IBM's website:

 


Addendum

Sorry guys I forgot the problem with FTP-ing files with packed fields. There is some combination of hexadecimal codes in packed fields that is misinterpreted by the FTP process as a CR (Carriage Return) or LF (Line Feed), which causes an additional record to appear. You know, 10 records in the file on QSRV1 and 11 in the file on QSRV2.

The way to prevent this is to code all numeric fields as signed numeric. What I have done in the past is either to change the packed fields in the file to be FTP-ed to signed, or I would a create a work file that is identical to the original file and used REF in the DDS to make the new work file identical to the original, apart from the packed field that I reassign as signed. The code below shows that:

  A                                      REF(FTP001P)
  A          R FTP001WR                              
  A            FLD001    R
  A            FLD002    R     S
  A            FLD003    R
  A            FLD004    R     S
  A            FLD005    R     S
  A            FLD006    R     S
  A            FLD007    R
  A            FLD008    R

Fields FLD002, FLD004, FLD005, and FLD006 are all packed fields in the original file, FTP001P, and are redefined as signed in the work file, FTP001W.

The data from FTP001P can be copied using CPYF into FTP001W and the data in the packed fields will be converted to signed.

  CPYF FROMFILE(FTP001P) TOFILE(FTP001W) MBROPT(*REPLACE) FMTOPT(*MAP)


This article was written for IBM i 7.1.

8 comments:

  1. Nice One!!! I followed the steps you explained here to FTP PF from one server to another. I sent totally 10 records. However 11 is received in target server out of which 1 is junk. Then I tried SAVF concept and then sent target using FTP. At this time I could not see any Junk. Any specific reason to occur this?

    ReplyDelete
    Replies
    1. The only time I have seen this has is when some combination of hex code in a packed numeric field was interpreted as a end of record marker. The way I have stopped this from happening is to define all numeric fields in the physical file that is sent as signed numeric.

      You can easily make a work file to send by using REFFLD to define all the fields in the original file. For the packed fields you put a "S" in the field type field, this defines them as signed numeric (not packed) with the same length and decimal places as the original packed field. You can then CPYF from the original file to this new file in QTEMP, and then FTP the file in QTEMP.

      Hope this helps.

      Delete
  2. If you know for sure that on the target machine the file will exist with the same field definitions (DDS/SQL), the following has worked for me to copy the data.

    cd
    lcd
    bin
    put filea.mbr fileb.mbr (replace

    The key is the "bin" keyword. Once done, it will exactly copy the data provided filea on source as well as fileb on target have the same field definitions.

    ReplyDelete
  3. If the file already exist in the FTP Client, but we again transferring the same file I believe we will error. Is there anyway to solve it

    ReplyDelete
    Replies
    1. In case of put you can use (replace. Not sure if MPUT works with "(replace".
      But think this way. In the IBM i world we have Physical files and they could be multi-member. And the default is *FIRST or the member with the same name as the file name. If you would like to replace a member that is already present, you have to use "(replace". But if the member is not present then a put command like

      put sourcelib/filea targetlib/filea.mbr1
      put sourcelib/filea targetlib/filea.mbr2

      should both work as they are writing to different members.

      Delete
  4. If you want to replace the existing file on the FTP host with the one you have on the FTP client then you can use the replace option.

    mput ftp* (replace

    ReplyDelete

To prevent "comment spam" all comments are moderated.
Learn about this website's comments policy here.

Some people have reported that they cannot post a comment using certain computers and browsers. If this is you feel free to use the Contact Form to send me the comment and I will post it for you, please include the title of the post so I know which one to post the comment to.