Wednesday, September 3, 2014

FTP i-to-i part 3

ftp program automate

In the past two posts about this subject I discussed how to manually copy files from one IBM i to another using FTP, File Transfer Protocol. In this post I am going to demonstrate how to create a program to start FTP, execute FTP subcommands, and then produce a printed log of what happened.

The program I will show is very simple and can be used for any types FTP subcommands. In this post I will use the example I discussed in Part 2, to copy a number of files from one library on QSRV1 to another library on QSRV2, and then call the program on QSRV2 to process the copied files.

I never think it is a good idea to use your profile as a generic profile used by automatically submitted jobs. Firstly, it means that others will be able to see your password. Secondly, I would imagine that your profile has all kinds of authorities granted to it that the submitted job does not need. Therefore, I strongly recommend that you create a new profile to use for running FTP jobs on the FTP host, QSRV2. When I created the generic user profile I used the following:

  CRTUSRPRF USRPRF(FTPUSER) PASSWORD(**********) PWDEXP(*NO) +
              USRCLS(*USER) INLPGM(*NONE) INLMNU(*SIGNOFF) +
              LMTCPB(*NO) TEXT('FTP user profile') +
              SPCAUT(*ALLOBJ) PWDEXPITV(*NOMAX) +
              JOBD(QGPL/QDFTJOBD) +
              OUTQ(QUSRSYS/FTPUSER) ATNPGM(*NONE) 

PASSWORD(**********): I have not given a password as I want you to think of a complicated 10 character one. Make it a mix of letters and numbers, and make it something that someone else is not going to be able to guess.

INLPGM(*NONE) INLMNU(*SIGNOFF) ATNPGM(*NONE): These three ensure that if someone tries to signon with the FTPUSER profile they will not be able to get to screen where they can enter any commands or view menu.

SPCAUT(*ALLOBJ): I am sure I am going to get feedback from people telling me that no profile should have *ALLOBJ authority. I find that by using *ALLOBJ I am not going to get an authority error on the FTP host. If you do not want to use *ALLOBJ then you will need to try the various security settings and see which works best for you.

PWDEXPITV(*NOMAX): I do not want the password to expire as I will have to change all the FTP scripts I have created to contain the new password.

OUTQ(QUSRSYS/FTPUSER): I create this profile its own output queue, therefore, all the generated spool files are in one place to be easily found, and deleted at a later date.

I am proponent of using a single source file, rather than having QCLSRC, QDDSSRC, QRPGLESRC, etc. See IMHO: Source files – one or many?. But I do keep my FTP script source members in their own source file, FTPSRC. When I create my FTPSRC file I would use the following command:

  CRTSRCPF FILE(MYLIB/FTPSRC) RCDLEN(112) +
             TEXT('FTP scripts source')

Now I create a new member in the FTPSRC source file for the FTP subcommands to copy the files. When creating the member the member type is unimportant, it can even be blank. I always give mine the member type of "FTP".

The first line of the member must contain the user profile and password used to log onto to the FTP host, QSRV2. As this will be clearly visible to anyone who can open the source member is why it is not a good idea to use your profile.

The FTP subcommands are entered in the order I want them to be executed, each subcommand on its own line. When I have finished my source member will look like:

  ftpuser **********
  lcd ftpout
  cd ftpin
  mput ftp*
  quote rcmd CALL MYLIB/PGM1

Save the source member.

When FTP is performed if it finds a file called INPUT it will perform the FTP subcommands in it, and if it finds a file called OUTPUT it will write the output messages to it. The first program I am going to create copies a member from the FTPSRC into a file that I will use as the input. I will then call a second program to execute the FTP command and copy the output to a printer file. By taking this approach the second program will be able to be called from any program that wants to perform FTP subcommands, another example of the "code once, call from many" principal.

The first program, shown below, checks to see if the file FTPINPUT in QTEMP exists, line 30. If it does not it is created on line 31. The FTP subcommands are copied from the member MBR001 in FTPSRC to FTPINPUT file in QTEMP, line 32, notice that FMTOPT(*CVTSRC) parameter has to be used to copy the data from a source file member to a data file member. The second program, RUNFTPCMDS, is called on line 33, passing the name of the FTP host I want to copy the data to.

01  PGM
    .
    .
    .

30  CHKOBJ  OBJ(QTEMP/FTPINPUT) OBJTYPE(*FILE)
31  MONMSG  MSGID(CPF0000) EXEC(+
             CRTPF FILE(QTEMP/FTPINPUT) RCDLEN(112) +
                    OPTION(*NOSRC *NOLIST))

32  CPYF    FROMFILE(FTPSRC) TOFILE(QTEMP/FTPINPUT) +
             FROMMBR(MBR001) MBROPT(*REPLACE) +
             FMTOPT(*CVTSRC)

33  CALL    PGM(RUNFTPCMDS) PARM('QSRV2')

    .
    .
    .
99  ENDPGM

In my "FTP program", RUNFTPCMDS shown below, is rather simple. I override the file I copied FTP subcommands into, QTEMP/FTPINPUT, to the name input on line 3. I clear the file FTPOUTPUT in QTEMP on line 4, and if it does not exist I create it on line 5. On line 6 I override QTEMP/FTPOUTPUT to the name output. As I now have files called input and output I execute the FTP command connecting to the remote system I passed in the &FTP_HOST variable, which in our scenario is QSRV2, line 7. As the FTP commands are executed the output is written to the output file. I want to copy this output to a spool file. I override the QSYSPRT spool file on line 8, and on line 9 I copy the contents of the output file, QTEMP/FTPOUTPUT, to it. As I have generated a spool file I can view this later to ensure that the FTP subcommands were executed without errors.

01  PGM     PARM(&FTP_HOST)

02  DCL     VAR(&FTP_HOST) TYPE(*CHAR) LEN(30)

03  OVRDBF  FILE(INPUT) TOFILE(QTEMP/FTPINPUT) +
              OVRSCOPE(*CALLLVL)

04  CLRPFM  FILE(QTEMP/FTPOUTPUT)
05  MONMSG  MSGID(CPF0000) EXEC(+
              CRTPF FILE(QTEMP/FTPOUTPUT) RCDLEN(132) +
                      OPTION(*NOSRC *NOLIST))

06  OVRDBF  FILE(OUTPUT) TOFILE(QTEMP/FTPOUTPUT) +
              OVRSCOPE(*CALLLVL)

07  FTP     RMTSYS(&FTP_HOST)

08  OVRPRTF FILE(QSYSPRT) PAGESIZE(*N 132) +
              OUTQ(FTPUSER) HOLD(*YES) USRDTA('ftp_log')    
 
09  CPYF    FROMFILE(QTEMP/FTPOUTPUT) TOFILE(QSYSPRT) +
              MBROPT(*ADD)

10  DLTOVR  FILE(*ALL)
11  MONMSG  MSGID(CPF0000)

12  ENDPGM

I recommend that the program called, MYLIB/PGM1, does a submit job to call the program that will process the copied files. I do this for two reasons:

  1. When a job is called from FTP there is no guarantee what the library list that the program will use, the output queue, etc.
  2. I do not want to leave the FTP connection open while the copied files are processed.

I just create a program, MYLIB/PGM1 in this example, that just submits to batch the processing program. An example is show below:

01  PGM

02  SBMJOB     CMD(CALL PGM(MYLIB/PGM2)) +
                 JOB(FTP_PROCSS) JOBD(MYLIB/MYJOBD) +
                 JOBQ(QBATCH1) USER(FTPUSER)

03  ENDPGM

With this structure I can now copy any FTP subcommands to the file QTEMP/FTPINPUT and call my program, RUNFTPCMDS, to execute them. I can run this without my input, in batch, and even run it as a job submitted from the job scheduler at specific times during the day.

 

You can learn more about this on IBM's website:

 

Other in this series:

 

This article was written for IBM i 7.1.

3 comments:

  1. I like this method. Is it possible to create this so that it will work with non IBM I servers?

    ReplyDelete
    Replies
    1. Yes, you can use this approach with any FTP server.

      I use a program similar to the one above on a daily basis to connect and retrieve files from two Windows FTP servers.

      Delete
  2. Good explanaition of FTP transfer.
    Just to add some details:

    If the target server is Linux oder Windows, trailing blanks in records will be truncated.
    To create a target file with all records in full length, add this before put:
    locsite trim 0

    If the target system is an IBMi, they will switch to extended FTP if the OS level is above stone age. Means they will move away to a random port out of a given range.
    This can be a problem if there is a firewall in between, and only the FTP port is open. The effect will be confusing: Opening FTP works, log in works, first command fails. So if this happens, look for a firewall and red lines in it's log. If the admins refuse to open a port range on the firewall, you have a problem.
    Solution: Switch off extended FTP mode using this line immediately after the user/password line:
    sendepsv 0
    And to make put/get commands work 100%, not only 99%, you may additionally turn off the passive mode completely:
    sendpasv 0

    ReplyDelete

To prevent "comment spam" all comments are moderated.
Learn about this website's comments policy here.

Some people have reported that they cannot post a comment using certain computers and browsers. If this is you feel free to use the Contact Form to send me the comment and I will post it for you, please include the title of the post so I know which one to post the comment to.