Data Queue in AS400

Data Queue

Data queue is the type of system object that one can create to which a program or procedure can send data and from which another program or procedure can receive data.

The program that receives the data can be already waiting or can receive the data later.

Advantages of using Data Queue

Data queues are the fastest means of asynchronous communication between two jobs.
Using a data queue to send and receive data requires less overhead than any other means such as database files, message queues, or data areas.
We can send, receive, clear, retrieve the data or description of a data queue and change some attributes of the data queue by calling APIs QSNDDTAQ, QRCVDTAQ, QCLRDTAQ, QMHRDQM, QMHQRDQD, and QMHQCDQ from any high-level program or procedure without the need to call any CL program or exiting from the High-level program.
More than one job can receive the data from the same data queue. We get some advantage here when we process a large number of entries which one job cannot be able to process by maintaining the performance.
Data queues have the ability to add the sender id to each message being placed on the data queue. The sender id contains the qualified job name and the current user profile.
We can set the timeout such that the job waits until an entry arrives on the respective data queue during data receive from the data queue.
We can journal the data queues so that we can recover the data queue to a consistent state at any time when any crash occurs.

Example 1

First example shows how the data queue works in the AS400 system. Let's suppose any number of jobs can send the entries to the data queue and these entries are handled by the server job.

A real-world example would be all the processed checks are coming to the single batch job that would do the printing of checks.

Example 2

Let's suppose the main job gets the requests for some work and sends the entries to the data queue by calling the API QSNDDTAQ. The server job receives the entry from the data queue by calling the QRCVDTAQ API and processing the data and then reverting back to the main job by using another data queue.

Server Jobs

Data queues allow the main job to transfer the work to the server jobs. This frees the main job to take the next request. Please note that any number of server jobs can receive from the same data queue.

When no entries are placed on a data queue, server jobs can perform the following things.

Wait until an entry is placed on the data queue.

Wait for a specific period of time, if the entry is not received even after waiting for the specific time then continue processing.

Do not wait, return immediately.

Data Queues with Display Files

Data queues can also be used when a program needs to wait for input from display files and data queues at the same time.

You need to specify the data queue name on the DTAQ parameter for the Create Display File *CRTDSPF),Change Display File (CHGDSPF), Override Display File (OVRDSPF).

We can specify the data queue that will have entries placed on it when an enabled command key or the ENTER key is pressed from a DSPF. Jobs running on the system can place the entry on the data queue using API QSNDDTAQ and another program receives the entry being placed on the data queue by using QRCVDTAQ API and process the entries as it was placed in the data queue by the display file or the QSNDDTAQ API.

                         Create Display File (CRTDSPF)                        
                                                                              
Type choices, press Enter.                                                    
                                                                              
File . . . . . . . . . . . . . . FILE                                         
  Library  . . . . . . . . . . .                  *CURLIB                     
Source file  . . . . . . . . . . SRCFILE        *NONE                         
  Library  . . . . . . . . . . .                                              
Source member  . . . . . . . . . SRCMBR         *FILE                         
Generation severity level  . . . GENLVL         20                            
Flagging severity level  . . . . FLAG           0                             
Display device . . . . . . . . . DEV            *REQUESTER                    
                          + for more values                                   
Text 'description' . . . . . . . TEXT           *SRCMBRTXT                    
                                                                              
                                                                              
                           Additional Parameters                              
                                                                              
Source listing options . . . . . OPTION                                       
                          + for more values                                   
Maximum devices  . . . . . . . . MAXDEV         1            
Enhanced display . . . . . . . . ENHDSP         *YES         
Restore display  . . . . . . . . RSTDSP         *NO          
Defer write  . . . . . . . . . . DFRWRT         *YES         
Character identifier:            CHRID                       
  Graphic character set  . . . .                *DEVD        
  Code page  . . . . . . . . . .                             
Decimal format . . . . . . . . . DECFMT         *JOB         
SFLEND text  . . . . . . . . . . SFLENDTXT      *MSG         
Maximum file wait time . . . . . WAITFILE       *IMMED       
Maximum record wait time . . . . WAITRCD        *NOMAX       
Data queue . . . . . . . . . . . DTAQ           *NONE        
  Library  . . . . . . . . . . .                             
Share open data path . . . . . . SHARE          *NO          
Sort sequence  . . . . . . . . . SRTSEQ         *JOB         
  Library  . . . . . . . . . . .                     
Language ID  . . . . . . . . . . LANGID         *JOB       
Record format level check  . . . LVLCHK         *YES       
Authority  . . . . . . . . . . . AUT            *LIBCRTAUT 
Replace file . . . . . . . . . . REPLACE        *YES

Data queue with an output queue

We can associate the data queue to an output queue by using the Create output queue (CRTOUTQ) or Change output queue (CHGOUTQ) command. The system placed the entries in the associated data queue when spool files are in Ready (RDY) status on the output queue. A program can check when a spool file is in RDY state in an OUTQ by using the Receive data queue API (QRCVDTAQ) to receive data from the data queue.

                          Create Output Queue (CRTOUTQ)                      
                                                                             
 Type choices, press Enter.                                                  
                                                                             
 Output queue . . . . . . . . . . OUTQ                                       
   Library  . . . . . . . . . . .                  *CURLIB                   
 Maximum spooled file size:       MAXPAGES                                   
   Number of pages  . . . . . . .                *NONE                       
   Starting time  . . . . . . . .                                            
   Ending time  . . . . . . . . .                                            
                           + for more values                                 
 Order of files on queue  . . . . SEQ            *FIFO                       
 Remote system  . . . . . . . . . RMTSYS         *NONE                       
                                                                             
                                                                             
                                                                             
 User defined option  . . . . . . USRDFNOPT      *NONE                       
                           + for more values                                 
 User defined object:             USRDFNOBJ                            
   Object . . . . . . . . . . . .                *NONE                 
     Library  . . . . . . . . . .                                      
   Object type  . . . . . . . . .                                      
 Spooled file ASP . . . . . . . . SPLFASP        *SYSTEM               
 Text 'description' . . . . . . . TEXT           *BLANK                
                                                                       
                                                                       
                            Additional Parameters                      
                                                                       
 Display any file . . . . . . . . DSPDTA         *NO                   
 Job separators . . . . . . . . . JOBSEP         0                     
 Operator controlled  . . . . . . OPRCTL         *YES                  
 Data queue . . . . . . . . . . . DTAQ           *NONE                 
   Library  . . . . . . . . . . .                                      
 Authority to check . . . . . . . AUTCHK         *OWNER

Prerequisites for using data queues

Before using a data queue, we should create the data queue using the command Create Data Queue command (CRTDTAQ).

                          Create Data Queue (CRTDTAQ)                    
                                                                         
Type choices, press Enter.                                               
                                                                         
Data queue . . . . . . . . . . . DTAQ         > DTAQ1                    
  Library  . . . . . . . . . . .              >   EASYCLASS1             
Type . . . . . . . . . . . . . . TYPE           *STD                     
Maximum entry length . . . . . . MAXLEN       > 128                      
Force to auxiliary storage . . . FORCE          *NO                      
Sequence . . . . . . . . . . . . SEQ            *FIFO                    
Key length . . . . . . . . . . . KEYLEN                                  
Include sender ID  . . . . . . . SENDERID       *NO                      
Queue size:                      SIZE                                    
  Maximum number of entries  . .                *MAX16MB                 
  Initial number of entries  . .                16                       
Automatic reclaim  . . . . . . . AUTORCL        *NO                      
Remote data queue  . . . . . . . RMTDTAQ                                 
  Library  . . . . . . . . . . .                                         
Remote location  . . . . . . . . RMTLOCNAME                              
Relational database  . . . . . . RDB                                     
APPC device description  . . . . DEV            *LOC                            
Local location . . . . . . . . . LCLLOCNAME     *LOC                            
Mode . . . . . . . . . . . . . . MODE           *NETATR                         
Remote network identifier  . . . RMTNETID       *LOC                            
Text 'description' . . . . . . . TEXT         > 'Test data queue'             
                                                                                
                                                                                
                           Additional Parameters                                
                                                                                
Authority  . . . . . . . . . . . AUT            *LIBCRTAUT

The MAXLEN parameter can have 1-64512 characters of the entries that can be sent to the data queue and this parameter is valid only when the Type specified is *STD. If the data queue is associated with the output queue, the maximum length value should be at least 128.

The possible values of Type are *STD (a standard data queue is created, a MAXLEN parameter is required with the use of this value.), *DDM(A DDM data queue is created. This value requires the name of the remote data queue accessed using the RMTDTAQ parameter and the name of the remote target system that the data queue is located on the RMTLOCNAME parameter.), *DSP(A display data queue is created. The data queue will be used with a display file. The MAXLEN parameter is required for using this type. An interactive job waiting on this data queue would be eligible for an inactive timeout.

Creating a Standard Data Queue

The below command creates the data queue named DTAQ1 in the current library.

CRTDTAQ   DTAQ(DTAQ1)  MAXLEN(128) TEXT('Test data queue')

Data Queue created.

===>                                                    
                                                        
F3=Exit   F4=Prompt   F9=Retrieve   F12=Cancel   F13=Inf
F23=Set initial menu                                    
Object DTAQ1 type *DTAQ created in library EASYCLASS1.

WRKOBJ EASYCLASS1/DTAQ1

                              Work with Objects                    
                                                                   
Type options, press Enter.                                         
  2=Edit authority        3=Copy   4=Delete   5=Display authority  
  8=Display description   13=Change description                    
                                                                   
Opt  Object      Type      Library     Attribute   Text            
     DTAQ1       *DTAQ     EASYCLASS1              Test data queue

Creating a DDM Data Queue to Access a Data Queue at Another system

The below command will create the DDM data queue named DTAQ2 in the EASYCLASS1 library on the source system. This DDM data queue uses the remote_location_name to access the remote data queue Remote_Data_Queue_Name in library Remote_Librray.

CRTDTAQ   DTAQ(EASYCLASS1/DTAQ2)  TYPE(*DDM)
RMTDTAQ(<Remote_Liraryb>/<Remote_Data_Queue_Name>)  RMTLOCNAME(<Remote_Location_Name>)

Creating a Data Queue Specifying Size Attributes and Automatic Reclaim

Below command will create the data queue named DTAQ3 in library EASYCLASS1 of type *STD (standard data queue) of MAXLEN 128 and maximum entries allowed on a data queue is 16 MB and an initial number of entries as 10 and Auto reclaim set as *YES means once 16MB storage is reached, the data queue gets automatically reclaimed.

                          Create Data Queue (CRTDTAQ)                        
                                                                             
Type choices, press Enter.                                                   
                                                                             
Data queue . . . . . . . . . . . DTAQ         > DTAQ3                       
  Library  . . . . . . . . . . .              >   EASYCLASS1                 
Type . . . . . . . . . . . . . . TYPE           *STD                         
Maximum entry length . . . . . . MAXLEN       > 128                          
Force to auxiliary storage . . . FORCE          *NO                          
Sequence . . . . . . . . . . . . SEQ            *FIFO                        
Include sender ID  . . . . . . . SENDERID       *NO                          
Queue size:                      SIZE                                        
  Maximum number of entries  . .              > *MAX16MB                     
  Initial number of entries  . .              > 10                           
Automatic reclaim  . . . . . . . AUTORCL      > *YES                         
Text 'description' . . . . . . . TEXT           *BLANK

Comparisons with using Database files as Queues

Data queues must not be used for storing data for a longer duration. For this purpose, we must use database files instead.

We should include exception handling in case of using data queues to recover any entries not yet processed before the system going to crash.

We must delete or recreate the data queue once a day so that the data queue will return to its optimal size. Performance can be affected if too many entries existed on the data queue without being removed from it. The most efficient approach is to use auto reclaim *YES.

Managing the Storage used by the Data Queue

Smaller data queues have better performance than large ones. Each entry consumes some storage when sent to a data queue. The storage allocated would be the same as the MAXLEN value specified at the time of CRTDTAQ. When receiving an entry from a data queue, the data queue removes the entry, but it does not free the auxiliary storage. When new entries receive the data queue grows larger. If the data queue grows too large then delete the data queue by using the command Delete Data Queue (DLTDTAQ). After deletion, recreate the data queue by using the command Create Data Queue (CRTDATQ).

DLTDTAQ   DTAQ(DTAQ1)

There is another way to do it by using the SIZE and AUTORCL(*YES) parameter on the Create Data Queue (CRTDTAQ) command.

Allocating Data Queues

We must allocate the data queue by sing command Allocate object (ALCOBJ) before using it and it should be deallocated using command Deallocate object (DLCOBJ) when it is used so that the data queue is not accessed by more than one job at a time.

Read data from data queue using QRCVDTAQ api in RPG AS400
Write data to data queue using QSNDDTAQ API in RPG AS400

AS400 and SQL Tricks