IDS Forum

HPL and compressed files

Posted By: JHAYS2@sears.com
Date:

Hello all,

I am trying to load some very large files (which are compressed) using
High Performance Loader. I finally got a load working using HPL and
uncompressing through a pipe. It loads the first 3 tables then dies with
the error below. I can't uncompress the file(s) before loading due to lack
of storage.

My hardware and software is:
Informix Dynamic Server 2000 Version 9.21.UC4
SunOS nova 5.8 Generic_108528-08 sun4u sparc SUNW,Ultra-Enterprise

Wed Nov 17 15:00:06 CST 2004 Log file is:
/export/home/informix/logs/cr-ld-tables-ccnc-rep.logWed
- - - - - - - - - - - - - - - - -
Wed Nov 17 15:00:50 2004

15:00:50 pl_main.c, line 546, thread 11, proc id 22441, Segmentation
Violation.
15:00:50 PANIC: Attempting to bring system down
2: 2: cannot open
3: 3: cannot open
3: 3: cannot open
./cr-ld-tables.sh.hpl[41]: 22431 Killed
3: 3: cannot open
2: 2: cannot open
2: 2: cannot open
2: 2: cannot open
2: 2: cannot open
3: 3: cannot open
Broken Pipe
- - - - - - - - - - - - - - -
These are the compressed files I am trying to load.

-rw-r--r-- 1 informix informix 444 Nov 18 16:16 ps-note.unl.Z
-rw-r--r-- 1 informix informix 959 Nov 18 16:16
ps-location=5Ftbl.unl.Z
-rw-r--r-- 1 informix informix 42926 Nov 18 16:16 ps-rb-wf-msg.unl.Z
-rw-r--r-- 1 informix informix 156838 Nov 18 16:16 ps-rc-case.unl.Z
-rw-r--r-- 1 informix informix 89901 Nov 18 16:16
ps-rc-case-note.unl.Z
-rw-r--r-- 1 informix informix 3 Nov 18 16:16
ps-rc-resolution.unl.Z
-rw-r--r-- 1 informix informix 390 Nov 18 16:16 ps-rd-site.unl.Z
-rw-r--r-- 1 informix informix 1121 Nov 18 16:16
ps-rf-grp-member.unl.Z
-rw-r--r-- 1 informix informix 661 Nov 18 16:16
ps-rf-provider-grp.unl.Z
-rw-r--r-- 1 informix informix 1219 Nov 18 16:16 ps-sr-msg-area.unl.Z

I can load the first two, but it fails on the third sometimes. I have
removed the third from the list and it fails on what is 4th above. Same
for the 5th. However, if I remove the first 5 files above, it will load
the next 5. So, bottom line, I can load files 1, 2, and sometimes 3, and
also 6, 7, 8, 9, 10, which are the smaller files. However, these are test
files. The real files are all much larger.
- - - - - - - - - - - - - - -
Here is a good output log:
Wed Nov 17 15:00:39 2004

SHMBASE 0xa000000
CLIENTNUM 0x49010000
Session ID 91

Load Database -> ccnc-rep
Load Table -> ps-rb-wf-msg
Device Array -> ps-rb-wf-msg
Record Mapping -> ps-rb-wf-msg
Convert Reject -> /tmp/ps-rb-wf-msg.rej

Database Load Completed -- Processed 406 Records
Records Inserted-> 406
Detected Errors--> 0
Engine Rejected--> 0

Wed Nov 17 15:00:41 2004
- - - - - - - - - - - - - -
Here is the bad output log:
Wed Nov 17 15:00:48 2004

SHMBASE 0xa000000
CLIENTNUM 0x49010000
Session ID 93

Load Database -> ccnc-rep
Load Table -> ps-rc-case
Device Array -> ps-rc-case
Record Mapping -> ps-rc-case
Convert Reject -> /tmp/ps-rc-case.rej
- - - - - - - - - - - - - - - - - -
My device in ipload is defined as:
PIPE zcat <compressed file name>
(The above "<>" are not redirection symbols)

The logic is:

Create array of table names
Delete all 10 tables
Create all 10 tables
Start loop
Create piped file
Call onpload as:
onpload -p ccnc-rep -j ${TBL[$I]} -flc -i $COMMITCT -l
~/scripts/ccnc=5Frep/logs/${TBL[$I]}.log
Remove piped file
End loop
Create indexes

I have varied the commit count from 500 down to 1 in steps, but it still
fails on the same files. Does anyone have a suggestion other than
upgrading IDS (which is being seriously considered at this time)?

Thanks all.

Messages In This Thread

• HPL and compressed files
  JHAYS2@sears.com -- Tuesday, 23 November 2004, at 2:47 p.m.
  • Re: HPL and compressed files
    MARION SUSSUJEW -- Wednesday, 24 November 2004, at 3:53 p.m.
  • Re: HPL and compressed files
    Pravin Kedia -- Tuesday, 30 November 2004, at 6:54 p.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.