 |
IDS Forum
Re: HPL and compressed files
Posted By: Pravin Kedia
Date: Tuesday, 30 November 2004, at 6:54 p.m.
In Response To: HPL and compressed files (JHAYS2@sears.com)
--0__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E
Content-type: multipart/alternative;
Boundary="1__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E"
--1__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable
Hi Jhays,
I think it is the proble with the data that your are trying to insert.
It is failing on files 3, 4 and 5.
So can you try loading these files one by one using onpload and see if =
it
fails sometimes/always.
Try loading these test files almost 5 times.
Also try adding -L 5 option at the end of the onpload command and pleas=
e
send me the log files.
Also do you have any access to latest 9.40 release. Can you please try
loading these files using that version.
9.21 version is quite old.
Also can you please contact our tech support people so that they open a=
case that will help me in working more closely with this case.
Regards,
Pravin.
=
JHAYS2@sears.com =
Sent by: =
forum.subscriber@ =
To
iiug.org ids@iiug.org =
=
cc
=
11/23/2004 01:47 Subj=
ect
PM HPL and compressed files [3764] =
=
=
=
=
=
=
Hello all,
I am trying to load some very large files (which are compressed) using
High Performance Loader. I finally got a load working using HPL and
uncompressing through a pipe. It loads the first 3 tables then dies wi=
th
the error below. I can't uncompress the file(s) before loading due to l=
ack
of storage.
My hardware and software is:
Informix Dynamic Server 2000 Version 9.21.UC4
SunOS nova 5.8 Generic_108528-08 sun4u sparc SUNW,Ultra-Enterprise
Wed Nov 17 15:00:06 CST 2004 Log file is:
/export/home/informix/logs/cr-ld-tables-ccnc-rep.logWed
- - - - - - - - - - - - - - - - -
Wed Nov 17 15:00:50 2004
15:00:50 pl_main.c, line 546, thread 11, proc id 22441, Segmentation
Violation.
15:00:50 PANIC: Attempting to bring system down
2: 2: cannot open
3: 3: cannot open
3: 3: cannot open
./cr-ld-tables.sh.hpl[41]: 22431 Killed
3: 3: cannot open
2: 2: cannot open
2: 2: cannot open
2: 2: cannot open
2: 2: cannot open
3: 3: cannot open
Broken Pipe
- - - - - - - - - - - - - - -
These are the compressed files I am trying to load.
-rw-r--r-- 1 informix informix 444 Nov 18 16:16 ps-note.unl.Z
-rw-r--r-- 1 informix informix 959 Nov 18 16:16
ps-location=3D5Ftbl.unl.Z
-rw-r--r-- 1 informix informix 42926 Nov 18 16:16 ps-rb-wf-msg.unl.=
Z
-rw-r--r-- 1 informix informix 156838 Nov 18 16:16 ps-rc-case.unl.Z
-rw-r--r-- 1 informix informix 89901 Nov 18 16:16
ps-rc-case-note.unl.Z
-rw-r--r-- 1 informix informix 3 Nov 18 16:16
ps-rc-resolution.unl.Z
-rw-r--r-- 1 informix informix 390 Nov 18 16:16 ps-rd-site.unl.Z
-rw-r--r-- 1 informix informix 1121 Nov 18 16:16
ps-rf-grp-member.unl.Z
-rw-r--r-- 1 informix informix 661 Nov 18 16:16
ps-rf-provider-grp.unl.Z
-rw-r--r-- 1 informix informix 1219 Nov 18 16:16 ps-sr-msg-area.un=
l.Z
I can load the first two, but it fails on the third sometimes. I have
removed the third from the list and it fails on what is 4th above. Sam=
e
for the 5th. However, if I remove the first 5 files above, it will loa=
d
the next 5. So, bottom line, I can load files 1, 2, and sometimes 3, a=
nd
also 6, 7, 8, 9, 10, which are the smaller files. However, these are t=
est
files. The real files are all much larger.
- - - - - - - - - - - - - - -
Here is a good output log:
Wed Nov 17 15:00:39 2004
SHMBASE 0xa000000
CLIENTNUM 0x49010000
Session ID 91
Load Database -> ccnc-rep
Load Table -> ps-rb-wf-msg
Device Array -> ps-rb-wf-msg
Record Mapping -> ps-rb-wf-msg
Convert Reject -> /tmp/ps-rb-wf-msg.rej
Database Load Completed -- Processed 406 Records
Records Inserted-> 406
Detected Errors--> 0
Engine Rejected--> 0
Wed Nov 17 15:00:41 2004
- - - - - - - - - - - - - -
Here is the bad output log:
Wed Nov 17 15:00:48 2004
SHMBASE 0xa000000
CLIENTNUM 0x49010000
Session ID 93
Load Database -> ccnc-rep
Load Table -> ps-rc-case
Device Array -> ps-rc-case
Record Mapping -> ps-rc-case
Convert Reject -> /tmp/ps-rc-case.rej
- - - - - - - - - - - - - - - - - -
My device in ipload is defined as:
PIPE zcat <compressed file name>
(The above "<>" are not redirection symbols)
The logic is:
Create array of table names
Delete all 10 tables
Create all 10 tables
Start loop
Create piped file
Call onpload as:
onpload -p ccnc-rep -j ${TBL[$I]} -flc -i $COMMITCT -l
~/scripts/ccnc=3D5Frep/logs/${TBL[$I]}.log
Remove piped file
End loop
Create indexes
I have varied the commit count from 500 down to 1 in steps, but it stil=
l
fails on the same files. Does anyone have a suggestion other than
upgrading IDS (which is being seriously considered at this time)?
Thanks all.
=
--1__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline
Content-transfer-encoding: quoted-printable
<html><body>
<p>Hi Jhays,<br>
<br>
I think it is the proble with the data that your are trying to insert.<=
br>
It is failing on files 3, 4 and 5.<br>
<br>
So can you try loading these files one by one using onpload and see if =
it fails sometimes/always.<br>
Try loading these test files almost 5 times.<br>
<br>
Also try adding -L 5 option at the end of the onpload command and pleas=
e send me the log files.<br>
<br>
Also do you have any access to latest 9.40 release. Can you please try =
loading these files using that version.<br>
9.21 version is quite old.<br>
<br>
Also can you please contact our tech support people so that they open a=
case that will help me in working more closely with this case.<br>
<br>
Regards,<br>
Pravin.<br>
<br>
<img src=3D"cid:10__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com" width=3D=
"16" height=3D"16" alt=3D"Inactive hide details for JHAYS2@sears.com">J=
HAYS2@sears.com<br>
<br>
<br>
<table width=3D"100%" border=3D"0" cellspacing=3D"0" cellpadding=3D"0">=
<tr valign=3D"top"><td style=3D"background-image:url(cid:20__=3D08BBE5C=
FDF11E26E8f9e8a93df938@us.ibm.com); background-repeat: no-repeat; " wid=
th=3D"40%">
<ul>
<ul>
<ul>
<ul><b><font size=3D"2">JHAYS2@sears.com</font></b><font size=3D"2"> </=
font><br>
<font size=3D"2">Sent by: forum.subscriber@iiug.org</font>
<p><font size=3D"2">11/23/2004 01:47 PM</font></ul>
</ul>
</ul>
</ul>
</td><td width=3D"60%">
<table width=3D"100%" border=3D"0" cellspacing=3D"0" cellpadding=3D"0">=
<tr valign=3D"top"><td width=3D"1%" valign=3D"middle"><img src=3D"cid:3=
0__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com" border=3D"0" height=3D"=
1" width=3D"58" alt=3D""><br>
<div align=3D"right"><font size=3D"2">To</font></div></td><td width=3D"=
100%"><img src=3D"cid:30__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com" =
border=3D"0" height=3D"1" width=3D"1" alt=3D""><br>
<font size=3D"2">ids@iiug.org</font></td></tr>
<tr valign=3D"top"><td width=3D"1%" valign=3D"middle"><img src=3D"cid:3=
0__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com" border=3D"0" height=3D"=
1" width=3D"58" alt=3D""><br>
<div align=3D"right"><font size=3D"2">cc</font></div></td><td width=3D"=
100%"><img src=3D"cid:30__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com" =
border=3D"0" height=3D"1" width=3D"1" alt=3D""><br>
</td></tr>
<tr valign=3D"top"><td width=3D"1%" valign=3D"middle"><img src=3D"cid:3=
0__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com" border=3D"0" height=3D"=
1" width=3D"58" alt=3D""><br>
<div align=3D"right"><font size=3D"2">Subject</font></div></td><td widt=
h=3D"100%"><img src=3D"cid:30__=3D08BBE5CFDF11E26E8f9e8a93df938@us.ibm.=
com" border=3D"0" height=3D"1" width=3D"1" alt=3D""><br>
<font size=3D"2">HPL and compressed files [3764]</font></td></tr>
</table>
<table border=3D"0" cellspacing=3D"0" cellpadding=3D"0">
<tr valign=3D"top"><td width=3D"58"><img src=3D"cid:30__=3D08BBE5CFDF11=
E26E8f9e8a93df938@us.ibm.com" border=3D"0" height=3D"1" width=3D"1" alt=
=3D""></td><td width=3D"336"><img src=3D"cid:30__=3D08BBE5CFDF11E26E8f9=
e8a93df938@us.ibm.com" border=3D"0" height=3D"1" width=3D"1" alt=3D""><=
/td></tr>
</table>
</td></tr>
</table>
<br>
<tt>Hello all,<br>
<br>
I am trying to load some very large files (which are compressed) using<=
br>
High Performance Loader. I finally got a load working using HPL a=
nd<br>
uncompressing through a pipe. It loads the first 3 tables then di=
es with<br>
the error below. I can't uncompress the file(s) before loading due to l=
ack<br>
of storage.<br>
<br>
My hardware and software is:<br>
Informix Dynamic Server 2000 Version 9.21.UC4<br>
SunOS nova 5.8 Generic_108528-08 sun4u sparc SUNW,Ultra-Enterprise<br>
<br>
Wed Nov 17 15:00:06 CST 2004 Log file is:<br>
/export/home/informix/logs/cr-ld-tables-ccnc-rep.logWed<br>
- - - - - - - - - - - - - - - - - <br>
Wed Nov 17 15:00:50 2004<br>
<br>
15:00:50 pl_main.c, line 546, thread 11, proc id 22441, Segmentat=
ion <br>
Violation.<br>
15:00:50 PANIC: Attempting to bring system down<br>
2: 2: cannot open<br>
3: 3: cannot open<br>
3: 3: cannot open<br>
./cr-ld-tables.sh.hpl[41]: 22431 Killed<br>
3: 3: cannot open<br>
2: 2: cannot open<br>
2: 2: cannot open<br>
2: 2: cannot open<br>
2: 2: cannot open<br>
3: 3: cannot open<br>
Broken Pipe<br>
- - - - - - - - - - - - - - - <br>
These are the compressed files I am trying to load.<br>
<br>
-rw-r--r-- 1 informix informix 444 Nov 18 16:16 ps=
-note.unl.Z<br>
-rw-r--r-- 1 informix informix 959 Nov 18 16:16 <b=
r>
ps-location=3D5Ftbl.unl.Z<br>
-rw-r--r-- 1 informix informix 42926 Nov 18 16:16 ps-rb-w=
f-msg.unl.Z<br>
-rw-r--r-- 1 informix informix 156838 Nov 18 16:16 ps-rc-c=
ase.unl.Z<br>
-rw-r--r-- 1 informix informix 89901 Nov 18 16:16 <br>
ps-rc-case-note.unl.Z<br>
-rw-r--r-- 1 informix informix 3 Nov 18 16:=
16 <br>
ps-rc-resolution.unl.Z<br>
-rw-r--r-- 1 informix informix 390 Nov 18 16:16 ps=
-rd-site.unl.Z<br>
-rw-r--r-- 1 informix informix 1121 Nov 18 16:16 <b=
r>
ps-rf-grp-member.unl.Z<br>
-rw-r--r-- 1 informix informix 661 Nov 18 16:16 <b=
r>
ps-rf-provider-grp.unl.Z<br>
-rw-r--r-- 1 informix informix 1219 Nov 18 16:16 ps=
-sr-msg-area.unl.Z<br>
<br>
I can load the first two, but it fails on the third sometimes. I =
have <br>
removed the third from the list and it fails on what is 4th above. &nbs=
p;Same <br>
for the 5th. However, if I remove the first 5 files above, it wil=
l load<br>
the next 5. So, bottom line, I can load files 1, 2, and sometimes=
3, and <br>
also 6, 7, 8, 9, 10, which are the smaller files. However, these =
are test <br>
files. The real files are all much larger.<br>
- - - - - - - - - - - - - - -<br>
Here is a good output log:<br>
Wed Nov 17 15:00:39 2004<br>
<br>
SHMBASE 0xa000000<br>
CLIENTNUM 0x49010000<br>
Session ID 91<br>
<br>
Load Database -> ccnc-rep<br>
Load Table -> ps-rb-wf-msg<br>
Device Array -> ps-rb-wf-msg<br>
Record Mapping -> ps-rb-wf-msg<br>
Convert Reject -> /tmp/ps-rb-wf-msg.rej<br>
<br>
Database Load Completed -- Processed 406 Records<br>
Records Inserted-> 406<br>
Detected Errors--> 0<br>
Engine Rejected--> 0<br>
<br>
Wed Nov 17 15:00:41 2004<br>
- - - - - - - - - - - - - -<br>
Here is the bad output log:<br>
Wed Nov 17 15:00:48 2004<br>
<br>
SHMBASE 0xa000000<br>
CLIENTNUM 0x49010000<br>
Session ID 93<br>
<br>
Load Database -> ccnc-rep<br>
Load Table -> ps-rc-case<br>
Device Array -> ps-rc-case<br>
Record Mapping -> ps-rc-case<br>
Convert Reject -> /tmp/ps-rc-case.rej<br>
- - - - - - - - - - - - - - - - - -<br>
My device in ipload is defined as:<br>
PIPE zcat <compressed file name><br>
(The above "<>" are not redirection symbols)<br>
<br>
The logic is:<br>
<br>
Create array of table names<br>
Delete all 10 tables<br>
Create all 10 tables<br>
Start loop<br>
Create piped file<br>
Call onpload as:<br>
onpload -p ccnc-rep -j ${TBL[$I]} -flc -i $COMMITCT -l <=
br>
~/scripts/ccnc=3D5Frep/logs/${TBL[$I]}.log<br>
Remove piped file<br>
End loop<br>
Create indexes<br>
<br>
I have varied the commit count from 500 down to 1 in steps, but it stil=
l <br>
fails on the same files. Does anyone have a suggestion other than=
<br>
upgrading IDS (which is being seriously considered at this time)?<br>
<br>
Thanks all.<br>
<br>
</tt><br>
</body></html>=
--1__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E--
--0__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E
Content-type: image/gif;
name="graycol.gif"
Content-Disposition: inline; filename="graycol.gif"
Content-ID: <10__=08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com>
Content-transfer-encoding: base64
R0lGODlhEAAQAKECAMzMzAAAAP///wAAACH5BAEAAAIALAAAAAAQABAAAAIXlI+py+0PopwxUbpu
ZRfKZ2zgSJbmSRYAIf4fT3B0aW1pemVkIGJ5IFVsZWFkIFNtYXJ0U2F2ZXIhAAA7
--0__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E
Content-type: image/gif;
name="pic18658.gif"
Content-Disposition: inline; filename="pic18658.gif"
Content-ID: <20__=08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com>
Content-transfer-encoding: base64
R0lGODlhWABDALP/AAAAAK04Qf79/o+Gm7WuwlNObwoJFCsoSMDAwGFsmIuezf///wAAAAAAAAAA
AAAAACH5BAEAAAgALAAAAABYAEMAQAT/EMlJq704682770RiFMRinqggEUNSHIchG0BCfHhOjAuh
EDeUqTASLCbBhQrhG7xis2j0lssNDopE4jfIJhDaggI8YB1sZeZgLVA9YVCpnGagVjV171aRVrYR
RghXcAGFhoUETwYxcXNyADJ3GlcSKGAwLwllVC1vjIUHBWsFilKQdI8GA5IcpApeJQt8L09lmgkH
LZikoU5wjqcyAMMFrJIDPAKvCFletKSev1HBw8KrxtjZ2tvc3d5VyKtCKW3jfz4uMKmq3xu4N0nK
BVoJQmx2LGVOmrqNjjJf2hHAQo/eDwJGTKhQMcgQEEAnEjFS98+RnW3smGkZU6ncCWav/4wYOnAI
TihRL/4FEwbp28BXMMcoscQCVxlepL4IGDSCyJyVQOu0o7CjmLN50OZlqWmyFy5/6yBBuji0AxFR
M00oQAqNIstqI6qKHUsWRAEAvagsmfUEAImyxgbmUpJk3IklNUtJOUAVLoUr1+wqDGTE4zk+T6FG
uQb3SizBCwatiiUgCBN8vrz+zFjVyQ8FWkOlg4NQiZMB5QS8QO3mpOaKnL0Z2EKvNMSILEThKhCg
zMKPVxYJh23qm9KNW7pArPynMqZDiErsTMqI+LRi3QAgkFUbXpuFKhSYZALd0O5RKa2z9EYKBbpb
qxIKsjUPRgD7I2XYV6wyrOw92ykExP8NW4URhknC5dKGE4v4NENQj2jXjmfNgOZDaXb5glRmXQ33
YEWQYNcZFnrYcIQLNzyTFDQNkXIff0ExVlY4srziQk43inZgL4rwxxINMvpFFAz1KOODHiu+4aEw
NEjFl5B3JIKWKF3k6I9bfUGp5ZZcdunll5IA4cuHvQQJ5gcsoCWOOUwgltIwAKRxJgbIkJAQZEq0
2YliZnpZZ4BH3CnYOXldOUOfQoYDqF1LFHbXCrO8xmRsfoXDXJ6ChjCAH3QlhJcT6VWE6FCkfCco
CgrMFsROrIEX3o2whVjWDjoJccN3LdggSGXLCdLEgHr1lyU3O3QxhgohNKXJCWv8JQr/PDdaqd6w
2rj1inLiGeiCJoDspAoQlYE6QWLSECehcWIYxIQES6zhbn1iImTHEQyqJ4eIxJJoUBc+3CbBuwZE
V5cJPPkIjFDdeEabQbd6WgICTxiiz0f5dBKquXF6k4senwEhYGnKEFJeGrxUZy8dB8gmAXI/sPvH
ESfCwVt5hTgYiqQqtdRNHQIU1PJ33ZqmzgE90OwLaoJcnMop1WiMmgkPHQRIrwgFuNV90A3doNKT
mrKIN07AnGcI9BQjhCBN4RfA1qIZnMqorJCogKfGQnxSCDilTVIA0yl5ciTovgLuBDKFUDE9aQcw
9SA+rjSNf9/M1gxrj6VwDTS0IUSElMzBfsj0NFXR2kwsV1A5IF1grLgLL/r1R40BZEnuBWgmQEyb
jqRwSAt6bqMCOFkvKFN2GPPkUzIm/SCF8z8pVzpbjVnMsy0vOr1hw3SaSRUhpY09v0z0J1FnwzPl
fmh+xl4WtR0zGu24I4KbMQm3lnVu2oNWxI9W/lcyzA+mCKF4DBikxb/+UWtOGRiFP8qEwAayIgIA
Ow==
--0__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E
Content-type: image/gif;
name="ecblank.gif"
Content-Disposition: inline; filename="ecblank.gif"
Content-ID: <30__=08BBE5CFDF11E26E8f9e8a93df938@us.ibm.com>
Content-transfer-encoding: base64
R0lGODlhEAABAIAAAAAAAP///yH5BAEAAAEALAAAAAAQAAEAAAIEjI8ZBQA7
--0__=08BBE5CFDF11E26E8f9e8a93df938690918c08BBE5CFDF11E26E--
Messages In This Thread
IDS Forum is maintained by Administrator with WebBBS 5.12.
|
|
