HI, Mark,
We already killed all the oninit processes and restarted IDS.
The following is some info I caught before we killed it, it may or may not
be usefull.
By the way, we can successfully finish it by using "lock table in
exclusive mode"
Hanging there, it did not accept any connections, commands,..... and
onstat -K show millions locks.
Thanks,
Frank
informix@erika $ onstat -g mem
IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) --
Up 2 days 21:03:56 -- 200
8880 Kbytes
Blocked:CKPT
Pool Summary:
name class addr totalsize freesize #allocfrag #freefrag
afpool V b000d020 4096 2128 2 1
tpcpool V b0756020 8192 3648 4 2
seqpool V b0771020 4096 2400 2 1
pnlpool V b0759020 8192 2536 5 2
sbtlist V b009f020 12288 5600 4 3
dstpool V b0755020 4096 1512 2 1
sqcrypto V b08ff020 4096 2272 2 1
ampool V b076b020 4096 1832 12 1
shmcon M 30001020 536576 6520 2 2
main_loop() V b0907020 372736 66224 83 40
sb_delundoq V b00bd020 28672 8424 4 3
XTF_mem V b07ad020 565248 7744 4 3
bf_prioswee V b0d57020 24576 4256 13 3
onmode_mon V b0d39020 24576 4256 13 3
lgflushpool V b0c59020 4096 2184 5 1
2 V b091b020 12288 4120 13 2
rsam V b012f020 308527104 84336 889 41
sscpool0 V b074f020 8192 2680 2 2
3 V b0938020 12288 4120 13 2
4 V b0955020 12288 3544 14 2
5 V b0962020 12288 2896 19 3
6 V b097d020 12288 2896 19 3
7 V b0994020 94208 5160 213 3
aslogflush V b0c5a020 24576 4256 13 3
aio V b0334020 5316608 364912 411 115
9 V b099b020 12288 4120 13 2
opcinstpool V b0758020 12288 1512 10 2
dbausrpool V b0775020 4096 1952 6 1
smartblob V b009e020 36864 13128 16 5
gls V b0749020 28672 5544 348 4
res-buff0 R 442bd020 1638424576 6520 2 2
DefConvWrit V b07ab020 8192 1016 102 2
resroutpool V b075b020 4096 1512 2 1
btscanner_0 V b0cf3020 24576 1376 21 4
sb_delq V b00a2020 28672 8424 4 3
resident R 40042020 69709824 6520 2 2
dictpool V b0750020 131072 1784 62 1
7*O0 V b0dd2020 4096 2448 1 1
mt V b000e020 1540096 5232 9060 77
10 V b09a7020 12288 2896 19 3
aggpool V b075a020 4096 1512 2 1
73 V b0c4d020 49152 5800 80 7
93 V b0998020 45056 4216 62 5
94 V b0de0020 12288 4120 13 2
procpool V b0752020 16384 2928 26 2
95 V b0df3020 12288 4120 13 2
extdpool V b0772020 4096 1256 4 1
69 V b099a020 12288 4120 13 2
global V b0003020 7315456 38120 1294 14
ctcpool V b0757020 4096 488 3 1
tmxapool V b0774020 4096 2384 3 1
sb_arcspace V b012c020 4096 416 8 1
sb_loheader V b00d8020 262144 25944 413 60
inhpool V b076d020 4096 1520 16 1
opcpool V b076c020 4096 376 34 1
Blkpool Summary:
name class addr size #blks
mt V b000fe10 1982464 44
global V b00036c0 0 0
informix@erika $ onstat -p
IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) --
Up 2 days 21:04:35 -- 200
8880 Kbytes
Blocked:CKPT
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits
%cached
1194011 3221512 32323277 96.31 305009 378823 6796070
95.51
isamtot open start read write rewrite delete
commit rollbk
32089058 960 3208713 9630965 3207687 493 0
45 0
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
16 0 0 0 0 0 8
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 1578.56 178.35 12 1615
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress
seqscans
13561 0 19276754 0 0 4 13
15
ixda-RA idx-RA da-RA RA-pgsused lchwaits
23 14848 0 14870 290
informix@erika $ onstat -g seg
IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) --
Up 2 days 21:04:56 -- 200
8880 Kbytes
Blocked:CKPT
Segment Summary:
id key addr size ovhd class blkused blkfree
7340036 1382041603 30000000 540672 672 M 132 0
9437187 1382041601 40000000 1708425216 269028 R* 417091 5
9437189 1382041602 b0000000 314572800 10248 V 73279 3521
6 1382041604 e0000000 33554432 1672 V 6482 1710
Total: - - 2057093120 - - 496984 5236
(* segment locked in memory)
informix@erika $ onstat -g rbm
IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) --
Up 2 days 21:05:19 -- 200
8880 Kbytes
Blocked:CKPT
Block bitmap for resident segment address 0x40000000:
address = 0x40000288, size(bits) = 417096
used = 417091, largest_free = 4294967295
0: startsearch = 17086, lastfree = -1
1: startsearch = 0, lastfree = 0
2: startsearch = 0, lastfree = 0
3: startsearch = 0, lastfree = 0
4: startsearch = 0, lastfree = 0
5: startsearch = 417091, lastfree = -1
0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2560:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2816:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3072:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3328:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
i
informix@erika $ onstat -g nbm
IBM Informix Dynamic Server Version 10.00.UC5 -- On-Line (CKPT REQ) --
Up 2 days 21:05:25 -- 200
8880 Kbytes
Blocked:CKPT
Block bitmap for virtual segment address 0xb0000000:
address = 0xb0000288, size(bits) = 76800
used = 73279, largest_free = 4294967295
0: startsearch = 3574, lastfree = 3797
1: startsearch = 3801, lastfree = -1
2: startsearch = 3795, lastfree = -1
3: startsearch = 3808, lastfree = 3904
4: startsearch = 3808, lastfree = 3895
5: startsearch = 76800, lastfree = -1
0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2560:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2816:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3072:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3328:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3584:ff7fffff ffffffff ffdfffff ffffffff ffffffff ffffffff ff7ffbaf
00000000
3840:00000001 fffffe00 00000000 00000000 00000000 00000000 00000000
00000000
4096:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
4352:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
4608:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
4864:000ff800 000000ff c07ffff8 0fe00000 00000000 00000000 00000000
00000000
5120:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000001
5376:fff80003 d8000000 20000000 00000000 0800001f f0007fff ffffffff
ffffffff
5632:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
5888:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
6144:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
6400:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
74496:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
74752:ffffffff ffffffff ffffffff ffffffff ffffffff ffff8000 00000000
00000000
75008:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
75264:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
75520:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
75776:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
76032:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
76288:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
76544:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
Block bitmap for virtual segment address 0xe0000000:
address = 0xe0000288, size(bits) = 8192
used = 6482, largest_free = 4294967295
0: startsearch = 1, lastfree = 0
1: startsearch = 0, lastfree = 0
2: startsearch = 0, lastfree = 0
3: startsearch = 0, lastfree = 0
4: startsearch = 0, lastfree = 0
5: startsearch = 6482, lastfree = -1
0:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
256:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
512:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
768:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1024:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1280:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1536:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
1792:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2048:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2304:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2560:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
2816:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3072:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3328:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3584:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
3840:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
4096:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
4352:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
4608:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
4864:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
5120:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
5376:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
5632:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
5888:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
6144:ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffffffff
6400:ffffffff ffffffff ffffc000 00000000 00000000 00000000 00000000
00000000
6656:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
6912:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
7168:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
7424:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
7680:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
7936:00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
On 10/16/06, Mark Jamison <majp51@yahoo.com> wrote:
>
>
> Hi Frank,
>
> Could you post an onstat -g ath when this happens?
>
> It's definitely a result of running out of memory while acquiring locks,
> but
> that shoulld not necessarily cause an engine hang.
>
>
>
> ------------------------------------------------------------------------------------
> Praise the Lord oh my soul, let my inmost being praise His holy name.
> -Psalm 103:1
>
> ----- Original Message ----
> From: FRANK <yunyaoqu@gmail.com>
> To: ids@iiug.org
> Sent: Monday, October 16, 2006 9:38:25 AM
> Subject: onmode -ky hangs [7630]
>
> We had a bulk LOAD -INSERT and caused the IDS hangs. see below log info.
>
> It remains hanging after I issued "onmode -ky".
>
> What should we do?
>
> Thanks,
> Frank
>
> ======
> 12:29:38 dynamically allocated 100000 locks
> 12:29:47 dynamically allocated 100000 locks
> 12:30:01 Checkpoint Completed: duration was 4 seconds.
> 12:30:01 Checkpoint loguniq 23, logpos 0x4f32018, timestamp: 0x1717af75
>
> 12:30:01 Maximum server connections 3
> 12:30:01 Contiguous shared memory segment allocation failed at 0xd0000000.
> Allocation successful at 0xe0000000.
> Check SHMBASE is consistent with the value in
> $INFORMIXDIR/etc/onconfig.std.
> If you are using the correct SHMBASE value in your ONCONFIG file, then
> consider this message informational only.
>
> 12:30:01 Dynamically allocated new virtual shared memory segment (size
> 32768KB)
> 12:30:01 dynamically allocated 100000 locks
> 12:30:10 dynamically allocated 100000 locks
> 12:30:20 dynamically allocated 100000 locks
> 12:30:29 shmat: [EMFILE][24]: out of shared memory segments, check system
> SHMSEG
> 12:30:29 out of virtual shared memory
>
> =======
> Program Name: onstat
> Build Version: 10.00.UC5
> Build Number: N202
> Build Host: ibm6c1b
> Build OS: AIX 5.2
> Build Date: Thu May 18 00:09:08 CDT 2006
> GLS Version: glslib-4.00.UC8
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>