Here's the telling bit, Kate nailed it. You onstat -g iov says:
AIO I/O vps:
class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors
msc 0 i 0.0 6 0 0 0 7 0.9 0
aio 0 s 18.9 60332 10763 49541 0 838 72.0 0
^^^
72.0 IO/WUP for this one lonely AIO VP means that 98.6% of your server's IO
requests are queued up waiting for service when the server is busy.
If you have 50 filesystem (ie COOKED) chunks, then (at least under 7.31
which cannot use DIRECT IO) you need between 75 and 100 AIO VPs in order to
properly service IO requests against this server without delays. By
improving the parameter settings I suggested you did improve the checkpoint
duration which was goal #1. However, we knowingly shifted much of the
checkpoint IO so that it happens slowly over time instead of all at once
during a checkpoint. Now, with not enough AIO VPs to service all those
requests, other IO bound requests like your big data load will suffer.
Solution, increase NUMAIOVPS to 100 and adjust from there. If after running
some normal processing and some peak processing onstat -g iov shows some
number of these aio vps with io/wup = 0.0 then you can reduce NUMAIOVPS by
that number. If, however, all of the AIO VPs show io/wup of 1.0 or greater,
then you still need to add more. If one or more show io/wup less than 1.0
and only a few are over 1.0 then you are cruising!
Art
On Tue, Mar 17, 2009 at 5:29 AM, KURT VAN GOMPEL <
kvangompel@caami-hziv.fgov.be> wrote:
> Hello,
>
> Thanks for the many responses.
> Hereby the information some of you asked:
> - chunks are on filesystem (50 chunks on server PRD, 10 on sever TST)
> - BUFFERS is set to 100000
> - NUMCPUVPS is 2
>
> Output of onstat on testserver while in checkpoint (47 secs):
>
> onstat -p
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 22619 35939 3961511 99.43 59737 132210 1068478 94.41
>
> isamtot open start read write rewrite delete commit rollbk
> 773583 182 187225 208215 75606 62917 82443 6092 0
>
> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> 0 0 0 0 0 0 0
>
> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> 0 0 0 11.99 5.98 6 21
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
> 2236 0 2372084 0 0 4 3527 11
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 11818 0 3244 15062 13
>
> ------------------------------------------------------------
> onstat -P
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
> partnum total btree data other resident dirty
> 0 79028 8289 1457 69282 0 0
> 1048578 2 1 1 0 0 0
> 1048579 4 2 2 0 0 0
> 1048622 5 2 3 0 0 0
> 1048623 4 2 2 0 0 0
> 1048625 2 1 1 0 0 0
> 1048626 1 1 0 0 0 0
> 1048628 2 1 1 0 0 0
> 1048632 1 1 0 0 0 0
> 1048635 1 1 0 0 0 0
> 1048641 1 1 0 0 0 0
> 1048642 1 1 0 0 0 0
> 1048643 1 1 0 0 0 0
> 1048646 1 1 0 0 0 0
> 1048650 3 3 0 0 0 0
> 1048659 1 0 1 0 0 0
> 1048662 1 0 1 0 0 0
> 2097154 5 2 3 0 0 0
> 2097155 9 4 5 0 0 0
> 2097156 3 1 2 0 0 0
> 2097157 2 1 1 0 0 0
> 2097158 2 1 1 0 0 0
> 2097160 2 1 1 0 0 0
> 2097164 3 1 2 0 0 0
> 2097165 1 1 0 0 0 0
> 2097167 1 1 0 0 0 0
> 2097168 2 1 1 0 0 0
> 2097173 1 1 0 0 0 0
> 2097174 1 1 0 0 0 0
> 2097175 1 1 0 0 0 0
> 2097177 1 1 0 0 0 0
> 2097178 1 1 0 0 0 0
> 2097181 6 4 2 0 0 0
> 2097182 3 3 0 0 0 0
> 2097189 20093 1136 13791 5166 0 110
> 7340034 804 563 241 0 0 0
>
> Totals: 100000 10033 15519 74448 0 110
>
> Percentages:
> Data 15.52
> Btree 10.03
> Other 74.45
>
> ------------------------------------------------------------
> onstat -F
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Fg Writes LRU Writes Chunk Writes
> 0 44607 12763
>
> address flusher state data
> 1a050514 0 C 2 = 0X2
> 1a050a10 1 I 0 = 0X0
> 1a050f0c 2 I 0 = 0X0
> 1a051408 3 I 0 = 0X0
> 1a051904 4 I 0 = 0X0
> 1a051e00 5 I 0 = 0X0
> 1a0522fc 6 I 0 = 0X0
> 1a0527f8 7 I 0 = 0X0
>
> states: Exit Idle Chunk Lru
>
> ------------------------------------------------------------
> onstat -R
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> 8 buffer LRU queue pairs priority levels
> # f/m pair total % of length LOW MED_LOW MED_HIGH HIGH
> 0 f 12500 99.8% 12480 9668 2788 24 0
> 1 m 0.2% 20 0 19 1 0
> 2 f 12500 99.9% 12488 9673 2802 12 1
> 3 m 0.1% 12 0 12 0 0
> 4 f 12501 99.9% 12491 9664 2812 14 1
> 5 m 0.1% 10 0 10 0 0
> 6 f 12500 99.9% 12489 9677 2794 16 2
> 7 m 0.1% 11 0 11 0 0
> 8 f 12500 100.0% 12499 9672 2813 13 1
> 9 m 0.0% 1 0 1 0 0
> 10 f 12500 99.8% 12480 9673 2790 15 2
> 11 m 0.2% 20 0 19 1 0
> 12 f 12499 99.9% 12490 9662 2812 14 2
> 13 m 0.1% 9 0 8 1 0
> 14 F 12500 99.9% 12489 9678 2802 9 0
> 15 m 0.1% 11 0 11 0 0
> 94 dirty, 100000 queued, 100000 total, 131072 hash buckets, 2048 buffer
> size
> start clean at 5% (of pair total) dirty, or 625 buffs dirty, stop at 2%
> 0 priority downgrades, 0 priority upgrades
> ------------------------------------------------------------
> onstat -l
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Physical Logging
> Buffer bufused bufsize numpages numwrits pages/io
> P-2 0 32 31771 996 31.90
>
> phybegin physize phypos phyused %used
>
> 600035 200000 71471 12204 6.10
>
> Logical Logging
> Buffer bufused bufsize numrecs numpages numwrits recs/pages pages/io
> L-2 0 32 879331 42042 1324 20.9 31.8
>
> Subsystem numrecs Log Space used
>
> OLDRSAM 879331 84510500
>
> address number flags uniqid begin size used %used
> c3d4978 1 U------ 1 100233 750 750 100.00
> c3d4994 2 U------ 2 100521 750 750 100.00
> c3d49b0 3 U------ 3 10080f 750 750 100.00
> c3d49cc 4 U------ 4 100afd 750 750 100.00
> c3d49e8 5 U------ 5 100deb 750 223 29.73
> c3d4a04 6 U------ 6 1010d9 750 750 100.00
> c3d4a20 7 U------ 7 800035 10000 10000 100.00
> c3d4a3c 8 U------ 8 802745 10000 10000 100.00
> c3d4a58 9 U------ 9 804e55 10000 10000 100.00
> c3d4a74 10 U------ 10 807565 10000 10000 100.00
> c3d4a90 11 U------ 11 809c75 10000 10000 100.00
> c3d4aac 12 U------ 12 80c385 10000 10000 100.00
> c3d4ac8 13 U------ 13 80ea95 10000 10000 100.00
> c3d4ae4 14 U------ 14 8111a5 10000 10000 100.00
> c3d4b00 15 U------ 15 8138b5 10000 10000 100.00
> c3d4b1c 16 U------ 16 815fc5 10000 10000 100.00
> c3d4b38 17 U------ 17 8186d5 10000 10000 100.00
> c3d4b54 18 U------ 18 81ade5 10000 10000 100.00
> c3d4b70 19 U-----L 19 81d4f5 10000 10000 100.00
> c3d4b8c 20 U------ 20 81fc05 10000 10000 100.00
> c3d4ba8 21 U---C-- 21 822315 10000 1792 17.92
> c3d4bc4 22 F------ 0 824a25 10000 0 0.00
> c3d4be0 23 F------ 0 827135 10000 0 0.00
> c3d4bfc 24 F------ 0 829845 10000 0 0.00
> c3d4c18 25 F------ 0 82bf55 10000 0 0.00
> c3d4c34 26 F------ 0 82e665 10000 0 0.00
>
> ------------------------------------------------------------
> onstat -g iov
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> AIO I/O vps:
> class/vp s io/s totalops dskread dskwrite dskcopy wakeups io/wup errors
> msc 0 i 0.0 6 0 0 0 7 0.9 0
> aio 0 s 18.9 60332 10763 49541 0 838 72.0 0
> pio 0 i 0.3 996 0 996 0 997 1.0 0
> lio 0 i 0.4 1324 0 1324 0 1325 1.0 0
>
> ------------------------------------------------------------
> onstat -g iof
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> AIO global files:
> gfd pathname totalops dskread dskwrite io/s
> 3 rootdbs 105 71 34 0.0
> 4 datadbs_ch01 56770 8280 48490 17.7
> 5 blobdbs 397 397 0 0.1
> 6 senddbs 3 3 0 0.0
> 7 receivedbs 3 3 0 0.0
> 8 physlogdbs 999 3 996 0.3
> 9 tempdbs 1824 808 1016 0.6
> 10 logsdbs 1327 3 1324 0.4
> 11 blobdbs_1p_ch01 198 198 0 0.1
> 12 blobdbs_1p_ch02 989 989 0 0.3
>
> ------------------------------------------------------------
> onstat -g glo
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> MT global info:
> sessions threads vps lngspins
> 1 22 9 0
>
> sched calls thread switches yield 0 yield n yield forever
> total: 12426045 51894 12374075 10590 17003
> per sec: 12 0 12 0 0
>
> Virtual processor summary:
> class vps usercpu syscpu total
> cpu 2 11.78 0.50 12.28
> aio 1 0.20 4.09 4.29
> tli 1 0.00 0.05 0.05
> shm 1 0.00 0.00 0.00
> lio 1 0.00 0.79 0.79
> pio 1 0.01 0.55 0.56
> adm 1 0.00 0.00 0.00
> msc 1 0.00 0.00 0.00
> total 9 11.99 5.98 17.97
>
> Individual virtual processors:
> vp pid class usercpu syscpu total
> 1 16355 cpu 4.47 0.44 4.91
> 2 16356 adm 0.00 0.00 0.00
> 3 16357 cpu 7.31 0.06 7.37
> 4 16358 lio 0.00 0.79 0.79
> 5 16359 pio 0.01 0.55 0.56
> 6 16360 aio 0.20 4.09 4.29
> 7 16361 msc 0.00 0.00 0.00
> 8 16362 tli 0.00 0.05 0.05
> 9 16363 shm 0.00 0.00 0.00
>
> tot 11.99 5.98 17.97
>
> ------------------------------------------------------------
> onstat -g ath
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Threads:
> tid tcb rstcb prty status vp-class name
> 2 1a007658 0 2 sleeping forever 4lio lio vp 0
> 3 1a007878 0 2 sleeping forever 5pio pio vp 0
> 4 1a007a98 0 2 running 6aio aio vp 0
> 5 1a007cb8 0 2 sleeping forever 7msc msc vp 0
> 6 1a007ed8 1a050018 4 sleeping secs: 1 1cpu main_loop()
> 7 1a00e220 0 2 running 8tli tlitcppoll
> 8 1a00e440 0 2 running 9shm sm_poll
> 9 1a00e688 0 3 sleeping forever 1cpu tlitcplst
> 10 1a00e8a8 0 3 sleeping forever 1cpu sm_listen
> 11 1a00eb88 0 2 sleeping secs: 2 1cpu sm_discon
> 12 1a00eda8 1a050514 2 sleeping forever 1cpu flush_sub(0)
> 13 1a00efc8 1a050a10 2 sleeping forever 1cpu flush_sub(1)
> 14 1a00f1e8 1a050f0c 2 sleeping forever 3cpu flush_sub(2)
> 15 1a00f408 1a051408 2 sleeping forever 1cpu flush_sub(3)
> 16 1a00f628 1a051904 2 sleeping forever 1cpu flush_sub(4)
> 17 1a00f848 1a051e00 2 sleeping forever 1cpu flush_sub(5)
> 18 1a00fa68 1a0522fc 2 sleeping forever 1cpu flush_sub(6)
> 19 1a00fc88 1a0527f8 2 sleeping forever 3cpu flush_sub(7)
> 20 1a00fea8 1a052cf4 3 sleeping forever 1cpu aslogflush
> 21 1a02dad8 1a0531f0 2 sleeping secs: 45 3cpu btclean
> 37 1a524358 1a0540e4 4 sleeping secs: 1 1cpu onmode_mon
> 40 1a53e3c0 1a0536ec 2 cond wait cp 1cpu sqlexec
>
> ------------------------------------------------------------
> onstat -b
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Buffers
> address userthread flgs pagenum memaddr nslots pgflgs xflgs owner waitlist
> c78cdf0 0 847 204ec4 eb09000 8 50 10 0 0
> c84fd78 0 847 204e9f fcc2800 0 0 10 0 0
> c8503a8 0 847 204ea9 fccb800 0 0 10 0 0
> c850a88 0 847 204ec6 fcd5800 0 0 10 0 0
> c850e50 0 847 204ee6 fcdb000 0 0 10 0 0
> c851168 0 847 204edc fcdf800 0 0 10 0 0
> c851218 0 847 204ed3 fce0800 0 0 10 0 0
> c851320 0 847 204ed9 fce2000 0 0 10 0 0
> c851480 0 847 204ef4 fce4000 0 0 10 0 0
> c851b08 0 847 204efa fced800 0 0 10 0 0
> c8521e8 0 847 204f16 fcf7800 0 0 10 0 0
> c8522f0 0 847 204f20 fcf9000 0 0 10 0 0
> c852450 0 847 204f37 fcfb000 0 0 10 0 0
> c8528c8 0 847 204f2a fd01800 0 0 10 0 0
> c852a28 0 847 204f32 fd03800 0 0 10 0 0
> c852b30 0 847 204f39 fd05000 0 0 10 0 0
> 94 modified, 0 resident, 100000 total, 131072 hash buckets, 2048 buffer
> size
>
> ------------------------------------------------------------
> onstat -u
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Userthreads
> address flags sessid user tty wait tout locks nreads nwrites
> 1a050018 ---P--D 1 informix - 0 0 0 32 119
> 1a050514 ---P--F 0 informix - 0 0 0 0 13547
> 1a050a10 ---P--F 0 informix - 0 0 0 0 1802
> 1a050f0c ---P--F 0 informix - 0 0 0 0 8286
> 1a051408 ---P--F 0 informix - 0 0 0 0 7524
> 1a051904 ---P--F 0 informix - 0 0 0 0 9791
> 1a051e00 ---P--F 0 informix - 0 0 0 0 1881
> 1a0522fc ---P--F 0 informix - 0 0 0 0 9411
> 1a0527f8 ---P--F 0 informix - 0 0 0 0 5272
> 1a052cf4 ---P--- 9 informix - 0 0 0 0 0
> 1a0531f0 ---P--B 10 informix - 0 0 0 0 2557
> 1a0536ec C-BP--- 15 informix 6 a019580 0 2 20886 72164
> 1a0540e4 ---P--D 13 informix - 0 0 0 1584 0
> 13 active, 128 total, 25 maximum concurrent
>
> ------------------------------------------------------------
> onstat -g stk all
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Stack for thread: 40 sqlexec
> base: 0x1a6f4000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a6fd1d8
> state: cond wait
>
> vp: 1
>
> 0x00431c1c (oninit)mt_fast_wait(0x6d0210, 0xa019574, 0xa019580, 0x1a0536ec,
> 0xa00a800, 0x800)
> 0x0027b410 (oninit)critwait(0x6c1840, 0x6d3dc8, 0x4000000, 0xedb4,
> 0xfffb51f3,
> 0x1a0536ec)
> 0x002c2d80 (oninit)btadditem(0x1a6fd524, 0x1a6fd634, 0x0, 0xef43aac,
> 0x1a6fd540, 0x1a6fd650)
> 0x003f4aec (oninit)wrtrecord(0x6d3db8, 0x6c13f0, 0x1000, 0x6c16ac, 0x7,
> 0x6c1628)
> 0x003f43a8 (oninit)rswrite (0x3, 0x6c16ac, 0x1a748230, 0x1a6a73e8,
> 0x1a6a73ec,
> 0x2)
> 0x004bcc34 (oninit)fmwrite (0x3, 0x1a748230, 0x1a6a73ec, 0x1, 0x0, 0x0)
> 0x000b48b8 (oninit)aud_sqiswrite(0x3, 0x6e, 0x1a748230, 0x0, 0x1a6dcc68,
> 0x16)
> 0x001267d0 (oninit)chkrowcons(0x6d3dd0, 0x0, 0x6c1398, 0x1a7458e0, 0x1,
> 0x0)
> 0x0010b66c (oninit)addone (0x6c1000, 0x8, 0x1a6d4030, 0x0, 0x1a748230,
> 0x1a701ac0)
> 0x0010af08 (oninit)insone_next(0x6c1398, 0x2000, 0x1a701960, 0x1a748230,
> 0x1a7484c0, 0x1094)
> 0x001294e8 (oninit)doinsert(0x1a701960, 0x6c1398, 0x1a7484c0, 0x8000000,
> 0x1a748420, 0x0)
> 0x0019936c (oninit)sq_putinsert(0x6c9fcc, 0x6d3dd0, 0x0, 0x1094, 0x142,
> 0x0)
> 0x001ab168 (oninit)sqmain (0x11, 0x6d3c00, 0x612400, 0x6c1398, 0x6c9fcc,
> 0x1670003)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 37 onmode_mon
> base: 0x1a6c6000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a6cf988
> state: sleeping
>
> vp: 1
>
> 0x000a1828 (oninit)onmode_monitor(0x6d3dc8, 0xa019684, 0xa00a824,
> 0x1a524358,
> 0xa0115a4, 0x6c07b4)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x1a524358, 0x1a006960, 0x1,
> 0x1a006920)
> 0x0043995c (oninit)mt_aio_wait(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 21 btclean
> base: 0x1a698000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a6a1ae0
> state: sleeping
>
> vp: 3
>
> 0x003a9d28 (oninit)btclean_loop(0xf000, 0xf248, 0xf244, 0x6d3dc8, 0x6c1840,
> 0xf24c)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 20 aslogflush
> base: 0x1a68a000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a693a08
> state: sleeping
>
> vp: 1
>
> 0x00398d30 (oninit)wait_4_Qbuf(0x6c16ac, 0x1a6840c0, 0xffffffff,
> 0x1a684090,
> 0x0, 0x0)
> 0x00398448 (oninit)get_full_Qbuf(0x1a684090, 0xffffffff, 0x0, 0x1a693b98,
> 0x1a693b9c, 0x1a684090)
> 0x003ee050 (oninit)aslogflush(0x0, 0x134a4, 0x134a0, 0xa00a800, 0x1,
> 0x1670003)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 19 flush_sub(7)
> base: 0x1a678000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a681ad0
> state: sleeping
>
> vp: 3
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0xa, 0x2000, 0x1a00fc88, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 18 flush_sub(6)
> base: 0x1a66c000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a675ad0
> state: sleeping
>
> vp: 1
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0xa, 0x2000, 0x1a00fa68, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 17 flush_sub(5)
> base: 0x1a65e000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a667ad0
> state: sleeping
>
> vp: 1
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0xa, 0x2000, 0x1a00f848, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 16 flush_sub(4)
> base: 0x1a650000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a659ad0
> state: sleeping
>
> vp: 1
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0xa, 0x2000, 0x1a00f628, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 15 flush_sub(3)
> base: 0x1a642000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a64bad0
> state: sleeping
>
> vp: 1
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0xa, 0x2000, 0x1a00f408, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 14 flush_sub(2)
> base: 0x1a632000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a63bad0
> state: sleeping
>
> vp: 3
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0xa, 0x28, 0x1a00f1e8, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x1a00f1e8, 0x1a006960, 0x0,
> 0x1a006920)
> 0x002f857c (oninit)flush_sub(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 13 flush_sub(1)
> base: 0x1a626000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a62fad0
> state: sleeping
>
> vp: 1
>
> 0x002f85cc (oninit)flush_sub(0x6c1800, 0x0, 0x28, 0x1a00f1e8, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x1a00efc8, 0x1a006960, 0x0,
> 0x1a006920)
> 0x002f857c (oninit)flush_sub(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 12 flush_sub(0)
> base: 0x1a61c000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a6259e8
> state: sleeping
>
> vp: 1
>
> 0x0043995c (oninit)mt_aio_wait(0x1a4ae018, 0x6d0080, 0x3, 0x6d0210,
> 0x1a4ae018, 0x4f39)
> 0x002f9244 (oninit)aclean_chunk(0x6c1840, 0x6d3dc8, 0x9ca0, 0x1381, 0x0,
> 0x0)
> 0x002f86f0 (oninit)flush_sub(0x6c1800, 0x0, 0x2000, 0x1a00eda8, 0x676d58,
> 0x6d3dc8)
> 0x00432388 (oninit)startup (0x6d0210, 0x0, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x00427f78 (oninit)idle_processor(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 11 sm_discon
> base: 0x1a610000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a619ae0
> state: sleeping
>
> vp: 1
>
> 0x0047d7f8 (oninit)net_sm_discon_thread(0x3efef4, 0x65d42c, 0x693c00,
> 0x612540, 0x1a522028, 0x12345678)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x0046e23c (oninit)net_startup_step(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 10 sm_listen
> base: 0x1a5f2000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a5fb048
> state: sleeping
>
> vp: 1
>
> 0x0047a194 (oninit)accpshm (0x1a5ede58, 0x65d42c, 0x1a5fb268, 0x1a5e44c0,
> 0x1a5e44b8, 0x1)
> 0x004a7624 (oninit)tlAccept(0x1a5e44a8, 0x1a5ede48, 0x1a5fb268, 0x1a5e35c8,
> 0x1a5ed524, 0x1a002018)
> 0x004a3fc4 (oninit)slSQIind(0x0, 0x1a5e4114, 0x1a5fb284, 0x1a5fb268, 0x0,
> 0x1a5e40dc)
> 0x0049e4bc (oninit)pfConInd(0x0, 0x1a5e4068, 0x1a5fb690, 0x1a5e36ac,
> 0x1a5ed524, 0x698380)
> 0x004966a4 (oninit)cmConInd(0x1a5e3640, 0x1a5e35c8, 0x1a5fb690, 0x1a5ed524,
> 0x1a5ed524, 0x78)
> 0x004940c8 (oninit)ascInd (0x1, 0x1a5fb690, 0x1a5e31f4, 0x1a5fb68c,
> 0x1a5ed2b0, 0x1a5ed524)
> 0x00461e10 (oninit)ASF_Call(0x40, 0x1a5fb68c, 0x1a5ed2b0, 0x4, 0x1a5fb68c,
> 0xffffffff)
> 0x0046a6fc (oninit)sql_listener(0x1a5ecf80, 0x612400, 0x1a5c58c0,
> 0x1a5220d8,
> 0x1a522028, 0x691300)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x0046e23c (oninit)net_startup_step(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 9 tlitcplst
> base: 0x1a5d8000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a5e0dd0
> state: sleeping
>
> vp: 1
>
> 0x00477258 (oninit)net_wait_for_io(0x1a5c5d40, 0x1, 0x1a5c5d40, 0x0,
> 0x3efc00,
> 0x1a5c5d40)
> 0x0048d1dc (oninit)accptli (0x1a5ec918, 0x1a605118, 0x65bebc, 0x1a5e2304,
> 0x0,
> 0x2d7c)
> 0x004a7624 (oninit)tlAccept(0x1a5e9770, 0x1a5ec908, 0x1a5e1268, 0x1a605118,
> 0x1a5e2304, 0x1a002018)
> 0x004a3fc4 (oninit)slSQIind(0x0, 0x1a5ec084, 0x1a5e1284, 0x1a5e1268, 0x0,
> 0x1a5ec04c)
> 0x0049e4bc (oninit)pfConInd(0x0, 0x1a5ebfd8, 0x1a5e1690, 0x1a6051fc,
> 0x1a5e2304, 0x698380)
> 0x004966a4 (oninit)cmConInd(0x1a605190, 0x1a605118, 0x1a5e1690, 0x1a5e2304,
> 0x1a5e2304, 0x78)
> 0x004940c8 (oninit)ascInd (0x1, 0x1a5e1690, 0x1a6a3b3c, 0x1a5e168c,
> 0x1a5e2090, 0x1a5e2304)
> 0x00461e10 (oninit)ASF_Call(0x40, 0x1a5e168c, 0x1a5e2090, 0x4, 0x1a5e168c,
> 0xffffffff)
> 0x0046a6fc (oninit)sql_listener(0x1a5c5d40, 0x1000000, 0x1a5c58c0, 0xc,
> 0x1a522028, 0x691300)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x0046e23c (oninit)net_startup_step(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 8 sm_poll
> base: 0x1a5c8000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a5d1ac8
> state: running
>
> vp: 9
>
> 0x82001a05 (***nosymtab***)0x82001a05
> 0x00000000 ***unknown***
>
> Stack for thread: 7 tlitcppoll
> base: 0x1a5b6000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a5bfa18
> state: running
>
> vp: 8
>
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 6 main_loop()
> base: 0x1a5a6000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a5af198
> state: sleeping
>
> vp: 1
>
> 0x002f7574 (oninit)dskflush(0x0, 0x612540, 0x6d3dc8, 0x1a72c648,
> 0x1a72c030,
> 0xa01902c)
> 0x002b2170 (oninit)checkpoint(0xf000, 0x1, 0xa01902c, 0xa01701c, 0xa00a800,
> 0x49bf64e0)
> 0x0009f3c0 (oninit)main_loop(0x1, 0xbd7, 0x6c07b4, 0x6d3e00, 0xec00,
> 0xa00a800)
> 0x00432388 (oninit)startup (0x6d0210, 0x6, 0x0, 0x1a006960, 0x0,
> 0x1a006920)
> 0x0009d014 (oninit)main (0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 5 msc vp 0
> base: 0x1a57a000
> len: 138240
>
> pc: 0x00427a8c
> tos: 0x1a59bae0
> state: sleeping
>
> vp: 7
>
> 0x0043bc50 (oninit)iowork (0x6d3c00, 0x49bf5f84, 0x1a4b3e38, 0x1a5220d8,
> 0x1a4b5e38, 0x2)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x0, 0x1a006960, 0x2,
> 0x1a006920)
> 0x0043a68c (oninit)iothreadOK(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 4 aio vp 0
> base: 0x1a562000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a56bae0
> state: running
>
> vp: 6
>
> 0x1a558800 (***nosymtab***)0x1a558800
> 0x00000000 ***unknown***
>
> Stack for thread: 3 pio vp 0
> base: 0x1a54a000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a553ae0
> state: sleeping
>
> vp: 5
>
> 0x0043bc50 (oninit)iowork (0x0, 0x0, 0x1a4a7e38, 0x1a5220d8, 0x1a4a9e38,
> 0x2)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x0, 0x1a006960, 0x2,
> 0x1a006920)
> 0x0043a68c (oninit)iothreadOK(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> Stack for thread: 2 lio vp 0
> base: 0x1a532000
> len: 39936
>
> pc: 0x00427a8c
> tos: 0x1a53bae0
> state: sleeping
>
> vp: 4
>
> 0x0043bc50 (oninit)iowork (0x0, 0x0, 0x1a4a5ca0, 0x1a5220d8, 0x1a4a5e18,
> 0x2)
> 0x00432388 (oninit)startup (0x6d0210, 0x7, 0x0, 0x1a006960, 0x2,
> 0x1a006920)
> 0x0043a68c (oninit)iothreadOK(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
> 0x00000000 (***nosymtab***)0x0
>
> ------------------------------------------------------------
> onstat -g wmx
>
> Informix Dynamic Server Version 7.31.UD1 -- On-Line (CKPT REQ) -- Up
> 00:53:20
> -- 412752 Kbytes
> Blocked:CKPT
>
> Mutexes with waiters:
> mid addr name holder lkcnt waiter waittime
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--
Art S. Kagel
Oninit (www.oninit.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Oninit, the IIUG, nor any other organization
with which I am associated either explicitly or implicitly. Neither do
those opinions reflect those of other individuals affiliated with any entity
with which I am affiliated nor those of the entities themselves.
--0016364ef07c8617a20465515f50
