|
IDS Forum
Re: Threat : swth:sbt_subclass_lpgsrc_ol_796
Posted By: david@smooth1.co.uk Date: Wednesday, 1 November 2017, at 12:16 p.m.
In Response To: Re: Threat : swth:sbt_subclass_lpgsrc_ol_796 (Alberto Romeu Pessonio Filho)
Hi Alberto,
Do you have APAR numbers for the contention issues you faced?
Regards,
David.
> On 01 November 2017 at 13:07 Alberto Romeu Pessonio Filho <albasic@gmail.com> wrote:
>
>
> Hi Ley!
>
> Probably you're facing a kind of memeory bottleneck.
>
> I had a similar issue related to memory contention on my environment (that
> was shown through onstat -g wmx) and it lead to a defect. Also, I had many
> other memory contention issues during my IBM journey on huge customer.
>
> The following steps were used to track this issue:
>
> 1- When you have a "memory" thread showed on "onstat -g act/ onstat -g rea"
> outputs, check the mutexes info from "onstat -g lmx/onstat -g wmx"
>
> Im my case(RR_ID hash), the contention happened on routine resolution cache:
>
> [informix@valencia01 ~]$ onstat -g wmx IBM Informix Dynamic Server Version
> 12.10.FC5X2 -- On-Line (Prim) -- Up 21 days 04:48:56 -- 25811176 Kbytes
> Mutexes with waiters: mid addr name holder lkcnt waiter waittime 9950
> 6207faa8 RR_ID hash 396 117437832 0 117437835 0 117437850 0 117432874 0
> 117437843 0 117386806 0 117417336 0 117437838 0 117437847 0 117437837 0
> 117437844 0 117437851 0 117437829 0 117437836 0 117386802 0 117432841 0
> 117432878 0 117432861 0 Number of mutexes on VP 1 free list: 127929 Number
> of mutexes on VP 2 free list: 0 Number of mutexes on VP 3 free list: 0
> Number of mutexes on VP 4 free list: 0 Number of mutexes on VP 5 free list:
> 0 Number of mutexes on VP 6 free list: 0 Number of mutexes on VP 7 free
> list: 0 Number of mutexes on VP 8 free list: 0 Number of mutexes on VP 9
> free list: 127986 Number of mutexes on VP 10 free list: 127990 Number of
> mutexes on VP 11 free list: 127994 Number of mutexes on VP 12 free list:
> 127836 Number of mutexes on VP 13 free list: 127993 Number of mutexes on VP
> 14 free list: 127997 Number of mutexes on VP 15 free list: 127999 Number of
> mutexes on VP 16 free list: 127997 Number of mutexes on VP 17 free list:
> 127966 Number of mutexes on VP 18 free list: 127979 Number of mutexes on VP
> 19 free list: 127978 Number of mutexes on VP 20 free list: 127997 Number of
> mutexes on VP 21 free list: 128000 Number of mutexes on VP 22 free list:
> 127982 Number of mutexes on VP 23 free list: 0 Number of mutexes on VP 24
> free list: 0 Number of mutexes on VP 25 free list: 0 Number of mutexes on
> VP 26 free list: 0 Number of mutexes on VP 27 free list: 0 Number of
> mutexes on VP 28 free list: 0 Number of mutexes on VP 29 free list: 0
> Number of mutexes on VP 30 free list: 0 Number of mutexes on VP 31 free
> list: 0 Number of mutexes on VP 32 free list: 0
>
> In this case, we can see a "mutex id" thread locking a memory resource, and
> several threads are waiting to acquire that resource.
>
> I can't be sure if it is related to large pages... Did you export
> IFX_LARGE_PAGES when you started IDS after enable it at OS level?
>
> 1 - In order to get more information from the issue you're facing, look at
> those outputs, and share what you think that is strange. I can support you
> properly if you share it, as well.
>
> - onconfig,
> - onstat -g env
> - onstat -g spi|head -n 6;onstat -g spi| sort -r -n -k 1|head -n 50
> *- onstat -g con |head -n 50*
> *- onstat -g lmx *
> *- onstat -g wmx*
> - onstat -g vpcache
> - onstat -g glo
> - onstat -g sch
> - onstat -g scn
> - some samples of vmstat or lparstat
> - some samples of onstat -g stk from those thread ID (
> swth:sbt_subclass_lpgsrc_ol_*)
>
> Those output highlighted above gonna show useful info.
>
> 2 - Have you increased hardware threads/virtual cores/Entlited Capacity on
> your AIX, when you enabled large pages? What is the AIX version?
>
> 3 - Some useful info to collect from AIX OS:
>
> http://www.unixmantra.com/2013/09/aix-listing-cpu-memory-ram-harddisks-ip-address.html
>
> Regards,
> Alberto.
>
> 2017-10-31 22:19 GMT-02:00 LEY PATRICK <patrickley@gmail.com>:
>
> > Platform : AIX 7
> > Informix version: 11.70.FC7W3XC
> > Config : 1 primary, 1 secondary and 4 RSS
> >
> > Note : We do turn on the large page on AIX level
> >
> > I run onstat -g rea -r 1 and occasionally see the threat name
> > swth:sbt_subclass_lpgsrc_ol_796
> >
> > Questions:
> > 1) What is this thread and the purpose of this threat?
> > 2) Is this threat related to Large Page ?
> >
> > Snip of onstat -g rea -r 1
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:00 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 170cpu* memory
> >
> > [informix@obnfrmxdb01p2] /home/informix #onstat -g rea -r 1
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:04 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 21cpu* memory
> > 31110 70000500a4d17f0 7000047f8046428 1 ready 22cpu sqlexec
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:05 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:06 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 63 7000047ae1814c0 0 2 ready 9cpu* soctcplst
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 239cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:07 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 20cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:08 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 172cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:09 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:10 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 115cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:11 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 115cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:12 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 220cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:13 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:14 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 131cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:15 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 230cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:16 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2323 7000047fd7f71e8 7000047f429afe8 3 ready 9cpu smxrcv sbdardbi02_tcp
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:17 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 177cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:18 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 284cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:19 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 97163 700004d3352c630 70000506246a4a8 1 ready 16cpu sqlexec
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:20 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 143cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:21 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 182cpu* memory
> > 2357 7000047fd2c1c58 7000047f42944a8 3 ready 21cpu RSS_Send_sbdardbi01_tcp
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:22 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 173cpu* memory
> > 4114 700004883b57028 70000487dfd6c28 1 ready 36cpu sqlexec
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:23 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 8cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:24 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 21cpu* memory
> > 48019 7000055d7505220 7000050624683a8 3 ready 21cpu readahead_6
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:25 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 186cpu* memory
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:26 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 3955 700004874c08a48 7000048699f66a8 1 ready 16cpu sqlexec
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:27 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 1cpu* memory
> > 2327 7000048048b45e0 7000047f429d0e8 3 ready 1cpu RSS_Send_sbdardbi02_tcp
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:28 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 79599 700005ca76912b0 70000488420e368 2 ready 179cpu* sqlexec
> > 81457 700005c4d1e4370 700005931acd3a8 1 ready 30cpu
> > swth:sbt_subclass_lpgsrc_ol_796
> > 81489 70000582c5c9d20 7000056dd5f3428 1 ready 28cpu
> > swth:sbt_subclass_lpgsrc_ol_796
> > 81576 700005cac602028 700005cf5871768 1 ready 28cpu
> > swth:sbt_subclass_lpgsrc_ol_797
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:29 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 193cpu* memory
> > 81489 70000582c5c9d20 7000056dd5f3428 1 ready 28cpu
> > swth:sbt_subclass_lpgsrc_ol_796
> > 97162 700005a43d042b0 700004956f71328 2 ready 308cpu* sqlexec
> >
> > IBM Informix Dynamic Server Version 11.70.FC7W3XC -- On-Line (Prim) -- Up
> > 05:20:30 -- 521929792 Kbytes
> >
> > Ready threads:
> > tid tcb rstcb prty status vp-class name
> > 59 7000047ae10dbb0 0 2 ready 8cpu* soctcplst
> > 2034 7000047fbeb7790 7000047a307a4e8 3 ready 9cpu* memory
> >
> > [informix@obnfrmxdb01p2] /home/informix #uname -a
> > AIX obnfrmxdb01p2 1 7 00CCBC174C00
> >
> > All input are welcome.
> >
> >
> > ************************************************************
> > *******************
> > Forum Note: Use "Reply" to post a response in the discussion forum.
> >
> >
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
Messages In This Thread
IDS Forum is maintained by Administrator with WebBBS 5.12.
|
|