IDS Forum

Re: Threads waiting on buffers

Posted By: Madison Pruet
Date: Monday, 3 November 2003, at 3:05 p.m.

In Response To: Threads waiting on buffers (John Bejarano )

What is LRU Min/Max set to?




"John Bejarano "
<jbejarano@sbcglo To: ids@iiug.org
bal.net> cc:
Sent by: Subject: Threads waiting on buffers [2118]
forum.subscriber@
iiug.org


11/03/2003 10:15
AM






Ok, so Informix tech support and I haven't been able
to figure this one out. I'm hoping someone else is
familiar with this problem and knows how to fix it.

Every now and then, often while we are experiencing
peak load, but not necessarily, our Informix engine
gets into a state where a huge number of users all of
a sudden begin waiting on a single buffer in the
buffer cache. Turns out this buffer page contains a
BTREE page for an index on the systables table in the
system catalog for our main database. Specifically,
the last page (alphabetically) of the index 'tabname'
on systables which contains keys on (tabname, owner).

When running onstat -b or onstat -X, the waiters on
this buffer seem to all appear, disappear, and
reappear, but when you run onstat -u, a HUGE number of
threads always have the 'B' flag for waiting on a
buffer. Needless to say, the entire database's
performance goes in the tank when this happens. One
time, the stack traces seemed to indicate that a ton
of threads were attempting to build a referential
constraint, but no such SQL commands were issued by
any thread. Plus, other times, the stack traces have
not necessarily shown this, but this may be related to
the appear/disappear/reappear thing we see on these
waiters in onstat -b and -X.

Our dbclients are configured with their own connection
pool managers. Any time they need additional
connections to accomplish something on the database,
they establish new connections. Thus, while we
usually have between 1000 and 1500 connections total
from all of the dbclient machines, the number of
connections will usually shoot up over 3500 when this
problem occurs (since the original ones do not respond
quickly any longer). Virtual memory usage goes up
concomitantly as well.

Most of the time, the database will not pull out of
this state. Usually, we need to bounce the engine in
order to get it out of this. The one time it did come
out, we were trying to have Informix dial in to look
at it while it was in progress, and it cleared up
before they could connect. There was no appearant
reason why it cleared up, but this was the ONE time
that this occurred when we were not experiencing peak
load. When we bounce the engine during one of these
episodes, as there is so much activity, the logical
recovery can take up to 10 minutes or so to complete.
While it does seem to occur most often with peak load,
we have experienced other times when load was even
higher, and we had no problems.

The engine is IDS 7.31.UD1XF running on Solaris 9.
The box is an 8-way Sunfire 4800. I have 800,000
buffers configured. (I'd raised it in a couple of
steps from 300,000 hoping that that would relieve this
issue.) We do have in place the LRUAGE=1 fix to
prevent BTREE pages from taking up all the buffers.
We have 100 LRU queues and MAX and MIN set to 2 and 1,
respectively.

We are about to go into our peak season, and having an
unstable database is really freaking out my
management, not to mention me. Any help or insight
that anyone can offer would be GREATLY appreciated.
I'm losing sleep on this one. Thanks very much.

--John Bejarano
DBA, Shutterfly, Inc.

Messages In This Thread

• Threads waiting on buffers
  John Bejarano -- Monday, 3 November 2003, at 11:15 a.m.
  • Re: Threads waiting on buffers
    Madison Pruet -- Monday, 3 November 2003, at 3:05 p.m.
  • Re: Threads waiting on buffers
    John Bejarano -- Monday, 3 November 2003, at 4:43 p.m.
    • Re: Threads waiting on buffers
      Danny Wright -- Tuesday, 4 November 2003, at 11:41 a.m.
      • RE: Threads waiting on buffers
        Darrell Murphy -- Tuesday, 4 November 2003, at 1:33 p.m.
      • RE: Threads waiting on buffers
        Javier Zayas -- Tuesday, 4 November 2003, at 1:52 p.m.
      • Re: Threads waiting on buffers
        ART KAGEL, BLOOMBERG/ 65E 55TH -- Tuesday, 4 November 2003, at 2:11 p.m.
        • RE: Threads waiting on buffers
          Meza, Silvana -- Wednesday, 5 November 2003, at 3:47 p.m.
      • Re: Threads waiting on buffers
        Martin Fuerderer -- Wednesday, 5 November 2003, at 3:50 a.m.
      • Re: Threads waiting on buffers
        Obnoxio The Clown -- Friday, 7 November 2003, at 4:06 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.