IDS Forum

Re: How to Monitor Open Connection Tiime???

Posted By: Cesar Inacio Martins
Date: Wednesday, 1 December 2010, at 1:01 p.m.

In Response To: Re: How to Monitor Open Connection Tiime??? (Fernando Nunes)

Hi Fernando!
Thanks for your message!
Sorry take to long to answer, I don't know why the last 2 days of
messages incoming from IIUG has arrived just now.

So, about the gethostbyaddr() , I'm not sure about that because isn't
what we see on the strace .
The request of DNS reverse lookup, appear to be executed "manually" by
Informix or the strace just traced the gethostbyaddr() too.
Check the output marked with "##" by me.
where: 172.18.0.57 and 172.18.0.119 are the OLD DNS when the server was
started. When we run this strace, the resolv.conf already have new
values (for at least 2 days):

11:24:28 munmap(0x2abeb6d75000, 4096) = 0
11:24:28 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
##11:24:28 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.18.0.57")}, 28) = 0
11:24:28 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
11:24:28 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
11:24:28 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
##11:24:28 sendto(3, "\242;\1\0\0\1\0\0\0\0\0\0\003185\00248\00222\003172\7in-ad"..., 44, MSG_NOSIGNAL, NULL, 0) = 44
##11:24:28 poll([{fd=3, events=POLLIN}], 1, 5000) = 0 (Timeout)
11:24:33 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
11:24:33 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.18.0.119")}, 28) = 0
11:24:33 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)
... they continue trying to third DNS server...
(check the timestamp,.. 5 seconds of timeout delay)

Before you ask, the Linux Admin already try change the timeout (option
into resolv.conf) and don't have effect....

Sorry , yes is a 11.50 FC7 GE over Linux Red Hat 5.5
This option to create a group is nice, but I not sure if is viable for
this environment because have a lot of applications spread on the
company (.net/windows, java/web, C/Linux) what we will need care about
to change the SQLHOSTS...

The nsswitch.conf is ok, it isn't modified from they default values...
and the problem is with connections what become from out of our network
(what don't exists into /etc/hosts) and create consequences to local
connections (what exists in hosts) when the instance get trouble to try
resolve their hostnames (not local clients)

And the problem what we detected is over MSC, because is it what freeze...
The most weird thing is, when the MSC "freeze" (stuck running in active
threads: onstat -g act) they stack dump (onstat -g stk #threadid) don't
change anything, still showing in yield process...and just change the
status from sleep to running....

About IPv6 I'm answering into the other message...

On 11/28/2010 09:12 PM, Fernando Nunes wrote:
> Hello Cesar.
> Please see the comments below.
>
> On Sun, Nov 28, 2010 at 3:20 AM, Cesar Inacio Martins<
> cesar_inacio_martins@yahoo.com.br> wrote:
>
>> Hi ,
>>
>> I'm working with Miguel on this situation .
>> After some debugs with strace, tcpdump, application log we could confirm
>> the problem is the DNS reverse.
>> But is more of one problem together.
>> Let me try explain what happen here:
>>
>> 1) the database was started and at this moment the DNS configuration on
>> Linux (/etc/resolv.conf) isn't correct, they appoint to old DNS.
>>
> This is an "obscure"/grey area. AFAIK Informix does not cache anything. It
> only calls gethostbyaddr(). The behavior of this function may differ from OS
> to OS.
>
>> 2) The Linux admin, change the resolv.conf and solve the problem with DNS.
>> 3) the problem into Informix persist and we cannot bounce the instance
>> neither restart the listeners.
>>
> You don't mention the version (maybe 11.50.xc7?). If your version allows for
> dynamic listener startup you could launch some other(s). Apparently you want
> to launch other listeners for the same port. But you dont' have to do
> that.... You can create several (not many) listeners on the server and
> configure your clients to use a group that includes them. This will spread
> the connections for several ports. You could also establish some more
> agressive values for INFORMIXCONNTIME and INFORMIXCONRETRY. So you could
> workaround this by launching further listeners and appropriate configuration
> on the client side...
>
>> 4) with strace / tcpdump we detected the Informix still trying do the
>> DNS reverse to old DNS (configured when the database was started)
>> 5) The problem of this long time to open connection occur with
>> connections of application what incoming from a differ network (remote
>> network, behind of a firewall).
>> 6) All other connections what occur at same time, suffer a delay ... the
>> application what we are monitoring is running on the same network and
>> the hostname is in /etc/hosts.
>>
> If you have the entries in the /etc/hosts it should not be talking to the
> DNS... Unless your settings in /etc/nsswitch.conf are not right.
> Please check that also. I'm not sure if a modification on this would require
> a bounce in the engine.
> If you have doubts please post your /etc/nsswitch.conf
>
>> 7) The instance was configured with 4 MSC VPs since they up. But only
>> one VP works and when they "freeze" into the DNS reverse, the others MSC
>> VPs just still doing nothing (they should threat the new connections, I
>> suppose)
>>
> I suppose you're becoming "hanged" on the listeners and not on the MSC VPs.
>
>> So, here we have 2 problems:
>> - Informix cache the IP of DNS servers and don't update it...
>>
> Already replied... I don't think it's IDS, for the matter is pretty
> irrelevant....
>
>> - The MSC VP don't parallelize the requests..
>> (we already open a PMR for this)
>>
> Also replied... I think the problem lies in the listeners, not the MSC
>
>> Reading the manual Admin Guide, I found this:
>> "When a poll thread receives a connection request from a client, it
>> passes the
>> request to the listen thread for the port. The listen thread
>> authenticates the user,
>> establishes the connection to the database server, and starts an sqlexec
>> thread, the
>> session thread that performs the primary processing for the client."
>>
>> So, based on this quote, the listen thread have the "responsibility" the
>> call the MSC VP to do the rest of the job and they "freeze" at the open
>> connection, I deduce..
>> If I have *more listen threads*, maybe I can soften my problem, where
>> the others listeners will work in parallel and *maybe* work with the
>> others MSC VPs while the first MSC is freeze over the DNS reverse.
>>
>> That's what I think should happen. So, the idea above (several listeners
> "grouped" on the client side) could help you.
>
>> Then I found a configuration on the manual what I don't have notice
>> before. Configuring more listeners threads for the same service!!!!
>> DBSERVERNAME ifxtest
>> change to
>> DBSERVERNAME ifxtest-3 # for 3 listener threads (soctcplst) .
>>
>> I tested on v11.70 and works (they create 3 soctcplst)... and appear to
>> be available on version 11.50 too . :)
>>
> I'm ashamed, but I haven't noticed this...
> But now that you mention, I'd say it's there since v10. One of the gurus may
> be able to confirm this.
>
>> Back to our situation here, how we can not bounce the instance (to
>> update the dbservername), this isn't an available solution at this
>> moment. But is very interesting for new configurations!
>>
>> My questions:
>> 1) Anyone know if more listeners threads will work fine with more MSC
>> VPs? (parallelizing)
>>
> I suppose so, but I can't give you a definitive answer.
>
>> 2) Know if is stable for production environment? (v11.50 xC7) ?
>>
>>
> If you mean several threads for the same service, as I say above I haven't
> used it. But if it's there since V10 (needs confirmation) than It should be
> stable.
>
> But again, I think you could solve this without bouncing the engine. Start
> more listeners and use a group in the client side that include some of them.
>
> Regards.
>

Messages In This Thread

• How to Monitor Open Connection Tiime???
  Miguel Carbone -- Wednesday, 24 November 2010, at 8:47 a.m.
  • Re: How to Monitor Open Connection Tiime???
    Art Kagel -- Wednesday, 24 November 2010, at 8:52 a.m.
    • Re: How to Monitor Open Connection Tiime???
      Miguel Carbone -- Wednesday, 24 November 2010, at 11:27 a.m.
  • Re: How to Monitor Open Connection Tiime???
    Fernando Nunes -- Wednesday, 24 November 2010, at 10:30 a.m.
    • RE: How to Monitor Open Connection Tiime???
      Paul Watson -- Wednesday, 24 November 2010, at 11:14 a.m.
    • Res: Re: How to Monitor Open Connection Tiime???
      Miguel Carbone -- Wednesday, 24 November 2010, at 2:03 p.m.
    • Re: How to Monitor Open Connection Tiime???
      Miguel Carbone -- Wednesday, 24 November 2010, at 2:13 p.m.
      • Re: How to Monitor Open Connection Tiime???
        John Miller iii -- Wednesday, 24 November 2010, at 4:13 p.m.
    • Re: How to Monitor Open Connection Tiime???
      Fernando Nunes -- Wednesday, 24 November 2010, at 4:39 p.m.
      • Re: How to Monitor Open Connection Tiime???
        Cesar Inacio Martins -- Saturday, 27 November 2010, at 10:20 p.m.
        • Re: How to Monitor Open Connection Tiime???
          Fernando Nunes -- Sunday, 28 November 2010, at 6:12 p.m.
          • Re: How to Monitor Open Connection Tiime???
            Cesar Inacio Martins -- Wednesday, 1 December 2010, at 1:01 p.m.
            • Re: How to Monitor Open Connection Tiime???
              Fernando Nunes -- Wednesday, 1 December 2010, at 8:07 p.m.
              • Re: How to Monitor Open Connection Tiime???
                Cesar Inacio Martins -- Friday, 3 December 2010, at 7:44 a.m.
                • Re: How to Monitor Open Connection Tiime???
                  Fernando Nunes -- Friday, 3 December 2010, at 10:47 a.m.
                  • Re: How to Monitor Open Connection Tiime???
                    Cesar Inacio Martins -- Friday, 3 December 2010, at 12:33 p.m.
                    • Re: How to Monitor Open Connection Tiime???
                      Fernando Nunes -- Friday, 3 December 2010, at 12:48 p.m.
                      • Re: How to Monitor Open Connection Tiime???
                        Cesar Inacio Martins -- Monday, 6 December 2010, at 8:22 a.m.
                        • Re: How to Monitor Open Connection Tiime???
                          Fernando Nunes -- Monday, 6 December 2010, at 1:40 p.m.
                  • Re: How to Monitor Open Connection Tiime???
                    Cesar Inacio Martins -- Friday, 3 December 2010, at 12:44 p.m.
        • Re: How to Monitor Open Connection Tiime???
          JIM XIAO -- Tuesday, 30 November 2010, at 1:23 p.m.
          • Re: How to Monitor Open Connection Tiime???
            Cesar Inacio Martins -- Wednesday, 1 December 2010, at 1:31 p.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.