IDS Forum

Re: Intermittent DNS failures

Posted By: Fernando Nunes
Date: Tuesday, 23 December 2014, at 6:04 p.m.

In Response To: Intermittent DNS failures (Kennedy, Randy)

It's a well known problem with Informix... or as you correctly put it, with
DNS that impacts Informix.
Both suggestion I've seen are interesting (ncd daemon and the ns_cache
parameter). Unfortunately you cannot consider the later one (you should be
aware that 11.1 is out of support)

There are some points I'd like to add. First and more important, and
eventually I missed some information or detail on this thread:

1- Do you need the reverse DNS to happen? Have you created trusts based on
client names? IF not, they you could configure the system to just go to the
local files and be immune to any DNS issue. In other words, if your
connections are based on USER/PASS and not trusted, then this would be
possible

2- If that's not the case, and you have to rely on your DNS, than the DNS
must be considered a top priority and be handled like any high availability
system. An Informix customer (specially on versions where NS_CACHE cannot
be used) that decides to use DNS will become dependent on it. The way we do
things make it particularly susceptible to any issues in the DNS. Let's
see....

- The reverse DNS is made by the MSC VP or VPs.
- Each call is synchronous
- The DNS configuration plans and default to very large timeouts (in some
systems I believe you cannot even ask for values below 5s)
- To add up to all this, the functions of the OS (gethostbyaddr() ) will by
design create "caches" of the DNS configurations (if DNS is used, or just
files, what DNS etc). Basically the information in files like
/etc/nsswitch.conf and /etc/resolv.conf are cached after the first call.
This means that you can change this files but the database will not change
the behavior unless you restart it. This is one of my "personal battles"
that is going nowhere...

So, what can we do to improve the situation?

- If possible don't use DNS (I have the impression that if using IPV6 this
is not an option)
- Eventually use more than 1 MSC VP. If the issue with calls to
gethostbyaddr() are occasional, this means that at any moment you may have
one "hanged" MSC VP while the others will continue to answer requests. So
eventually only some of the connection attempts will fail
- Use ncd daemon as very well suggested by another person
- Use NS_CACHE if the Informix version has it, as Stefan very well reminded
us
- If you need to change the DNS configuration frequently there is a trick
that can be used while IBM does not solve the problem. I described the
trick in this article:
http://informix-technology.blogspot.pt/2014/02/dns-changes-ok-mudancas-no-dns-ok.html
It's relatively simple to follow, but some programming skills will make it
easier to implement. It basically takes advantage of the fact that PAM
authentication functions also run on the MSC VPs... so we can create a
"fake" PAM module that will clear the MSC VP DNS cache by calling the OS
function that clears it (res_init() ). The authentication will always
return false, so this fake module should be used only in a "maintenance
listener" created only for this purpose.

Regards.

On Mon, Dec 22, 2014 at 7:06 PM, Kennedy, Randy <RKennedy@scottsdaleaz.gov>
wrote:

> While this is probably not primarily an Informix question, it is impacting
> our
> users' ability to establish a database connection and causing a great deal
> of
> 'pain' and I am looking for any help I can get.
>
> IDS: 11.1FC3
> OS: RHEL 5.11 on VMWARE
>
> Our configuration has been in place for several years and most of it
> trouble
> free. However, recently we started having issues with users logging in to
> our
> application or getting kicked-out after trying to apply a transaction. We
> had
> some network issues in the past so we redesigned the app to create the
> connection, use it, close it, as having an open connection and using it all
> session caused issues if the connection was lost during a network 'hiccup'.
> This worked well for the last couple of years, but recently ran into more
> issues. What it appears like it happening is the client is connecting to
> the
> server and while doing a reverse lookup back to the client, it is
> failing/timing out and then the connection fails and user can't login or
> gets
> booted out and has to re-launch and re-do the transaction. To mitigate the
> problem for the short term, I added all of our users to the server's host
> file, but their IP's could change and cause more problems moving forward
> with
> this approach.
>
> Couple of questions.
> Has anyone experienced this issue and resolved? If so, what areas should I
> be
> targeting as a starting point of investigation?
> Is there a way to change the configuration of Informix so it doesn't
> require
> the reverse lookup to establish the database connection?
>
> TIA,
> Randy
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>

--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...

--001a1140f88edd55e4050aea31e8

Messages In This Thread

• Intermittent DNS failures
  Kennedy, Randy -- Monday, 22 December 2014, at 2:06 p.m.
  • RE: Intermittent DNS failures
    Link, David A -- Monday, 22 December 2014, at 2:19 p.m.
  • Re: Intermittent DNS failures
    David Zinder -- Monday, 22 December 2014, at 3:31 p.m.
    • Re: Intermittent DNS failures
      Stefan Sammut -- Monday, 22 December 2014, at 4:18 p.m.
      • RE: Intermittent DNS failures
        Kennedy, Randy -- Monday, 22 December 2014, at 9:24 p.m.
  • Re: Intermittent DNS failures
    Fernando Nunes -- Tuesday, 23 December 2014, at 6:04 p.m.
    • RE: Intermittent DNS failures
      Kennedy, Randy -- Tuesday, 23 December 2014, at 6:21 p.m.
      • Re: Intermittent DNS failures
        Fernando Nunes -- Tuesday, 23 December 2014, at 6:34 p.m.
    • RE: Intermittent DNS failures
      Paul Watson -- Wednesday, 24 December 2014, at 10:48 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.