IDS Forum

Re: Mysterious Master Daemon death

Posted By: Preetinder Dhaliwal
Date: Monday, 31 May 2004, at 4:18 a.m.

In Response To: Mysterious Master Daemon death (John Bejarano )

Hi John,

If somebody is killing ur oninit process you can try some of the below -

1) If u run informix thru informix id, u can try to change and run
using root. This will increase one level of security as then anybody
killing oninit will need root id or a script with setuid bit of root to
kill the same.

2) In my last company, for audit purposes , they had a utility which
would run when u login ( as root ) and write all inputs to a file owned
by root ( and u can place this file somewhere which only few people know
abt ), there are ways to get around this too, but still can help.

hope this helps

ps: I dont have the software which logs activities of all users, ask ur
sysadmin if he/she can help

Rgds
Preetinder
John Bejarano wrote:

>Ok, so I'm ripping my hair out on this one now.
>
>For the fourth time this year, and the second time
>this month, our main Informix database running on
>version 7.31.UD7X2/Solaris 9 crashed mysteriously with
>only the lines:
>
>14:13:33 The Master Daemon Died
>14:13:34 PANIC: Attempting to bring system down
>
>There is no Assert Failure message or any other
>forboding warning above or below this message. Nor is
>there anything suspicious in /var/adm/messages. It's
>just that CPU VP #1 seems to kick the bucket all of a
>sudden.
>
>I'm aware that this is consistent with someone or
>something external to Informix sending a kill -9
>signal to CPU VP #1. I've tested that and seen it for
>myself. However, the only other applications we have
>on there are Veritas Cluster Server for
>high-availability (because my company likes it better
>than HDR), and Big Brother for monitoring purposes.
>We did experience these Master Daemon deaths a couple
>times even when VCS wasn't running. Really, neither I
>nor anyone else here can think of anything that could
>possibly be messing with Informix like this.
>
>IBM Informix technical support, of course, insists
>that this is external to Informix, and largely based
>on my experience before this started happening, I'm
>inclined to agree with them. Unfortunately, no one
>else at my company is buying that explanation.
>
>We've attempted to put a wrapper script around the
>kill command to identify who's running it, but with
>Solaris, almost all kills, even from the shell command
>line, don't actually use /bin/kill, they use a kernel
>call. We're not too fond of the idea of hacking the
>kernel to monitor that. :)
>
>We were also given the suggestion to truss CPU VP 1.
>This might at least identify that a kill has been
>issued, but this is a pretty hefty burden to place on
>the db, particularly if it's weeks before this happens
>again. We can run a cron that lops off the voluminous
>output from this every so often to keep disk space
>from filling, but just the I/O involved here could
>feed back badly onto the database.
>
>We also tried running onstat -i (interactive mode) in
>a persistent window so that if this were to occur
>again, at least we could take onstat commands for
>further analysis. Informix told me, and I tested and
>confirmed this, that even if the engine dies, the
>interactive onstat command stays attached to the
>shared memory, and can continue giving onstat outputs
>based on that memory. You can even re-start Informix,
>and the interactive onstat still looks at the memory
>from the instance before it was killed. (Actually,
>that's kind of creepy. Cool, but creepy.)
>
>Here's the interesting bit, though. This last time,
>when the Master Daemon died, I went back to the
>persistent window running the interactive onstat, and
>it didn't stay attached to shared memory. I tried to
>run one command there, and it gave me the "shared
>memory not initialized" message, and exited.
>
>This seems really suspicious to me. Why would it be
>in a test environment that I could kill Informix with
>a kill -9, and onstat -i does indeed continue to see
>the memory of the dead instance, but in this
>mysterious case in production, it doesn't? Indeed,
>this behavior makes it seem more likely that something
>bad really did happen within Informix.
>
>Informix has really become persona-non-grata around
>here, and without any af files or other log messages
>to go on, this problem has become impossible to
>troubleshoot. Has *anyone* seen this behavior before,
>or know anything about it? Any help here may save my
>job and/or our continued use of Informix. Thanks very
>much.
>
>--John Bejarano
>Informix DBA, Shutterfly
>
>
>
>
>

Messages In This Thread

• Mysterious Master Daemon death
  John Bejarano -- Friday, 28 May 2004, at 2:45 p.m.
  • Re: Mysterious Master Daemon death
    Preetinder Dhaliwal -- Monday, 31 May 2004, at 4:18 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.