IDS Forum

Re: 11.5 UC7E crashes

Posted By: Martin Fuerderer
Date: Thursday, 22 December 2011, at 5:17 a.m.

In Response To: Re: 11.5 UC7E crashes (KLAUS GLAHN)

Hallo Klaus,

you got already some (new) answers, so here
I'll only write down some of my thoughts that may
not have been covered by those previous answers ...

> ... tried onclean, but this doesn't help. ...
I didn't check out the source code of this utility,
but there can't be too much magic for what it is
doing (or trying to do). Does the onclean show
any error or warning message?

> /proc/<pid>/maps is empty for all my oninit-processes
What exactly do you mean by "empty" ?
After the server crashed, when the oninit processes have
died, there should not even be a /proc/<pid> directory
anymore for those processes, as they don't exist anymore.
If a process (still) exists, its maps file in /proc/<pid> should
never be empty (i.,e. contain nothing at all), as any process
will at least occupy some memory, if it is only the TEXT
portion (i.e. the program code itself, that is loaded into memory).

These two statements from you ... somehow do not fit the
picture, that is expected of a Linux system ... hmm.

My guess is, that when there is the situation of
"shmget: [EEXIST][17]: key 52574802: shared memory
already exists", there is still some (old) process "hanging
around" and still attached to that shared memory segment.
In this case, the onclean utility will not be able to remove
the SHM segment, because Linux doesn't allow it. Also
as user "root" using the ipcrm system utility you will not
be able to remove an SHM segment to which some process
is still attached.

Using "ipcs -m" as user root, you should be able to
identify that SHM segment (using the key). The output of
"ipcs -m" usually has a column named "nattch" which means
"number of processes attached". It is likely non-zero (otherwise
onclean would have removed it already). Possibly the column
"status" says something like "destroyed", which means that
the Linux OS will automatically remove it once the last process
has detached from it. Unfortunately, "ipcs" doesn't tell, which
process (pid) is still attached. :-(
So the task is to find and kill that process (or those processes).

There are two possibilities:

a) There is an (old) oninit process left over, that is still attached
to the SHM segment. If this Informix Server instance is the
only one running on this machine, then simply do a command
like "ps -efw | grep oninit" to see, if there is an oninit process
still (half) alive. If you identify some oninit processes that
should not be there anymore: kill them as user "root":
"kill -9 <pid> [ <pid> ... ]"
In case you have other Informix Server instances running on
this machine, set the respective environment for each instance
and do "onstat -g sch". The output will include a listing of all the
associated oninit processes with their PID. These PIDs should
not be killed with above "kill -9 ..." command ... :)

b) Some client process (i.e. not an oninit process) is still running
and attached to the SHM segment. This could e.g. be an onbar
process (try "ps -efw | grep onbar"). In case you have configured
SHM communication between clients and Informix Server, it could
be any local client process that has connected to the server using
SHM for communication. Check in the sqlhosts file, whether there
is an entry with "onipcshm" as the second parameter. Check for
any client process (you have to know or figure out, what could be
their name ... usually "dbaccess" is one candidate for running ad
hoc test queries), using the same "ps -efw | grep ..." method.
If you find something ... kill it as well, using "kill -9 <pid>".

Once all process are gone (have been killed), the SHM segments
should have been removed. In a rare case where a segment could
still be left over, again running onclean should do the job.

Regards, Martin
--
Martin Fuerderer
IBM Informix Development Munich, Germany
Information Management

Read about the Informix Warehouse Accelerator:
http://tinyurl.com/the-iwa-blog

IBM Deutschland Research & Development GmbH
Chairman of the Supervisory Board: Martina Koederitz
Board of Management: Dirk Wittkopp
Corporate Seat: Boeblingen, Germany
Reg.-Gericht: Amtsgericht Stuttgart, HRB 243294

ids-bounces@iiug.org wrote on 12/21/2011 08:40:45 PM:
> Hello,
>
> thank you for your answers. I already assumed that it´s a memory
problem. But
> with my limited knowledge (and I have the Eric-Herber-book ;-) it´s
already
> difficult to ask the right questions.
>
> We have a test instance and I tried
>
> SHMVIRTSIZE 800000
> SHMADD 150000
>
> but when I do oninit I get an error message
>
> 20:20:33 IBM Informix Dynamic Server gestartet.
> 20:20:33 shmget: [EEXIST][17]: key 52574802: shared memory already
exists
> 20:20:33 mt_shm_init: Segment resident kann nicht angelegt werden
>
> After that the instance couldn´t start until I changed the
SERVERNUM-Parameter
> to another number (thanks to Google :-) I don´t know how to clean this
up, I
> tried onclean - but this doesn´t help. Is reboot the only solution?
>
> As far as I know (I didn´t install the machine, there is little
documentation)
> it´s a 32-Bit-Informix running on Debian with Bigmem-Extension. The
crashes
> seem real random - but we have approx. 100 users working with the
database
> with different applications - and some of them using MS-Access and can
do
> large transactions.
>
> /proc/<pid>/maps is empty for all my oninit-processes
>
> Should I try lower values for SHMVIRTSIZE? I don´t want to experiment,
because
> I don´t want to change the SERVERNUM after each failed test.
>
> Sorry, I am really a newbie to the Informix internals. The last years
> everything was fine - until we wanted logging ... Thanks again for your
help.
>
> All the best
> Klaus
>
>
>
*******************************************************************************

> Forum Note: Use "Reply" to post a response in the discussion forum.
>

Messages In This Thread

• 11.5 UC7E crashes
  KLAUS GLAHN -- Tuesday, 20 December 2011, at 2:06 p.m.
  • RE: 11.5 UC7E crashes
    Joerg Volz -- Tuesday, 20 December 2011, at 2:58 p.m.
  • RE: 11.5 UC7E crashes
    Marcus Haarmann -- Wednesday, 21 December 2011, at 3:06 a.m.
  • Re: 11.5 UC7E crashes
    KLAUS GLAHN -- Wednesday, 21 December 2011, at 2:40 p.m.
    • Re: 11.5 UC7E crashes
      Art Kagel -- Wednesday, 21 December 2011, at 2:56 p.m.
      • Re: 11.5 UC7E crashes
        rroussey@comcast.net -- Wednesday, 21 December 2011, at 3:50 p.m.
        • Re: 11.5 UC7E crashes
          KLAUS GLAHN -- Wednesday, 21 December 2011, at 6:00 p.m.
    • Re: 11.5 UC7E crashes
      Joerg Volz -- Wednesday, 21 December 2011, at 2:58 p.m.
      • Re: 11.5 UC7E crashes
        KLAUS GLAHN -- Wednesday, 21 December 2011, at 5:50 p.m.
        • RE: 11.5 UC7E crashes
          Joerg Volz -- Wednesday, 21 December 2011, at 6:06 p.m.
        • Re: 11.5 UC7E crashes
          Art Kagel -- Wednesday, 21 December 2011, at 7:20 p.m.
        • Re: 11.5 UC7E crashes
          Khaled Bentebal -- Wednesday, 21 December 2011, at 10:04 p.m.
          • Re: 11.5 UC7E crashes
            Fernando Nunes -- Wednesday, 21 December 2011, at 10:28 p.m.
            • Re: 11.5 UC7E crashes
              Khaled Bentebal -- Thursday, 22 December 2011, at 7:26 a.m.
    • Re: 11.5 UC7E crashes
      Martin Fuerderer -- Thursday, 22 December 2011, at 5:17 a.m.
      • Re: 11.5 UC7E crashes
        Harshavardhan Changappa -- Thursday, 22 December 2011, at 6:37 a.m.
  • Re: 11.5 UC7E crashes
    KLAUS GLAHN -- Friday, 23 December 2011, at 6:51 a.m.
    • RE: 11.5 UC7E crashes
      Joerg Volz -- Friday, 23 December 2011, at 7:07 a.m.
      • Re: RE: 11.5 UC7E crashes
        KLAUS GLAHN -- Friday, 23 December 2011, at 9:55 a.m.
        • RE: Re: RE: 11.5 UC7E crashes
          Marcus Haarmann -- Friday, 23 December 2011, at 10:39 a.m.
  • Status
    KLAUS GLAHN -- Thursday, 29 December 2011, at 2:24 p.m.
    • crash again
      KLAUS GLAHN -- Wednesday, 4 January 2012, at 9:39 a.m.
      • RE: crash again
        Marcus Haarmann -- Wednesday, 4 January 2012, at 9:48 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.