 |
IDS Forum
RE: Informix will not start
Posted By: Everett Mills
Date: Thursday, 22 January 2009, at 6:05 p.m.
In Response To: Re: Informix will not start (Richard Clark)
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> Richard Clark
> Sent: Thursday, January 22, 2009 4:46 PM
> To: ids@iiug.org
> Subject: Re: Informix will not start [14625]
>
> I recreated the links as user informix and rebooted. No help.
>
> I'm not familiar with the dd process but I used it to create a file
that
> obviously had the data from my database in it. Not sure what that
means
> in relation to what you're saying about reserved pages.
>
> What I'm doing is recovering with a tape (make_tape_recovery -I -v -x
> inc_entire=vg00 -x inc_entire=vg01 -x inc_entire=vg02) to an identical
> test box. This has worked several times in the past year but I've
missed
> something this time.
>
> The af files look like they are taking the chunks down and bringing
them
> back:
>
> 17:06:29 Assert Warning: chunk failed sanity check
>
> 17:06:29 Who: Session(1, root@sabers, 0, c80e5018)
>
> Thread(17, main_loop(), c80bb018, 3)
>
> File: rspartn.c Line: 8672
> 17:06:29 Results: Chunk 3 is being taken OFFLINE.
> 17:06:29 Action: Restore chunk from archive.
> 17:06:29 Stack for thread: 17 main_loop()
You are obviously restoring a tape made outside of the database
engine. This is only supported if the engine is put into outside backup
mode (onmode -c block) to stop transactions from being written to the
disk during your backup.
Did you block the server before running the archive?
--EEM
>
> base: 0xc8790000
> len: 69632
>
> pc: 0x00000000
> tos: 0xc8791e90
> state: running
>
> vp: 3
>
> ( 0) 0x008a7a1c legacy_hp_afstack + 0x198 [/opt/informix/bin/oninit]
> ( 1) 0x008a72c0 afstack + 0x48 [/opt/informix/bin/oninit]
> ( 2) 0x008a668c afhandler + 0x8a0 [/opt/informix/bin/oninit]
> ( 3) 0x008a5d4c afwarn_interface + 0x48 [/opt/informix/bin/oninit]
> ( 4) 0x005b6f88 sane_chopen + 0x240 [/opt/informix/bin/oninit]
> ( 5) 0x005b6950 chopen_util + 0x1cc [/opt/informix/bin/oninit]
> ( 6) 0x005b66e0 chopen + 0x10 [/opt/informix/bin/oninit]
> ( 7) 0x0012244c initchunk + 0x1d8 [/opt/informix/bin/oninit]
> ( 8) 0x0011eaa8 initseg + 0xbf4 [/opt/informix/bin/oninit]
> ( 9) 0x00113ed4 main_loop + 0x534 [/opt/informix/bin/oninit]
> (10) 0x008af9e4 startup + 0x94 [/opt/informix/bin/oninit]
> (11) 0x008847e0 mt_swap_threads + 0x10c [/opt/informix/bin/oninit]
>
> base: 0xc8790000
> len: 69632
>
> pc: 0x00000000
> tos: 0xc8791e90
> state: running
>
> vp: 3
>
> 17:06:29 See Also: /opt/informix/tmp/af.3f9ede4
>
> ---------------------------------
> Begin System Alarm Program Output
> ---------------------------------
>
> Assertion Failure Type: Warning
> Host Name: sabers
> Database Server Name: sabers
> Time of failure: Thu Jan 22 17:06:30 EST 2009
> AF file: /opt/informix/tmp/af.3f9ede4
> Shared memory file: None
> System Blocking: OFF
>
> -------------------------------
> End System Alarm Program Output
> -------------------------------
>
> 17:06:30 sh /opt/informix/etc/evidence.sh 1 0
> /opt/informix/tmp/af.3f9ede4 1 0xc80bb018 17 0xc8786e00 1025 0 0 0 0
> 17:06:30
> ------------------ End of assertion failure 0 -----------------
>
> Jarrod Teale wrote:
> > Do the af files that have been generated show any additional
> > information?
> > Are the errors the same or has it marked the chunks as down and now
you
> > need to bring them back online?
> > Also, is this a newly created instance or one that has been running
for
> > a while?
> >
> > The next suggestion that I have for you is getting ugly. The
assumption
> > is that the chunks still contain Informix data. You could dd out the
> > first couple of pages from the chunks and confirm that they are
Informix
> > reserved pages. Art Kagel gave a great presentation about this stuff
at
> > the Informix On Demand conference in Las Vegas last year - reading
from
> > them is safe - it's when you get to writing them back is when IBM
> > support will tell you to go away
> >
> > -----Original Message-----
> > From: Richard Clark [mailto:clark@salem.edu]
> > Sent: Friday, 23 January 2009 10:34 a.m.
> > To: Jarrod Teale
> > Subject: Re: Informix will not start [14617]
> >
> > I changed the chunks to 660. Logical volumes, volume groups and
physical
> > volumes all check out. Informix can follow the link to the device
since
> > everyone has an x.
> >
> > Alas, when I rebooted it gives me the same messages.
> >
> > Richard
> >
> > Jarrod Teale wrote:
> >
> >> I suspect that they shouldn't have the execute bit set.
> >> Permissions should be 660.
> >> Also the symbolic link isn't owned by informix, so informix can't
> >> follow it to write to the device?
> >>
> >> I would:
> >> chmod 777 /opt/informix/dev/dbs0.2
> >> chmod 660 /dev/vg01/rlvol13
> >>
> >> Is the volume group okay? I aren't familiar with HPUX volume
groups,
> >> but on my systems I would make sure that the volume, logical and
> >> physical volumes are all okay as well.
> >>
> >> And try to start again and see if the chuck passes it check.
> >>
> >> -----Original Message-----
> >> From: Richard Clark [mailto:clark@salem.edu]
> >> Sent: Friday, 23 January 2009 9:20 a.m.
> >> To: Jarrod Teale
> >> Subject: Re: Informix will not start [14617]
> >>
> >> Jarrod,
> >>
> >> They're there:
> >>
> >> ls -la /opt/informix/dev/dbs0.2
> >> lrwxr-xr-x 1 root common 17 Jan 17 12:24
> >> /opt/informix/dev/dbs0.2@ -> /dev/vg01/rlvol13
> >>
> >> ls -la /dev/vg01/rlvol13
> >> crwxrwx--- 1 informix informix 64 0x01000d Jan 17 14:14
> >> /dev/vg01/rlvol13*
> >>
> >> No SAN involved.
> >>
> >> Richard
> >>
> >> Jarrod Teale wrote:
> >>
> >>
> >>> So are the chunks there? Paths look to be /opt/informix/dev/dbs0.2
> >>> etc...
> >>> Are they permissions 660 and owned by informix.informix?
> >>>
> >>> If you have a SAN behind this box, are the LUN's active?
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Richard Clark [mailto:clark@salem.edu]
> >>> Sent: Friday, 23 January 2009 8:47 a.m.
> >>> To: Jarrod Teale
> >>> Subject: Re: Informix will not start [14617]
> >>>
> >>> Jarrod,
> >>>
> >>> It looks like every chunk failed sanity check. Here's the last
> >>> attempt
> >>>
> >>>
> >>
> >>> to boot the system:
> >>>
> >>> 13:19:19 IBM Informix Dynamic Server Started.
> >>>
> >>> Thu Jan 22 13:19:20 2009
> >>>
> >>> 13:19:20 Event alarms enabled. ALARMPROG =
> >>> '/opt/informix/etc/no_log.sh'
> >>> 13:19:20 Booting Language <c> from module <> 13:19:20 Loading
> >>> Module
> >>>
> >>>
> >>
> >>> <CNULL> 13:19:20 Booting Language <builtin> from module <>
13:19:20
> >>> Loading Module <BUILTINNULL>
> >>> 13:19:26 DR: DRAUTO is 0 (Off)
> >>> 13:19:27 Dynamically allocated new message shared memory segment
> >>> (size
> >>> 2588KB)
> >>> 13:19:27 IBM Informix Dynamic Server Version 10.00.HC5 Software
> >>> Serial Number AAA#B000000
> >>> 13:19:27 Assert Warning: chunk failed sanity check
> >>>
> >>> 13:19:27 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:27 Who: Session(1, root@sabers, 0, d390c018)
> >>> Thread(17, main_loop(), d38e2018, 1)
> >>> File: rspartn.c Line: 8672
> >>> 13:19:27 Results: Chunk 3 is being taken OFFLINE.
> >>> 13:19:27 Action: Restore chunk from archive.
> >>> 13:19:27 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:27 See Also: /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:28 chunk failed sanity check
> >>>
> >>> 13:19:28 I/O error, Primary Chunk '/opt/informix/dev/dbs0.2' --
> >>> Offline
> >>> (sanity)
> >>> 13:19:28 Assert Warning: chunk failed sanity check
> >>>
> >>> 13:19:28 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:28 Who: Session(1, root@sabers, 0, d390c018)
> >>> Thread(17, main_loop(), d38e2018, 1)
> >>> File: rspartn.c Line: 8672
> >>> 13:19:28 Results: Chunk 4 is being taken OFFLINE.
> >>> 13:19:28 Action: Restore chunk from archive.
> >>> 13:19:28 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:29 See Also: /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:30 chunk failed sanity check
> >>>
> >>> 13:19:30 I/O error, Primary Chunk '/opt/informix/dev/temp0.1' --
> >>> Offline (sanity)
> >>> 13:19:31 Assert Warning: chunk failed sanity check
> >>>
> >>> 13:19:31 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:31 Who: Session(1, root@sabers, 0, d390c018)
> >>> Thread(17, main_loop(), d38e2018, 1)
> >>> File: rspartn.c Line: 8672
> >>> 13:19:31 Results: Chunk 5 is being taken OFFLINE.
> >>> 13:19:31 Action: Restore chunk from archive.
> >>> 13:19:31 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:31 See Also: /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:32 chunk failed sanity check
> >>>
> >>> 13:19:33 I/O error, Primary Chunk '/opt/informix/dev/temp1.1' --
> >>> Offline (sanity)
> >>> 13:19:33 Assert Warning: chunk failed sanity check
> >>>
> >>> 13:19:33 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:33 Who: Session(1, root@sabers, 0, d390c018)
> >>> Thread(17, main_loop(), d38e2018, 1)
> >>> File: rspartn.c Line: 8672
> >>> 13:19:33 Results: Chunk 6 is being taken OFFLINE.
> >>> 13:19:33 Action: Restore chunk from archive.
> >>> 13:19:33 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:33 See Also: /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:34 chunk failed sanity check
> >>>
> >>> 13:19:35 I/O error, Primary Chunk '/opt/informix/dev/dbs1.1' --
> >>> Offline
> >>> (sanity)
> >>> 13:19:35 Assert Warning: chunk failed sanity check
> >>>
> >>> 13:19:35 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:35 Who: Session(1, root@sabers, 0, d390c018)
> >>> Thread(17, main_loop(), d38e2018, 1)
> >>> File: rspartn.c Line: 8672
> >>> 13:19:35 Results: Chunk 7 is being taken OFFLINE.
> >>> 13:19:35 Action: Restore chunk from archive.
> >>> 13:19:35 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:35 See Also: /opt/informix/tmp/af.3f9b8ae
> >>> 13:19:36 chunk failed sanity check
> >>>
> >>> 13:19:37 I/O error, Primary Chunk '/opt/informix/dev/dbs1.2' --
> >>> Offline
> >>> (sanity)
> >>> 13:19:37 IBM Informix Dynamic Server Initialized -- Shared Memory
> >>> Initialized.
> >>>
> >>> 13:19:37 Warning: Invalid (non-existent/blobspace/disabled)
dbspace
> >>> listed
> >>> in DBSPACETEMP: 'temp0'
> >>> 13:19:37 Warning: Invalid (non-existent/blobspace/disabled)
dbspace
> >>> listed
> >>> in DBSPACETEMP: 'temp1'
> >>> 13:19:37 Physical Recovery Started at Page (1:722).
> >>> 13:19:37 Physical Recovery Complete: 0 Pages Examined, 0 Pages
> >>> Restored.
> >>> 13:19:37 Assert Warning: pthdrpage:ptalloc:bad bfget
> >>> 13:19:37 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:37 Who: Session(16, root@sabers, 0, d390da80)
> >>> Thread(39, fast_rec, d38e63f0, 1)
> >>> File: rspartn.c Line: 6725
> >>> 13:19:37 Results: Cannot use TBLSpace page for TBLSpace 5242881
> >>> 13:19:37 Action: Run 'oncheck -pt 5242881'
> >>> 13:19:37 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.40fb8b9
> >>> 13:19:37 See Also: /opt/informix/tmp/af.40fb8b9
> >>> 13:19:38 pthdrpage:ptalloc:bad bfget
> >>> 13:19:38 Assert Failed: Chunk 6 is being taken OFFLINE.
> >>> 13:19:38 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:38 Who: Session(16, root@sabers, 0, d390da80)
> >>> Thread(39, fast_rec, d38e63f0, 1)
> >>> File: rspartn.c Line: 6744
> >>> 13:19:38 Results: Cannot Open DBspace 5.
> >>> 13:19:38 Action: Restore chunk from archive.
> >>> in DBSPACETEMP: 'temp1'
> >>> 13:19:37 Physical Recovery Started at Page (1:722).
> >>> 13:19:37 Physical Recovery Complete: 0 Pages Examined, 0 Pages
> >>> Restored.
> >>> 13:19:37 Assert Warning: pthdrpage:ptalloc:bad bfget
> >>> 13:19:37 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:37 Who: Session(16, root@sabers, 0, d390da80)
> >>> Thread(39, fast_rec, d38e63f0, 1)
> >>> File: rspartn.c Line: 6725
> >>> 13:19:37 Results: Cannot use TBLSpace page for TBLSpace 5242881
> >>> 13:19:37 Action: Run 'oncheck -pt 5242881'
> >>> 13:19:37 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.40fb8b9
> >>> 13:19:37 See Also: /opt/informix/tmp/af.40fb8b9
> >>> 13:19:38 pthdrpage:ptalloc:bad bfget
> >>> 13:19:38 Assert Failed: Chunk 6 is being taken OFFLINE.
> >>> 13:19:38 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:38 Who: Session(16, root@sabers, 0, d390da80)
> >>> Thread(39, fast_rec, d38e63f0, 1)
> >>> File: rspartn.c Line: 6744
> >>> 13:19:38 Results: Cannot Open DBspace 5.
> >>> 13:19:38 Action: Restore chunk from archive.
> >>> 13:19:38 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.40fb8b9
> >>> 13:19:38 See Also: /opt/informix/tmp/af.40fb8b9
> >>> 13:19:45 Chunk 6 is being taken OFFLINE.
> >>> 13:19:45 Assert Warning: Chunk 6 is being taken OFFLINE.
> >>> 13:19:45 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:45 Who: Session(16, root@sabers, 0, d390da80)
> >>> Thread(39, fast_rec, d38e63f0, 1)
> >>> File: rsmirror.c Line: 1921
> >>> 13:19:45 Results: DBspace dbs1 is disabled.
> >>> 13:19:45 Action: Restore DBspace dbs1
> >>> 13:19:45 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.40fb8b9
> >>> 13:19:46 See Also: /opt/informix/tmp/af.40fb8b9
> >>> 13:19:46 Chunk 6 is being taken OFFLINE.
> >>> 13:19:47 Cannot Open DBspace 5.
> >>> 13:19:47 Logical Recovery Started.
> >>> 13:19:47 10 recovery worker threads will be started.
> >>> 13:19:47 Assert Failed: Dynamic Server must abort
> >>> 13:19:47 IBM Informix Dynamic Server Version 10.00.HC5
> >>> 13:19:47 Who: Session(16, root@sabers, 0, d390da80)
> >>> Thread(39, fast_rec, d38e63f0, 1)
> >>> File: rslog.c Line: 3654
> >>> 13:19:47 Results: Dynamic Server must abort
> >>> 13:19:47 Action: Reinitialize shared memory
> >>> 13:19:47 stack trace for pid 9345 written to
> >>> /opt/informix/tmp/af.40fb8b9
> >>> 13:19:47 See Also: /opt/informix/tmp/af.40fb8b9
> >>> 13:19:54 rslog.c, line 3654, thread 39, proc id 9345, Dynamic
Server
> >>>
> >
> >
> >>> must abort.
> >>> 13:19:54 PANIC: Attempting to bring system down
> >>> 13:19:54 semctl: errno = 22
> >>> ......
> >>>
> >>>
> >>> Jarrod Teale wrote:
> >>>
> >>>
> >>>
> >>>> Can you post the last 20 lines or so of the online log - it
should
> >>>> have something in there about the engine starting / failing.
> >>>> Jarrod Teale
> >>>> Fonterra NZ
> >>>>
> >>>> -----Original Message-----
> >>>> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On
Behalf
> >>>> Of
> >>>>
> >>>>
> >>
> >>>> RICHARD CLARK
> >>>> Sent: Friday, 23 January 2009 8:19 a.m.
> >>>> To: ids@iiug.org
> >>>> Subject: Informix will not start [14616]
> >>>>
> >>>> I'm trying to get Informix 10 to start on an HP-UX 11.2 system.
The
> >>>> message I get at boot time is:
> >>>>
> >>>> Starting Informix OnLine engine(s)
> >>>> Output from "/sbin/rc3.d/S97informix start":
> >>>> ----------------------------
> >>>> "/sbin/rc3.d/S97informix start" FAILED
> >>>>
> >>>> If I run oninit -v everything looks ok (succeeds). But no oninit
> >>>> process shows up.
> >>>>
> >>>> If I try to run sql I get the message:
> >>>>
> >>>> The appl process cannot connect to the database server sabers.
> >>>> See error number -25588.
> >>>> Database cars not found or not correct format
> >>>>
> >>>> onconf:
> >>>> DBSERVERNAME sabers
> >>>>
> >>>> sqlhosts:
> >>>> sabers onipcshm 10.81.101.1 sabers
> >>>> carsi_tcp onsoctcp 10.81.101.1 istarcarsi
> >>>>
> >>>> The command dbspace brings up nothing.
> >>>>
> >>>> Richard Clark
> >>>>
> >>>>
********************************************************************
> >>>> *
> >>>> *
> >>>> **
> >>>> *******
> >>>> Forum Note: Use "Reply" to post a response in the discussion
forum.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
*********************************************************************
> >>> *
> >>> **
> >>> *******
> >>>
> >>>
> >>>
> >>>> Forum Note: Use "Reply" to post a response in the discussion
> >>>>
> > forum.
> >
> >>
> >>>>
> >>>>
> >>> --
> >>> Richard Clark
> >>> CX Administrator
> >>> Salem Academy & College
> >>>
> >>>
> >>>
> >>> DISCLAIMER:
> >>> This email contains confidential information and may be legally
> >>>
> >>>
> >> privileged. If you are not the intended recipient or have received
> >> this email in error, please notify the sender immediately and
destroy
> >> this email.
> >>
> >>
> >>> You may not use, disclose or copy this email or its attachments in
> >>> any
> >>>
> >>>
> >> way.
> >>
> >>
> >>> Any opinions expressed in this email are those of the author and
are
> >>>
> >>>
> >> not necessarily those of the Fonterra Co-operative Group.
> >>
> >>
> >>> http://www.fonterra.com/
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Richard Clark
> >> CX Administrator
> >> Salem Academy & College
> >>
> >>
> >>
> >> DISCLAIMER:
> >> This email contains confidential information and may be legally
> >>
> > privileged. If you are not the intended recipient or have received
this
> > email in error, please notify the sender immediately and destroy
this
> > email.
> >
> >> You may not use, disclose or copy this email or its attachments in
any
> >>
> > way.
> >
> >> Any opinions expressed in this email are those of the author and
are
> >>
> > not necessarily those of the Fonterra Co-operative Group.
> >
> >> http://www.fonterra.com/
> >>
> >>
> >>
> >
> >
>
> --
> Richard Clark
> CX Administrator
> Salem Academy & College
>
>
>
************************************************************************
**
> *****
> Forum Note: Use "Reply" to post a response in the discussion forum.
Messages In This Thread
IDS Forum is maintained by Administrator with WebBBS 5.12.
|
|
