I recreated the links as user informix and rebooted. No help.
I'm not familiar with the dd process but I used it to create a file that
obviously had the data from my database in it. Not sure what that means
in relation to what you're saying about reserved pages.
What I'm doing is recovering with a tape (make_tape_recovery -I -v -x
inc_entire=vg00 -x inc_entire=vg01 -x inc_entire=vg02) to an identical
test box. This has worked several times in the past year but I've missed
something this time.
The af files look like they are taking the chunks down and bringing them
back:
17:06:29 Assert Warning: chunk failed sanity check
17:06:29 Who: Session(1, root@sabers, 0, c80e5018)
Thread(17, main_loop(), c80bb018, 3)
File: rspartn.c Line: 8672
17:06:29 Results: Chunk 3 is being taken OFFLINE.
17:06:29 Action: Restore chunk from archive.
17:06:29 Stack for thread: 17 main_loop()
base: 0xc8790000
len: 69632
pc: 0x00000000
tos: 0xc8791e90
state: running
vp: 3
( 0) 0x008a7a1c legacy_hp_afstack + 0x198 [/opt/informix/bin/oninit]
( 1) 0x008a72c0 afstack + 0x48 [/opt/informix/bin/oninit]
( 2) 0x008a668c afhandler + 0x8a0 [/opt/informix/bin/oninit]
( 3) 0x008a5d4c afwarn_interface + 0x48 [/opt/informix/bin/oninit]
( 4) 0x005b6f88 sane_chopen + 0x240 [/opt/informix/bin/oninit]
( 5) 0x005b6950 chopen_util + 0x1cc [/opt/informix/bin/oninit]
( 6) 0x005b66e0 chopen + 0x10 [/opt/informix/bin/oninit]
( 7) 0x0012244c initchunk + 0x1d8 [/opt/informix/bin/oninit]
( 8) 0x0011eaa8 initseg + 0xbf4 [/opt/informix/bin/oninit]
( 9) 0x00113ed4 main_loop + 0x534 [/opt/informix/bin/oninit]
(10) 0x008af9e4 startup + 0x94 [/opt/informix/bin/oninit]
(11) 0x008847e0 mt_swap_threads + 0x10c [/opt/informix/bin/oninit]
base: 0xc8790000
len: 69632
pc: 0x00000000
tos: 0xc8791e90
state: running
vp: 3
17:06:29 See Also: /opt/informix/tmp/af.3f9ede4
---------------------------------
Begin System Alarm Program Output
---------------------------------
Assertion Failure Type: Warning
Host Name: sabers
Database Server Name: sabers
Time of failure: Thu Jan 22 17:06:30 EST 2009
AF file: /opt/informix/tmp/af.3f9ede4
Shared memory file: None
System Blocking: OFF
-------------------------------
End System Alarm Program Output
-------------------------------
17:06:30 sh /opt/informix/etc/evidence.sh 1 0
/opt/informix/tmp/af.3f9ede4 1 0xc80bb018 17 0xc8786e00 1025 0 0 0 0
17:06:30
------------------ End of assertion failure 0 -----------------
Jarrod Teale wrote:
> Do the af files that have been generated show any additional
> information?
> Are the errors the same or has it marked the chunks as down and now you
> need to bring them back online?
> Also, is this a newly created instance or one that has been running for
> a while?
>
> The next suggestion that I have for you is getting ugly. The assumption
> is that the chunks still contain Informix data. You could dd out the
> first couple of pages from the chunks and confirm that they are Informix
> reserved pages. Art Kagel gave a great presentation about this stuff at
> the Informix On Demand conference in Las Vegas last year - reading from
> them is safe - it's when you get to writing them back is when IBM
> support will tell you to go away
>
> -----Original Message-----
> From: Richard Clark [mailto:clark@salem.edu]
> Sent: Friday, 23 January 2009 10:34 a.m.
> To: Jarrod Teale
> Subject: Re: Informix will not start [14617]
>
> I changed the chunks to 660. Logical volumes, volume groups and physical
> volumes all check out. Informix can follow the link to the device since
> everyone has an x.
>
> Alas, when I rebooted it gives me the same messages.
>
> Richard
>
> Jarrod Teale wrote:
>
>> I suspect that they shouldn't have the execute bit set.
>> Permissions should be 660.
>> Also the symbolic link isn't owned by informix, so informix can't
>> follow it to write to the device?
>>
>> I would:
>> chmod 777 /opt/informix/dev/dbs0.2
>> chmod 660 /dev/vg01/rlvol13
>>
>> Is the volume group okay? I aren't familiar with HPUX volume groups,
>> but on my systems I would make sure that the volume, logical and
>> physical volumes are all okay as well.
>>
>> And try to start again and see if the chuck passes it check.
>>
>> -----Original Message-----
>> From: Richard Clark [mailto:clark@salem.edu]
>> Sent: Friday, 23 January 2009 9:20 a.m.
>> To: Jarrod Teale
>> Subject: Re: Informix will not start [14617]
>>
>> Jarrod,
>>
>> They're there:
>>
>> ls -la /opt/informix/dev/dbs0.2
>> lrwxr-xr-x 1 root common 17 Jan 17 12:24
>> /opt/informix/dev/dbs0.2@ -> /dev/vg01/rlvol13
>>
>> ls -la /dev/vg01/rlvol13
>> crwxrwx--- 1 informix informix 64 0x01000d Jan 17 14:14
>> /dev/vg01/rlvol13*
>>
>> No SAN involved.
>>
>> Richard
>>
>> Jarrod Teale wrote:
>>
>>
>>> So are the chunks there? Paths look to be /opt/informix/dev/dbs0.2
>>> etc...
>>> Are they permissions 660 and owned by informix.informix?
>>>
>>> If you have a SAN behind this box, are the LUN's active?
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Richard Clark [mailto:clark@salem.edu]
>>> Sent: Friday, 23 January 2009 8:47 a.m.
>>> To: Jarrod Teale
>>> Subject: Re: Informix will not start [14617]
>>>
>>> Jarrod,
>>>
>>> It looks like every chunk failed sanity check. Here's the last
>>> attempt
>>>
>>>
>>
>>> to boot the system:
>>>
>>> 13:19:19 IBM Informix Dynamic Server Started.
>>>
>>> Thu Jan 22 13:19:20 2009
>>>
>>> 13:19:20 Event alarms enabled. ALARMPROG =
>>> '/opt/informix/etc/no_log.sh'
>>> 13:19:20 Booting Language <c> from module <> 13:19:20 Loading
>>> Module
>>>
>>>
>>
>>> <CNULL> 13:19:20 Booting Language <builtin> from module <> 13:19:20
>>> Loading Module <BUILTINNULL>
>>> 13:19:26 DR: DRAUTO is 0 (Off)
>>> 13:19:27 Dynamically allocated new message shared memory segment
>>> (size
>>> 2588KB)
>>> 13:19:27 IBM Informix Dynamic Server Version 10.00.HC5 Software
>>> Serial Number AAA#B000000
>>> 13:19:27 Assert Warning: chunk failed sanity check
>>>
>>> 13:19:27 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:27 Who: Session(1, root@sabers, 0, d390c018)
>>> Thread(17, main_loop(), d38e2018, 1)
>>> File: rspartn.c Line: 8672
>>> 13:19:27 Results: Chunk 3 is being taken OFFLINE.
>>> 13:19:27 Action: Restore chunk from archive.
>>> 13:19:27 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.3f9b8ae
>>> 13:19:27 See Also: /opt/informix/tmp/af.3f9b8ae
>>> 13:19:28 chunk failed sanity check
>>>
>>> 13:19:28 I/O error, Primary Chunk '/opt/informix/dev/dbs0.2' --
>>> Offline
>>> (sanity)
>>> 13:19:28 Assert Warning: chunk failed sanity check
>>>
>>> 13:19:28 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:28 Who: Session(1, root@sabers, 0, d390c018)
>>> Thread(17, main_loop(), d38e2018, 1)
>>> File: rspartn.c Line: 8672
>>> 13:19:28 Results: Chunk 4 is being taken OFFLINE.
>>> 13:19:28 Action: Restore chunk from archive.
>>> 13:19:28 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.3f9b8ae
>>> 13:19:29 See Also: /opt/informix/tmp/af.3f9b8ae
>>> 13:19:30 chunk failed sanity check
>>>
>>> 13:19:30 I/O error, Primary Chunk '/opt/informix/dev/temp0.1' --
>>> Offline (sanity)
>>> 13:19:31 Assert Warning: chunk failed sanity check
>>>
>>> 13:19:31 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:31 Who: Session(1, root@sabers, 0, d390c018)
>>> Thread(17, main_loop(), d38e2018, 1)
>>> File: rspartn.c Line: 8672
>>> 13:19:31 Results: Chunk 5 is being taken OFFLINE.
>>> 13:19:31 Action: Restore chunk from archive.
>>> 13:19:31 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.3f9b8ae
>>> 13:19:31 See Also: /opt/informix/tmp/af.3f9b8ae
>>> 13:19:32 chunk failed sanity check
>>>
>>> 13:19:33 I/O error, Primary Chunk '/opt/informix/dev/temp1.1' --
>>> Offline (sanity)
>>> 13:19:33 Assert Warning: chunk failed sanity check
>>>
>>> 13:19:33 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:33 Who: Session(1, root@sabers, 0, d390c018)
>>> Thread(17, main_loop(), d38e2018, 1)
>>> File: rspartn.c Line: 8672
>>> 13:19:33 Results: Chunk 6 is being taken OFFLINE.
>>> 13:19:33 Action: Restore chunk from archive.
>>> 13:19:33 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.3f9b8ae
>>> 13:19:33 See Also: /opt/informix/tmp/af.3f9b8ae
>>> 13:19:34 chunk failed sanity check
>>>
>>> 13:19:35 I/O error, Primary Chunk '/opt/informix/dev/dbs1.1' --
>>> Offline
>>> (sanity)
>>> 13:19:35 Assert Warning: chunk failed sanity check
>>>
>>> 13:19:35 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:35 Who: Session(1, root@sabers, 0, d390c018)
>>> Thread(17, main_loop(), d38e2018, 1)
>>> File: rspartn.c Line: 8672
>>> 13:19:35 Results: Chunk 7 is being taken OFFLINE.
>>> 13:19:35 Action: Restore chunk from archive.
>>> 13:19:35 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.3f9b8ae
>>> 13:19:35 See Also: /opt/informix/tmp/af.3f9b8ae
>>> 13:19:36 chunk failed sanity check
>>>
>>> 13:19:37 I/O error, Primary Chunk '/opt/informix/dev/dbs1.2' --
>>> Offline
>>> (sanity)
>>> 13:19:37 IBM Informix Dynamic Server Initialized -- Shared Memory
>>> Initialized.
>>>
>>> 13:19:37 Warning: Invalid (non-existent/blobspace/disabled) dbspace
>>> listed
>>> in DBSPACETEMP: 'temp0'
>>> 13:19:37 Warning: Invalid (non-existent/blobspace/disabled) dbspace
>>> listed
>>> in DBSPACETEMP: 'temp1'
>>> 13:19:37 Physical Recovery Started at Page (1:722).
>>> 13:19:37 Physical Recovery Complete: 0 Pages Examined, 0 Pages
>>> Restored.
>>> 13:19:37 Assert Warning: pthdrpage:ptalloc:bad bfget
>>> 13:19:37 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:37 Who: Session(16, root@sabers, 0, d390da80)
>>> Thread(39, fast_rec, d38e63f0, 1)
>>> File: rspartn.c Line: 6725
>>> 13:19:37 Results: Cannot use TBLSpace page for TBLSpace 5242881
>>> 13:19:37 Action: Run 'oncheck -pt 5242881'
>>> 13:19:37 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.40fb8b9
>>> 13:19:37 See Also: /opt/informix/tmp/af.40fb8b9
>>> 13:19:38 pthdrpage:ptalloc:bad bfget
>>> 13:19:38 Assert Failed: Chunk 6 is being taken OFFLINE.
>>> 13:19:38 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:38 Who: Session(16, root@sabers, 0, d390da80)
>>> Thread(39, fast_rec, d38e63f0, 1)
>>> File: rspartn.c Line: 6744
>>> 13:19:38 Results: Cannot Open DBspace 5.
>>> 13:19:38 Action: Restore chunk from archive.
>>> in DBSPACETEMP: 'temp1'
>>> 13:19:37 Physical Recovery Started at Page (1:722).
>>> 13:19:37 Physical Recovery Complete: 0 Pages Examined, 0 Pages
>>> Restored.
>>> 13:19:37 Assert Warning: pthdrpage:ptalloc:bad bfget
>>> 13:19:37 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:37 Who: Session(16, root@sabers, 0, d390da80)
>>> Thread(39, fast_rec, d38e63f0, 1)
>>> File: rspartn.c Line: 6725
>>> 13:19:37 Results: Cannot use TBLSpace page for TBLSpace 5242881
>>> 13:19:37 Action: Run 'oncheck -pt 5242881'
>>> 13:19:37 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.40fb8b9
>>> 13:19:37 See Also: /opt/informix/tmp/af.40fb8b9
>>> 13:19:38 pthdrpage:ptalloc:bad bfget
>>> 13:19:38 Assert Failed: Chunk 6 is being taken OFFLINE.
>>> 13:19:38 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:38 Who: Session(16, root@sabers, 0, d390da80)
>>> Thread(39, fast_rec, d38e63f0, 1)
>>> File: rspartn.c Line: 6744
>>> 13:19:38 Results: Cannot Open DBspace 5.
>>> 13:19:38 Action: Restore chunk from archive.
>>> 13:19:38 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.40fb8b9
>>> 13:19:38 See Also: /opt/informix/tmp/af.40fb8b9
>>> 13:19:45 Chunk 6 is being taken OFFLINE.
>>> 13:19:45 Assert Warning: Chunk 6 is being taken OFFLINE.
>>> 13:19:45 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:45 Who: Session(16, root@sabers, 0, d390da80)
>>> Thread(39, fast_rec, d38e63f0, 1)
>>> File: rsmirror.c Line: 1921
>>> 13:19:45 Results: DBspace dbs1 is disabled.
>>> 13:19:45 Action: Restore DBspace dbs1
>>> 13:19:45 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.40fb8b9
>>> 13:19:46 See Also: /opt/informix/tmp/af.40fb8b9
>>> 13:19:46 Chunk 6 is being taken OFFLINE.
>>> 13:19:47 Cannot Open DBspace 5.
>>> 13:19:47 Logical Recovery Started.
>>> 13:19:47 10 recovery worker threads will be started.
>>> 13:19:47 Assert Failed: Dynamic Server must abort
>>> 13:19:47 IBM Informix Dynamic Server Version 10.00.HC5
>>> 13:19:47 Who: Session(16, root@sabers, 0, d390da80)
>>> Thread(39, fast_rec, d38e63f0, 1)
>>> File: rslog.c Line: 3654
>>> 13:19:47 Results: Dynamic Server must abort
>>> 13:19:47 Action: Reinitialize shared memory
>>> 13:19:47 stack trace for pid 9345 written to
>>> /opt/informix/tmp/af.40fb8b9
>>> 13:19:47 See Also: /opt/informix/tmp/af.40fb8b9
>>> 13:19:54 rslog.c, line 3654, thread 39, proc id 9345, Dynamic Server
>>>
>
>
>>> must abort.
>>> 13:19:54 PANIC: Attempting to bring system down
>>> 13:19:54 semctl: errno = 22
>>> ......
>>>
>>>
>>> Jarrod Teale wrote:
>>>
>>>
>>>
>>>> Can you post the last 20 lines or so of the online log - it should
>>>> have something in there about the engine starting / failing.
>>>> Jarrod Teale
>>>> Fonterra NZ
>>>>
>>>> -----Original Message-----
>>>> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf
>>>> Of
>>>>
>>>>
>>
>>>> RICHARD CLARK
>>>> Sent: Friday, 23 January 2009 8:19 a.m.
>>>> To: ids@iiug.org
>>>> Subject: Informix will not start [14616]
>>>>
>>>> I'm trying to get Informix 10 to start on an HP-UX 11.2 system. The
>>>> message I get at boot time is:
>>>>
>>>> Starting Informix OnLine engine(s)
>>>> Output from "/sbin/rc3.d/S97informix start":
>>>> ----------------------------
>>>> "/sbin/rc3.d/S97informix start" FAILED
>>>>
>>>> If I run oninit -v everything looks ok (succeeds). But no oninit
>>>> process shows up.
>>>>
>>>> If I try to run sql I get the message:
>>>>
>>>> The appl process cannot connect to the database server sabers.
>>>> See error number -25588.
>>>> Database cars not found or not correct format
>>>>
>>>> onconf:
>>>> DBSERVERNAME sabers
>>>>
>>>> sqlhosts:
>>>> sabers onipcshm 10.81.101.1 sabers
>>>> carsi_tcp onsoctcp 10.81.101.1 istarcarsi
>>>>
>>>> The command dbspace brings up nothing.
>>>>
>>>> Richard Clark
>>>>
>>>> ********************************************************************
>>>> *
>>>> *
>>>> **
>>>> *******
>>>> Forum Note: Use "Reply" to post a response in the discussion forum.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> *********************************************************************
>>> *
>>> **
>>> *******
>>>
>>>
>>>
>>>> Forum Note: Use "Reply" to post a response in the discussion
>>>>
> forum.
>
>>
>>>>
>>>>
>>> --
>>> Richard Clark
>>> CX Administrator
>>> Salem Academy & College
>>>
>>>
>>>
>>> DISCLAIMER:
>>> This email contains confidential information and may be legally
>>>
>>>
>> privileged. If you are not the intended recipient or have received
>> this email in error, please notify the sender immediately and destroy
>> this email.
>>
>>
>>> You may not use, disclose or copy this email or its attachments in
>>> any
>>>
>>>
>> way.
>>
>>
>>> Any opinions expressed in this email are those of the author and are
>>>
>>>
>> not necessarily those of the Fonterra Co-operative Group.
>>
>>
>>> http://www.fonterra.com/
>>>
>>>
>>>
>>>
>> --
>> Richard Clark
>> CX Administrator
>> Salem Academy & College
>>
>>
>>
>> DISCLAIMER:
>> This email contains confidential information and may be legally
>>
> privileged. If you are not the intended recipient or have received this
> email in error, please notify the sender immediately and destroy this
> email.
>
>> You may not use, disclose or copy this email or its attachments in any
>>
> way.
>
>> Any opinions expressed in this email are those of the author and are
>>
> not necessarily those of the Fonterra Co-operative Group.
>
>> http://www.fonterra.com/
>>
>>
>>
>
>
--
Richard Clark
CX Administrator
Salem Academy & College
