IDS Forum

RE: Any assistance greatly appreciated:

Posted By: Burton, Clifford
Date: Monday, 24 January 2005, at 5:22 p.m.

In Response To: RE: Any assistance greatly appreciated: (Alexey Sonkin)

Hi All,

Well, first I'll address the questions that have been raised ~ and
then I'll give you something of a status update.

1) The preceding lines in the online log relate to the instance
being brought off-line prior to the restore, there are no entries during the
restore process, the only entries relate to when the restore completed
(however successfully) and the instance was initialised, and subsequent
attempts to start the instance.

2) onbar -r -w was the restore command used.

3) The backup completed successfully that was used for the
restore. (in fact the 4 tapesets we have used all reported success).

4) IBM have looked into the matter, and have suggested a full
physical restore, onbar -r -p -w, as some of you had suggested we had hoped
that we would not need to perform the full restore I don't have the 12
seconds to waste :-) or rather 30 hours it takes, they have had all of the
logs and could not give much of an explanation, they were more interested in
the very very many attempts to start the server.

5) The O/S has no errors during that time.

6) The hardware had no issues at that time.

7) The SAN reported no faults.

8) I have exhausted every avenue to find a cause. (which is not
good enough, as I have to complete a report identifying the cause and
detailing the resolution...)

We are performing the restore again, we will hopefully have more
success this time, we are not too concerned if we can't get the logical logs
rolled forward. We have had to move to other platforms and invoke other
parts of our DR policies and procedures to cover the loss of functionality
we now have, and all critical business functions are operational... Some of
the nice to have functionality is still off-line but not impacting
operations.

I will attempt to get a clear statement as to why the restore
vanished, and more importantly for me why there were no messages relating to
any sort of failure when clearly it did fail.

Again, Thanks to all for all your advice, all of it sound in one form or
another ~ it is a credit to the Informix Community that there are so many
people willing to freely offer assistance.

Cheers,
Cliff.



-----Original Message-----
From: Alexey Sonkin [mailto:alexeis@grandvirtual.com]
Sent: Tuesday, 25 January 2005 8:00 AM
To: ids@iiug.org
Subject: RE: Any assistance greatly appreciated: [4092]

Norma,

This fragment of 'online.log' is about an attempt to start the engine after
restore.

This fragment is not generated during the restore itself

Just take a look at timings: is it possible to restore the 400-GB instance
in 12 seconds? :-)

To understand the problem, it is necessary to see the 100 or 1000 lines
preceding this fragment

Also, it might be useful to verify the status of the backup (also from
online.log, like:
04:52:45 Archive on rootdbs, physdbs, logsdbs, indexdbs, maindbs ABORTED.)
that could not be restored

-Alexey

> -----Original Message-----
> From: Sebastian, Norma J. [mailto:NormaJean.Sebastian@tellabs.com]
> Sent: Monday, January 24, 2005 2:49 PM
> To: Alexey Sonkin; ids@iiug.org
> Subject: RE: Any assistance greatly appreciated: [4089]
>
>
> From what he posted originally, it just goes to la-la land?
> Onlinelog:
>
> >> The output from the log file is: the last lines:
> >> 09:47:30 Could not create single shared memory segment with
> >> resident and non-re sident partitions. Proceeding to create 2
> >> shared memory
> segments instead.
> >> 09:47:32 Segment locked: addr=0xc1841000, size=683679744 Sat Jan
> >> 22 09:47:32 2005
> >> 09:47:32 Event alarms enabled. ALARMPROG =
> >> '/apps/informix_ids/etc/ifx_alert.sh'
> >> 09:47:38 DR: DRAUTO is 0 (Off)
> >> 09:47:39 Dynamically allocated new message shared memory segment
> >> (size
> 1104KB)
> >> 09:47:39 Informix Dynamic Server Version 7.31.UD1 Software Serial
> Number AAD #J216570
> >> 09:47:39 Informix Dynamic Server Initialized -- Shared Memory
Initialized.
> >> 09:47:39 Physical Recovery Started.
> >> 09:47:39 Physical Recovery Complete: 0 Pages Restored.
> >> 09:47:39 Logical Recovery Started.
> >> 09:47:42 Logical Recovery Complete.
> >> 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
> >>
> >> However:...(his original mail continues)...
>
>
> Clifford,
> You said " after a cold restore"....
> How did you do the restore, what was the command? Did you do physical
only,
> then followed by logical? Or all in the same shot?...
>
> I agree with others...
> Yes, maybe seeing some of your ixbar.X file AND the bar_activity.log
> file from the same timeframe would be good (perhaps there was a
> storage manager problem during that time?)?
>
> Also, do you have any o/s server/system messages during that timeframe
> that would help?
>
> Thanks,
> Norma Jean
>
>
>
>
> -----Original Message-----
> From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org] On
> Behalf Of Alexey Sonkin
> Sent: Monday, January 24, 2005 11:11 AM
> To: ids@iiug.org
> Subject: RE: Any assistance greatly appreciated: [4089]
>
> I was meaning 'online.log'
>
> Since the actual restore job (for both 'ontape' and 'onbar') is
> performed by the Informix engine itself, it produces the output to
> 'online.log', when the engine starts, when the restore starts, and
> when the restore completes/aborts, when the server switches to
> 'quiescent' mode after successful restore
>
> This is an example of what 'online.log' might contain in case of a
> failed
> restore:
>
> #######
> 23:19:13 Informix Dynamic Server 2000 Initialized -- Shared Memory
> Initialized.
> 23:19:13 Dataskip is now OFF for all dbspaces
> 23:19:13 Recovery Mode
> 23:19:18 Physical Restore of rootdbs, physdbs, logsdbs, indexdbs,
> maindbs started.
> ................
> 04:23:49 Aborting physical restore of ROOT DBSpace
> 04:23:49 Aborted by client.
> 04:23:49 Informix Dynamic Server 2000 Stopped.
> ########
>
> Corrupt restore caused exactly the same ' Checking database partition
> index...FAILED'
> message when 'oninit -v' was attempted after restore
>
> -Alexey
>
> > -----Original Message-----
> > From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org]
> > On Behalf Of Obnoxio The....
> > Sent: Sunday, January 23, 2005 3:20 AM
> > To: ids@iiug.org
> > Subject: RE: Any assistance greatly appreciated: [4086]
> >
> >
> > Burton, Cli.... said:
> > > The message log is of no use, it contains NO output from the
> > > onbar, it merely (as included in the original mail) has:
> >
> > I think he might mean the ix_bar.log file?
> >
> > > -----Original Message-----
> > > From: Alexey Sonkin [mailto:alexeis@grandvirtual.com]
> > > Sent: Sunday, 23 January 2005 3:03 PM
> > > To: Burton, Cli....; ids@iiug.org
> > > Subject: RE: Any assistance greatly appreciated: [4079]
> > >
> > > Burton,
> > >
> > > What version of Informix: 7.31ud1 (reported in the log) or 7.31ud4
> > > (that you report in the message) are you using?
> > > May be, both are installed on your machine?
> > >
> > > Note, that 7.31ud1 was a very bad version
> > >
> > > Can You send a fragment of 'online log' containing the 'onbar'
> > > restore output?
> > >
> > > Are You using KAIO on your HP box with Informix?
> > >
> > >
> > > ------------------------------------------
> > > Alexey Sonkin
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: forum.subscriber@iiug.org
> > >> [mailto:forum.subscriber@iiug.org]
> > >> On Behalf Of Burton, Cli....
> > >> Sent: Friday, January 21, 2005 6:52 PM
> > >> To: ids@iiug.org
> > >> Subject: Any assistance greatly appreciated: [4079]
> > >>
> > >>
> > >> Hi All,
> > >>
> > >> I have a situation where the IDS-731UD4 instance I run on HP-UX
> > >> 11 is failing after a cold restore.
> > >>
> > >> I do not have time to perform another restore without serious
> > >> business ramifications.
> > >>
> > >> The output from the log file is: the last lines:
> > >> 09:47:30 Could not create single shared memory segment with
> > >> resident
> > > and
> > >> non-re
> > >> sident partitions. Proceeding to create 2 shared memory segments
> > > instead.
> > >> 09:47:32 Segment locked: addr=0xc1841000, size=683679744
> > >>
> > >> Sat Jan 22 09:47:32 2005
> > >>
> > >> 09:47:32 Event alarms enabled. ALARMPROG =
> > >> '/apps/informix_ids/etc/ifx_alert.sh'
> > >> 09:47:38 DR: DRAUTO is 0 (Off)
> > >> 09:47:39 Dynamically allocated new message shared memory segment
> > > (size
> > >> 1104KB)
> > >>
> > >> 09:47:39 Informix Dynamic Server Version 7.31.UD1 Software Serial
> > >> Number
> > >> AAD
> > >> #J216570
> > >> 09:47:39 Informix Dynamic Server Initialized -- Shared Memory
> > >> Initialized.
> > >> 09:47:39 Physical Recovery Started.
> > >> 09:47:39 Physical Recovery Complete: 0 Pages Restored.
> > >> 09:47:39 Logical Recovery Started.
> > >> 09:47:42 Logical Recovery Complete.
> > >> 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
> > >>
> > >> However:
> > >>
> > >> informix@machinename$ onstat -
> > >> shared memory not initialized for INFORMIXSERVER 'newworld'
> > >>
> > >> The results of the oninit -v are:
> > >>
> > >> Checking group membership to determine server run modesucceeded
> > >> Reading configuration file
> > >> '/apps/informix_ids/etc/onconfig.newworld'...succeeded
> > >> Creating /INFORMIXTMP/.infxdirs ... succeeded Creating infos file
> > >> "/apps/informix_ids/etc/.infos.newworld" ...
> > >> "/apps/informix_ids/etc/.conf.newworld" ... succeeded Writing to
> > >> infos file ... succeeded Checking config parameters...succeeded
> > >> Allocating and attaching to shared memory...succeeded Creating
> > >> resident pool
> > >> 143252 kbytes...succeeded Creating buffer pool 524290
> > >> kbytes...succeeded Initializing rhead structure...succeeded
> > >> Initializing ASF ...succeeded Initializing Dictionary Cache and
> > >> Stored Procedure Cache...succeeded Bringing up ADM VP...succeeded
> > >> Creating VP classes...succeeded Onlining 4 additional cpu
> > >> vps...succeeded Onlining
> > >> 18 IO vps...succeeded Forking main_loop thread...succeeded
> > >> Initialzing DR structures...succeeded Forking 1 'ipcshm' listener
> > >> threads...succeeded Forking 2 'soctcp' listener
> > >> threads...succeeded Forking 1 'ipcshm' listener threads...succeeded
Forking 0 'soctcp'
> > >> listener threads...succeeded Forking 0 'soctcp' listener
> > >> threads...succeeded Starting tracing...succeeded Initializing 32
> > >> flushers...succeeded Initializing log/checkpoint
> > >> information...succeeded Opening primary chunks...succeeded
> > >> Opening mirror chunks...succeeded Initializing
> > >> dbspaces...succeeded Validating chunks...succeeded Initialize
> > >> Async Log Flusher...succeeded Forking btree cleaner...succeeded
> > >> Initializing DBSPACETEMP list Checking database partition
> > >> index...FAILED
> > >> oninit: Fatal error in shared memory initialization
> > >>
> > >>
> > >> I realise that it is unlikely that too many people are out there
> > >> right
> > > at
> > >> this moment, however, I have been working on this issue so long
> > >> now
> > > that I
> > >> am loosing track of time, Does anyone have any ideas as to how to
> > >> get
> > > this
> > >> online, I do have a vague recollection of an issue like this in
> > >> the
> > > past
> > >> that was resolved quite simply. I am attempting to contact IBM
> > >> Support
> > > on
> > >> the issue although it is a moderately complex task out of hours
> > >> when I cannot locate the support contracts etc...
> > >>
> > >>
> > >>
> > >> Clifford Burton
> > >> Senior Database Administrator
> > >> SIMplus Mobile Pty Ltd [ www.SIMplus.com.au
> > > <http://www.simplus.com.au/>
> > >> ]
> > >> Ph: (02) 9028 1716
> > >> Fx: (02) 9030 6284
> > >> Mb: 0414 381 381
> > >>
> > >> Em: Clifford_Burton@SIMplus.com.au
> > > <mailto:Clifford_Burton@SIMplus.com.au>
> > >>
> > >> SIMplus Mobile Pty Ltd ABN 37 098 145 218.
> > >> SIMplus is the trademark of SingTel Optus Ltd trading as Optus
> > >> Communications.
> > >>
> > >> ----------------------------------------------------
> > >>
> > >> This message is for the named person's use only.
> > >>
> > >> Privileged/confidential information may be contained in this message.
> > >> If you are not the addressee indicated in this message (or
> > >> responsible for delivery of the message to such person), you may
> > >> not copy or deliver this message to anyone. In such case, you
> > >> should destroy this message, and notify us immediately.
> > >>
> > >> Any views expressed in this message are those of the individual
> > >> sender, except where the message states otherwise and the sender
> > >> is authorised to state them to be the views of any such entity.
> > >> ----------------------------------------------------
> > >>
> > >
> >
> >
> > --
> >
> > Bye now,
> > Obnoxio
> >
> > "C'est pas parce qu'on n'a rien à dire qu'il faut fermer sa gueule"
> > - Coluche
> >
> > "I'm trying to see things your way, but I can't get my head up my ass"
> > - JCH
> >
> > "Ogni uomo mi guarda come se fossi una testa di cazzo"
> > - Marco
> >
> > If anyone told you to be yourself they would be giving you bad advice.
> >
> > I went to the airport to check in and they asked what I did because
> > I looked like a terrorist. I said I was a comedian. They said, "Say
> > something funny then." I told them I had just graduated from flying
> > school.
> >
> > -- Ahmed Ahmed
>
> ============================================================
> The information contained in this message may be privileged and
> confidential and protected from disclosure. If the reader of this
> message is not the intended recipient, or an employee or agent
> responsible for delivering this message to the intended recipient, you
> are hereby notified that any reproduction, dissemination or
> distribution of this communication is strictly prohibited. If you have
> received this communication in error, please notify us immediately by
> replying to the message and deleting it from your computer. Thank you.
> Tellabs ============================================================

Messages In This Thread

• Any assistance greatly appreciated:
  Burton, Clifford -- Friday, 21 January 2005, at 6:51 p.m.
  • Re: Any assistance greatly appreciated:
    chris hill -- Saturday, 22 January 2005, at 3:42 a.m.
  • RE: Any assistance greatly appreciated:
    Alexey Sonkin -- Saturday, 22 January 2005, at 11:07 p.m.
  • RE: Any assistance greatly appreciated:
    Burton, Clifford -- Saturday, 22 January 2005, at 11:24 p.m.
    • RE: Any assistance greatly appreciated:
      Obnoxio The Chav -- Sunday, 23 January 2005, at 3:19 a.m.
      • RE: Any assistance greatly appreciated:
        Alexey Sonkin -- Monday, 24 January 2005, at 12:10 p.m.
        • RE: Any assistance greatly appreciated:
          Sebastian, Norma J. -- Monday, 24 January 2005, at 2:54 p.m.
        • RE: Any assistance greatly appreciated:
          Alexey Sonkin -- Monday, 24 January 2005, at 3:59 p.m.
          • RE: Any assistance greatly appreciated:
            Burton, Clifford -- Monday, 24 January 2005, at 5:22 p.m.
        • RE: Any assistance greatly appreciated:
          Sebastian, Norma J. -- Monday, 24 January 2005, at 4:19 p.m.
    • RE: Any assistance greatly appreciated:
      Sebastian, Norma J. -- Monday, 24 January 2005, at 11:08 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.