IDS Forum

RE: Poor performance on replacement server

Posted By: Art S. Kagel
Date: Tuesday, 13 December 2005, at 2:56 p.m.

In Response To: Poor performance on replacement server (Simon Coyne)

Simon:

Hi, see below:

Art S. Kagel

> ----- Original Message -----
> From: Simon Coyne <simon.coyne@dla.com>
> At: 12/13 11:42
>
> Dear Informix users
>
> The firm I work for have recently purchased a 4 processor ES45 Alpha server to
> run our practice management (sales ledger) system. This is replacing a 2
> processor DS25 Alpha server (which struggles at period end). Outline
> specification is
>
> 4x1.2Ghz Alpha processors
> 8Gb RAM
> 5304 4 Channel Ultra SCSI controller
> 2xMA300(I think) disk cages populated with 36Gb disks and configured as RAID 1+0
> for both OS and Informix raw chunks.

Be careful. I recommend no more than 4 spindles per controller channel. It
only takes 5-6 drives to exceed the controllers throughput capability. I
also recommend that you set up separate RAID10 structures for the OS and for
IDS.

> Tru 64 5.1B
> Informix 9.3.FC3
>
> The instance is about 80Gb total.
>
> Unfortunately, we are experiencing crippling performance problems. When we get
> to about 55 green screen users on the system, performance within Informix
> becomes terrible. The 3 CPU VPs go to 100% and disk I/O drops to almost 0 across

Actually the ONCONFIG file below shows only 2 CPU VPs. Are those users
accessing the DB locally using shared memory connections?

> every one of the Informix disks. On occassion, when CPU utilisation dips below
> ~85%, disk I/O goes back to normal levels and performance is great. At all

Someone suggested that lock contention is your problem. I would definitely
not rule that out. You say a 'green screen' application is using the DB?
Is it possible that the app locks records while the users is modifying the
coy on screen? This is a common DBMS application problem. Apps that access
RDBMS's for interactive users input systems should use optimistic locking
techniques (ie don't lock the row until the users finishing modifying it,
then lock the row, compare the original version to the current DB contents,
if no changes perform the update and commit, if the record has been changed
since the user fetched it, reject the changes and make the user try again.)

> times, the disk performance for O/S based operations is more than acceptable.
> Informix performance is almost binary; it's either working or stopped. There
> doesn't seem to be an inbetween position where performance starts degrading.
>
> Has anyone seen anything like this at all or have any helpful suggestions?
> (Update statistics runs daily.)
>
> Thanks in advance
>
>
> Simon Coyne
> Infrastructure Specialist
> DLA Piper Rudnick Gray Cary LLP
>
> ONconfig follows
>
> #**************************************************************************
> #
> # INFORMIX SOFTWARE, INC.
> #
> # Title: onconfig.std
> # Description: Informix Dynamic Server 2000 Configuration Parameters
> #
> #**************************************************************************
>
> # Root Dbspace Configuration
>
> ROOTNAME rootdbs # Root dbspace name
> ROOTPATH /dev/rootdbs # Path for device containing root dbspace
> ROOTOFFSET 500 # Offset of root dbspace into device (Kbytes)
> ROOTSIZE 2000000 # Size of root dbspace (Kbytes)
>
> # Disk Mirroring Configuration Parameters
>
> MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
> MIRRORPATH # Path for device containing mirrored root
> MIRROROFFSET 0 # Offset into mirrored device (Kbytes)
>
> # Physical Log Configuration
>
> PHYSDBS physdbs # Location (dbspace) of physical log
> PHYSFILE 1000000 # Physical log file size (Kbytes)
>
> # Logical Log Configuration
>
> LOGFILES 1034 # Number of logical log files
> LOGSIZE 1500 # Logical log size (Kbytes)
>
> # Diagnostics
>
> MSGPATH /usr/informix/online.log # System message log file path
> CONSOLE /usr/informix/console.log # System console message path
> ALARMPROGRAM /usr/informix/etc/log_full.sh # Alarm program path
> TBLSPACE_STATS 1 # Maintain tblspace statistics
>
> # System Archive Tape Device
>
> TAPEDEV /devices/tape/tape17c # Tape device path
> #TAPEDEV /devices/tape/tape16c # FS Backup device
> #TAPEDEV /dev/null # Black Hole
> #TAPEDEV /nfsdump/aristadbbackup.img
> TAPEBLK 1024 # Tape block size (Kbytes)
> TAPESIZE 85000000 # Maximum amount of data to put on tape (Kbytes)
>
> # Log Archive Tape Device
>
> LTAPEDEV /devices/tape/tape16
> LTAPEBLK 1024 # Log tape block size (Kbytes)
> LTAPESIZE 35000000 # Max amount of data to put on log tape (Kbytes)
>
> # Optical
>
> STAGEBLOB # Informix Dynamic Server 2000 staging area
>
> # System Configuration
>
> SERVERNUM 0 # Unique id corresponding to a OnLine instance
> DBSERVERNAME arista # Name of default database server
> DBSERVERALIASES arista_tcp # List of alternate dbservernames
> DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
> RESIDENT 0 # Forced residency flag (Yes = 1, No = 0)

I don't remember any problems using RESIDENT with tru64, so I'm going to
suggest you change this to 1 (resident segment locked in memory) or -1
(initial virtual segment locked into memory as well as the resident
segment). You have plenty of RAM, use it. This can be a huge performance
gain on a busy system.

> MULTIPROCESSOR 0 # 0 for single-processor, 1 for multi-processor

You have 4 CPUs and you set CPUVPs >1, MULTIPROCESSOR must be 1 as well to
properly implement resource control.

> NUMCPUVPS 2 # Number of user (cpu) vps

Don't be squeemish, set this to at least 4. You have 4 very fast CPUs, use
them. Processors faster than about 400MHZ (~600MHZ for Intel) can support
more than one CPU VP per physical CPU according to a round of user testing
several years ago.

> SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one
>
> NOAGE 0 # Process aging

Check you release notes to see if this is supported on tru64. I don't know
how TRU64 works but many UNIXes, especially Solaris and HPUX, are agressive
about lowering the runtime priority of long running processes, and you
oninits are normally running for a VERY LONG time before you restart them.
Especially compared to user tasks that come and go in minutes. So, best if
NOAGE is available on your platform to set it to 1.

> # NRW changed proc affinity to second cpu to help system performance
> AFF_SPROC 0 # Affinity start processor
> AFF_NPROCS 0 # Affinity number of processors
>
> # Shared Memory Parameters
>
> LOCKS 1000000 # Maximum number of locks

> BUFFERS 500000 # Maximum number of shared buffers
> NUMAIOVPS 16 # Number of IO vps

If all of your chunks are RAW and tru64 supports it you are using KAIO for
chunk IO, so you typically need from 2-6 AIO VPs. Check onstat -g iov, if
there are kaio threads reported for each CPU VP, then you should look to
reduce NUMAIOPVS. On the same report, any aio vps with an io/wup value of
0.0 is completely unused and you can safely reduce NUMAIOVPS by the number
of such.

> PHYSBUFF 864 # Physical log buffer size (Kbytes)

This is a bit large. A larger PHYSUFF will delay physically writing
physical log pages to disk and will slow down checkpointing. A PHYSBUFF of
32 is usually fine. Use 64 if you actually have very large row sizes.

> LOGBUFF 38 # Logical log buffer size (Kbytes)
> LOGSMAX 2048 # Maximum number of logical log files
> CLEANERS 16 # Number of buffer cleaner processes

CLEANERS should be >= LRUS and you have LRUS set to 128 below. Set CLEANERS
to 128 as well to improve LRU flush performance.

> SHMBASE 0x200000000 # Shared memory base address
> SHMVIRTSIZE 753644 # initial virtual shared memory segment size
> SHMADD 75366 # Size of new shared memory segments (Kbytes)
> SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
> CKPTINTVL 300 # Check point interval (in sec)
> LRUS 128 # Number of LRU queues
> LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning limit
> LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit
> LTXHWM 50 # Long transaction high water mark percentage
> LTXEHWM 60 # Long transaction high water mark (exclusive)
> TXTIMEOUT 0x12c # Transaction timeout (in sec)
> STACKSIZE 32 # Stack size (Kbytes)
<SNIP>

> # Read Ahead Variables
> RA_PAGES 32 # Number of pages to attempt to read ahead
> RA_THRESHOLD 28 # Number of pages left before next group

For a modern RAID10 disk farm, these values are a bit high, especially the
RA_THRESHOLD. Try 32 & 16 at first, but I'd recommend moving toward 16 & 8
or 32 & 8 unless you don't have much cache at the controller/array level.
You can check the RA Utilization Ratio to see if this is hurting you.

> # DBSPACETEMP:
> # OnLine equivalent of DBTEMP for SE. This is the list of dbspaces
> # that the OnLine SQL Engine will use to create temp tables etc.
> # If specified it must be a colon separated list of dbspaces that exist
> # when the OnLine system is brought online. If not specified, or if
> # all dbspaces specified are invalid, various ad hoc queries will create
> # temporary files in /tmp instead.
>
> DBSPACETEMP tempdbs,tempdbs01 # Default temp dbspaces

Three or more temp spaces for max performance if you are doing any
significant data sorting or grouping. This is due to the way that the sort
algorithm merges sort-work files.

<SNIP>> # method for OnLine to use when determining current time
> USEOSTIME 0 # 0: use internal time(fast), 1: get time from
> OS(slow)

You can set this to 1 if yo like to get sub-second DATETIME resolution. The
overhead is minimal.

<SNIP>

> LBU_PRESERVE 0 # Preserve last log for log backup

This one is a must. Set LBU_PRESERVE to 1, it may save your butt if you
logical logs ever fill (then you shutdown, reset LBU_PRESERVE to zero,
restart, and archive the logs quickly). Not neccessary after 9.40, but in
9.30 and earlier it's required.

<SNIP>

> NETTYPE ipcshm,4,100,CPU # Configure poll thread(s) for nettype

Here you specified that you want 4 CPU VP listeners, but you only have 2 CPU
VPs configured. Therefore the engine is starting up 2 NET VPs to take up
the slack and NET VPs listening for shared memory work burn CPU cycles for
no reason. If you update NUMCPUVPS to 3 or more set the ipcshm NETTYPE to
the same number of CPU listeners, if you keep 2 PCU VPs for now, decrease
this to 2 and increase the connection count if you need to support 400
concurrent connections over shared memory.

> NETTYPE soctcp,4,100,NET # Configure poll thread(s) for nettype

Someone suggested looking a bunch of onstats, you might want to post the
following here for us to look at:

onstat -p
onstat -P
onstat -d
onstat -D
onstat -k
onstat -g glo
onstat -g iov
onstat -g iof
onstat -g act
onstat -g wai

Art S. Kagel

Messages In This Thread

• Poor performance on replacement server
  Simon Coyne -- Tuesday, 13 December 2005, at 11:03 a.m.
  • Re: Poor performance on replacement server
    John Bejarano -- Tuesday, 13 December 2005, at 12:03 p.m.
    • RE: Poor performance on replacement server
      Phil Torkington -- Wednesday, 14 December 2005, at 7:43 a.m.
  • Re: Poor performance on replacement server
    Ravi T -- Tuesday, 13 December 2005, at 1:30 p.m.
  • RE: Poor performance on replacement server
    Rafael Padilla V -- Tuesday, 13 December 2005, at 2:01 p.m.
  • RE: Poor performance on replacement server
    Art S. Kagel -- Tuesday, 13 December 2005, at 2:56 p.m.
  • RE: Poor performance on replacement server
    simon@coynes.eclipse.co.uk -- Wednesday, 14 December 2005, at 4:59 a.m.
    • RE: Poor performance on replacement server
      ART KAGEL, BLOOMBERG/ 731 LEXIN -- Wednesday, 14 December 2005, at 12:39 p.m.
    • RE: Poor performance on replacement server
      Ravi T -- Wednesday, 14 December 2005, at 12:50 p.m.
    • Re: RE: Poor performance on replacement server
      DARKO KRSTIC -- Wednesday, 14 December 2005, at 1:59 p.m.
    • RE: Poor performance on replacement server
      Simon -- Wednesday, 14 December 2005, at 7:00 p.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.