Hello,
We finally upgraded from 7.32 to 9.40FC6. Some of our batch processes and most user OLTP are performing as well or better than the 732 version. However 3 of our large batch processes are now taking 30-40% longer to complete. For example a process that used to take 7 hrs is now taking 11.5 hrs. I am seeking some advice on what may be occurring. I have included our onconfig and an sqexplain for the process in question. One factor that seems strange in the sqexplain is the cost of the insert statement. I am unsure if that is just how sqexplain reports the data or if that is the issue.
We are on hp-ux 11.11 with a 4 way 8 gig N class server. IDS 64 bit engine and 32 bit tools/network. The program in question is 4gl. Machine notes were followed and kernel params slightly modified. The machine does not appear to be taxed, not paging, cpu ok, etc. No excessive check points and onstat -p, g ioq, g iof, g iov are OK.
We have run Art's dostats for the database in question and manually update stats for the system tables recommended by IBM. We did a dbexport and dbimport for the upgrade. We have rebuilt most of the indexes in question (but not all). PDQ is not enabled, no fragmentation scheme, did not specify detached indexes. (During the install we did have one major faux pas, we installed the 64 bit tools, 64 bit engine, 32 bit network. We realised afterwards that we grabbed the wrong tools cd (64 bit), and needed the 32 bit tools. Our application vendor and IDS reseller stated we could just re-install the 32 bit tools over the top w/o having to reinstall the engine and network.)
I want to rule out IDS and server issues before address the business rule set up or the vendor's code. So any advice or critique is fine.
Thanks in advance,
Doug
ROOTNAME rootdbs # Root dbspace name
ROOTPATH /dbms/links/rootdbs9 # Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 30000 # Size of root dbspace (Kbytes)
# Physical Log Configuration
PHYSDBS plogdbs94 # Location (dbspace) of physical log
PHYSFILE 127000 # Physical log file size (Kbytes)
# Logical Log Configuration
LOGFILES 101 # Number of logical log files
LOGSIZE 2000 # Logical log size (Kbytes)
TABLSPACE_STATS 0 # Maintain tblspace statistics
# System Configuration
SERVERNUM 1 # Unique id corresponding to a OnLine instance
DBSERVERNAME online9 # Name of default database server
DBSERVERALIASES test9 # List of alternate dbservernames
NETTYPE ipcshm,1,100,CPU
NETTYPE soctcp,2,100,NET
DEADLOCK_TIMEOUT 120 # Max time to wait of lock in distributed env.
RESIDENT 1 # Forced residency flag (Yes = 1, No = 0)
MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor
VPCLASS CPU,num=3,aff=1-3,noage
VPCLASS AIO,num=2,aff=1-3
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one
# Shared Memory Parameters
LOCKS 200000 # Maximum number of locks
BUFFERS 900000 # Maximum number of shared buffers
PHYSBUFF 64 # Physical log buffer size (Kbytes)
LOGBUFF 64 # Logical log buffer size (Kbytes)
CLEANERS 8 # Number of buffer cleaner processes
SHMBASE 0x0L # Shared memory base address
SHMVIRTSIZE 327680
SHMADD 32768 # Size of new shared memory segments (Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
LRUS 128 # Number of LRU queues
LRU_MAX_DIRTY 10.000000 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 5.000000 # LRU percent dirty end cleaning limit
TXTIMEOUT 0x12c # Transaction timeout (in sec)
STACKSIZE 64 # Stack size (Kbytes)
# DYNAMIC_LOGS:
DYNAMIC_LOGS 0
LTXHWM 40
LTXEHWM 50
# OFF_RECVRY_THREADS:
OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads
# Backup/Restore variables
BAR_ACT_LOG /dbms/informix9/log/bar_act.log
BAR_DEBUG_LOG /dbms/informix9/log/informix/bar_dbug.log
# ON-Bar Debug Log - not in /tmp please
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 10
BAR_XFER_BUF_SIZE 31
RESTARTABLE_RESTORE on
BAR_PROGRESS_FREQ 0
# Read Ahead Variables
RA_PAGES 32 # Number of pages to attempt to read ahead
RA_THRESHOLD 30 # Number of pages left before next group
DBSPACETEMP tempdbs6:tempdbs7:tempdbs8:tempdbs9:tempdbs10
FILLFACTOR 90 # Fill factor for building indexes
USEOSTIME 0 # 0: use internal time(fast), 1: get time from O
# Parallel Database Queries (pdq)
MAX_PDQPRIORITY 90 # Maximum allowed pdqpriority
DS_MAX_QUERIES # Maximum number of decision support queries
DS_TOTAL_MEMORY # Decision support memory (Kbytes)
DS_MAX_SCANS 1048576 # Maximum number of decision support scans
DATASKIP off
# OPTCOMPIND
OPTCOMPIND 1 # To hint the optimizer
DIRECTIVES 1 # Optimizer DIRECTIVES ON (1/Default) or OFF (0)
ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT,
# HETERO_COMMIT (Gateway participation in distributed transactions)
HETERO_COMMIT 0
SBSPACENAME # Default smartblob space name - this is where b
SYSSBSPACENAME # Default smartblob space for use by the Informi
BLOCKTIMEOUT 3600 # Default timeout for system block
SYSALARMPROGRAM /dbms/informix9/etc/evidence.sh # System Alarm program path
# Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
OPT_GOAL -1
ALLOW_NEWLINE 0 # embedded newlines(Yes = 1, No = 0 or anything
START OF PROCESS
QUERY:
------
delete from retwahst where empid = ?
Estimated Cost: 7
Estimated # of Rows Returned: 36
1) bsidba.retwahst: INDEX PATH
(1) Index Keys: empid (Serial, fragments: ALL)
Lower Index Filter: bsidba.retwahst.empid = '101289 '
QUERY:
------
select count ( * ) from hr_retirewa , hr_pe_mstr where id = ? and hr_pe_id = ? and ( currbeg <= ? or extractbeg is null or extractbeg = " " ) and ( currend >= ? or extractend is null or extractend = " " ) and ( retirestat = "A" or retirestat = "O" or retirestat = "G" or retirestat = "F" )
Estimated Cost: 4
Estimated # of Rows Returned: 1
1) bsi.hr_pe_mstr: INDEX PATH
(1) Index Keys: hr_pe_id (Key-Only) (Serial, fragments: ALL)
Lower Index Filter: bsi.hr_pe_mstr.hr_pe_id = '101289 '
2) bsidba.hr_retirewa: INDEX PATH
Filters: ((((bsidba.hr_retirewa.currend >= 05/01/2006 OR bsidba.hr_retirewa.extractend IS NULL ) OR bsidba.hr_retirewa.extractend = ) AND (((bsidba.hr_retirewa.retirestat = 'A' OR bsidba.hr_retirewa.retirestat = 'O' ) OR bsidba.hr_retirewa.retirestat = 'G' ) OR bsidba.hr_retirewa.retirestat = 'F' ) ) AND ((bsidba.hr_retirewa.currbeg <= 05/31/2006 OR bsidba.hr_retirewa.extractbeg IS NULL ) OR bsidba.hr_retirewa.extractbeg = ) )
(1) Index Keys: id currbeg currend (Serial, fragments: ALL)
Lower Index Filter: bsidba.hr_retirewa.id = '101289 '
NESTED LOOP JOIN
QUERY:
------
select retirestat , currbeg , currend , start_per , reportbeg , extractend , currend , currsys , currplan , currrate , reportbeg , reportend , hr_pe_mstr . * from hr_retirewa , hr_pe_mstr where id = ? and hr_pe_id = ? and ( currbeg <= ? or extractbeg is null or extractbeg = " " ) and ( currend >= ? or extractend is null or extractend = " " ) and ( retirestat = "A" or retirestat = "O" or retirestat = "G" or retirestat = "F" )
Estimated Cost: 7
Estimated # of Rows Returned: 1
1) bsi.hr_pe_mstr: INDEX PATH
(1) Index Keys: hr_pe_id (Serial, fragments: ALL)
Lower Index Filter: bsi.hr_pe_mstr.hr_pe_id = '101289 '
2) bsidba.hr_retirewa: INDEX PATH
Filters: ((((bsidba.hr_retirewa.currend >= 05/01/2006 OR bsidba.hr_retirewa.extractend IS NULL ) OR bsidba.hr_retirewa.extractend = ) AND (((bsidba.hr_retirewa.retirestat = 'A' OR bsidba.hr_retirewa.retirestat = 'O' ) OR bsidba.hr_retirewa.retirestat = 'G' ) OR bsidba.hr_retirewa.retirestat = 'F' ) ) AND ((bsidba.hr_retirewa.currbeg <= 05/31/2006 OR bsidba.hr_retirewa.extractbeg IS NULL ) OR bsidba.hr_retirewa.extractbeg = ) )
(1) Index Keys: id currbeg currend (Serial, fragments: ALL)
Lower Index Filter: bsidba.hr_retirewa.id = '101289 '
NESTED LOOP JOIN
QUERY:
------
select syscode from hr_rettblwa where system = ? and plan = ?
Estimated Cost: 2
Estimated # of Rows Returned: 1
1) bsidba.hr_rettblwa: SEQUENTIAL SCAN
Filters: (bsidba.hr_rettblwa.plan = 3 AND bsidba.hr_rettblwa.system = 'SRS ' )
QUERY:
------
select min ( extractbeg ) from hr_retirewa where id = ? and ( retirestat = "A" or retirestat = "O" or retirestat = "G" or retirestat = "F" )
Estimated Cost: 3
Estimated # of Rows Returned: 1
1) bsidba.hr_retirewa: INDEX PATH
Filters: (((bsidba.hr_retirewa.retirestat = 'A' OR bsidba.hr_retirewa.retirestat = 'O' ) OR bsidba.hr_retirewa.retirestat = 'G' ) OR bsidba.hr_retirewa.retirestat = 'F' )
(1) Index Keys: id currbeg currend (Serial, fragments: ALL)
Lower Index Filter: bsidba.hr_retirewa.id = '101289 '
QUERY:
------
select reportbeg , reportend , currbeg , currend , currsys , currplan , currrate , extractbeg , extractend , retirestat , ovrtype from hr_retirewa where id = ?
Estimated Cost: 3
Estimated # of Rows Returned: 1
1) bsidba.hr_retirewa: INDEX PATH
(1) Index Keys: id currbeg currend (Serial, fragments: ALL)
Lower Index Filter: bsidba.hr_retirewa.id = '101289 '
QUERY:
------
select drs_period [ 1 , 4 ] , drs_period [ 5 , 6 ] , earned_dol , adj_dollar , report_dol , report_hrs , drs_period from retwa_reported where ( empssn = ? and retwa_reported . warn <> "NoDf" ) order by drs_period
Estimated Cost: 1
Estimated # of Rows Returned: 1
Temporary Files Required For: Order By
1) bsidba.retwa_reported: SEQUENTIAL SCAN
Filters: (bsidba.retwa_reported.empssn = 'xxxxxxxx ' AND bsidba.retwa_reported.warn != 'NoDf' )
QUERY:
------
select * from retwa_b where recstatus = "FC" and ssn = ?
Estimated Cost: 22
Estimated # of Rows Returned: 80
1) bsidba.retwa_b: INDEX PATH
Filters: bsidba.retwa_b.recstatus = 'FC'
(1) Index Keys: ssn (Serial, fragments: ALL)
Lower Index Filter: bsidba.retwa_b.ssn = 'xxxxxxxx '
QUERY:
------
select pyt_per_cc, py_batch_name, pyt_date01, pyt_hrs_no01, pyt_hrs01, pyt_rt01, pyt_amt01, type, statuscd, system, plan, deferflag, py_per_check_dt, py_per_end, RetirePB, RetireHB, pyt_num_cd from pyt_hrs_dtl, cdhtmp25001, py_per_mstr where hr_pe_id = ? and (py_batch_name like 'SYSTM%' or py_batch_name like 'DRS%') and (pyt_status = 'DS' or pyt_status = 'DM' or pyt_status = 'DT') and (pyt_date01 <= ?) and (pyt_date01 >= ?) and pyt_hrs_no01= cdhtmp25001.CdhNo and ((cdhtmp25001.RetirePB > ' ' and cdhtmp25001.RetirePB is not NULL) or (cdhtmp25001.RetireHB > ' ' and cdhtmp25001.RetireHB is not NULL)) and (cdhtmp25001.Type <> 'P') and ((py_per_check_dt >= cdhtmp25001.dtBeg or cdhtmp25001.dtBeg is NULL or cdhtmp25001.dtBeg = ' ') and (py_per_check_dt <= cdhtmp25001.dtEnd or cdhtmp25001.dtEnd is NULL or cdhtmp25001.dtEnd = ' ')) and py_per_cc = pyt_per_cc
Estimated Cost: 306
Estimated # of Rows Returned: 1
1) bsi.pyt_hrs_dtl: INDEX PATH
Filters: ((((bsi.pyt_hrs_dtl.py_batch_name LIKE 'SYSTM%' OR bsi.pyt_hrs_dtl.py_batch_name LIKE 'DRS%' ) AND bsi.pyt_hrs_dtl.pyt_date01 >= 02/01/2001 ) AND bsi.pyt_hrs_dtl.pyt_date01 <= 05/31/2006 ) AND ((bsi.pyt_hrs_dtl.pyt_status = 'DS' OR bsi.pyt_hrs_dtl.pyt_status = 'DM' ) OR bsi.pyt_hrs_dtl.pyt_status = 'DT' ) )
(1) Index Keys: hr_pe_id (Serial, fragments: ALL)
Lower Index Filter: bsi.pyt_hrs_dtl.hr_pe_id = '101289 '
2) dougf.cdhtmp25001: INDEX PATH
Filters: ((dougf.cdhtmp25001.retirepb > ' ' AND dougf.cdhtmp25001.retirepb IS NOT NULL ) OR (dougf.cdhtmp25001.retirehb > ' ' AND dougf.cdhtmp25001.retirehb IS NOT NULL ) )
(1) Index Keys: cdhno type retirepb retirehb dtbeg dtend (Key-First) (Serial, fragments: ALL)
Lower Index Filter: bsi.pyt_hrs_dtl.pyt_hrs_no01 = dougf.cdhtmp25001.cdhno
Key-First Filters: (dougf.cdhtmp25001.type != 'P' )
NESTED LOOP JOIN
3) bsi.py_per_mstr: INDEX PATH
Filters: (((bsi.py_per_mstr.py_per_check_dt <= dougf.cdhtmp25001.dtend OR dougf.cdhtmp25001.dtend IS NULL ) OR dougf.cdhtmp25001.dtend = ) AND ((bsi.py_per_mstr.py_per_check_dt >= dougf.cdhtmp25001.dtbeg OR dougf.cdhtmp25001.dtbeg IS NULL ) OR dougf.cdhtmp25001.dtbeg = ) )
(1) Index Keys: py_per_cc (Serial, fragments: ALL)
Lower Index Filter: bsi.py_per_mstr.py_per_cc = bsi.pyt_hrs_dtl.pyt_per_cc
NESTED LOOP JOIN
..SNIP
REPEATS 25 times for array
...SNIP
QUERY:
------
select pyh_per_cc, pyh_no02, pyh_amt02,pyh_ck_dt, py_per_end, type, system, plan, statuscd, py_per_beg, RetirePB, RetireHB, DeferFlag from pyh_hst_dtl, cdhtmp25001, py_per_mstr where pyh_hst_dtl.hr_pe_id = ? and pyh_hst_dtl.pyh_no02 = cdhtmp25001.CdhNo and ((cdhtmp25001.RetirePB > ' ' and cdhtmp25001.RetirePB is not NULL)) and ((pyh_ck_dt >= cdhtmp25001.dtBeg or cdhtmp25001.dtBeg is NULL or cdhtmp25001.dtBeg = ' ') and (pyh_ck_dt <= cdhtmp25001.dtEnd or cdhtmp25001.dtEnd is NULL or cdhtmp25001.dtEnd = ' ')) and (cdhtmp25001.Type <> 'E') and py_per_beg <= ? and py_per_end >= ? and pyh_per_cc = py_per_cc and (pyh_hst_dtl.pyh_ck_note in ('DP','DT') or (pyh_hst_dtl.pyh_ck_note in ('WP','WT') and pyh_per_cc = ?))
Estimated Cost: 87
Estimated # of Rows Returned: 1
1) dougf.cdhtmp25001: SEQUENTIAL SCAN
Filters: ((dougf.cdhtmp25001.retirepb > ' ' AND dougf.cdhtmp25001.type != 'E' ) AND dougf.cdhtmp25001.retirepb IS NOT NULL )
2) bsi.pyh_hst_dtl: INDEX PATH
(1) Index Keys: hr_pe_id (Serial, fragments: ALL)
Lower Index Filter: bsi.pyh_hst_dtl.hr_pe_id = '101289 '
DYNAMIC HASH JOIN
Dynamic Hash Filters: bsi.pyh_hst_dtl.pyh_no02 = dougf.cdhtmp25001.cdhno
Other Join Filters: ((((bsi.pyh_hst_dtl.pyh_ck_dt >= dougf.cdhtmp25001.dtbeg OR dougf.cdhtmp25001.dtbeg IS NULL ) OR dougf.cdhtmp25001.dtbeg = ) AND ((bsi.pyh_hst_dtl.pyh_ck_dt <= dougf.cdhtmp25001.dtend OR dougf.cdhtmp25001.dtend IS NULL ) OR dougf.cdhtmp25001.dtend = ) ) AND (bsi.pyh_hst_dtl.pyh_ck_note IN ('DP' , 'DT' )OR (bsi.pyh_hst_dtl.pyh_ck_note IN ('WP' , 'WT' )AND bsi.pyh_hst_dtl.pyh_per_cc = 601105 ) ) )
3) bsi.py_per_mstr: INDEX PATH
Filters: (bsi.py_per_mstr.py_per_beg <= 05/31/2006 AND bsi.py_per_mstr.py_per_end >= 02/01/2001 )
(1) Index Keys: py_per_cc (Serial, fragments: ALL)
Lower Index Filter: bsi.pyh_hst_dtl.pyh_per_cc = bsi.py_per_mstr.py_per_cc
NESTED LOOP JOIN
..snip
REPEATS 10 times for array
...snip
QUERY:
------
select unique earningper [ 1 , 4 ] , earningper [ 5 , 6 ] , systemcode , plancode , statuscode , typecd from retwa_b where ( recstatus = "FC" ) and ( ssn = ? )
Estimated Cost: 22
Estimated # of Rows Returned: 40
1) bsidba.retwa_b: INDEX PATH
Filters: bsidba.retwa_b.recstatus = 'FC'
(1) Index Keys: ssn (Serial, fragments: ALL)
Lower Index Filter: bsidba.retwa_b.ssn = 'xxxxxxxxx '
QUERY:
------
select * from retwaadj,hr_pe_mstr where ssn = hr_pe_ssn and hr_pe_id = ?
Estimated Cost: 1
Estimated # of Rows Returned: 1
1) bsidba.retwaadj: SEQUENTIAL SCAN
2) bsi.hr_pe_mstr: INDEX PATH
(1) Index Keys: hr_pe_ssn hr_pe_id (Serial, fragments: ALL)
Lower Index Filter: (bsidba.retwaadj.ssn = bsi.hr_pe_mstr.hr_pe_ssn AND bsi.hr_pe_mstr.hr_pe_id = '101289 ' )
NESTED LOOP JOIN
QUERY:
------
select * from retwaadj,hr_pe_mstr where ssn = hr_pe_ssn and hr_pe_id = ?
Estimated Cost: 1
Estimated # of Rows Returned: 1
1) bsidba.retwaadj: SEQUENTIAL SCAN
2) bsi.hr_pe_mstr: INDEX PATH
(1) Index Keys: hr_pe_ssn hr_pe_id (Serial, fragments: ALL)
Lower Index Filter: (bsidba.retwaadj.ssn = bsi.hr_pe_mstr.hr_pe_ssn AND bsi.hr_pe_mstr.hr_pe_id = '101289 ' )
NESTED LOOP JOIN
QUERY:
------
insert into retwahst values ( ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? )
Estimated Cost: 18598
Estimated # of Rows Returned: 163966
...SNIP REPEATS 38 times for all data
Process then continues for the next ID (9000 total id's)
