 |
IDS Forum
Re: RE: Query works fine in IDS 10 but crashes....
Posted By: Art Kagel
Date: Thursday, 12 August 2010, at 1:33 p.m.
In Response To: RE: RE: Query works fine in IDS 10 but crashes.... (Wyza, Jonathon)
Cost can only be compared for the identical query with optimizer directives,
data distribution levels, or indexes available changed. They mean nothing
in absolute terms. I've seen queries with costs in the millions that ran in
milliseconds and ones with costs in the hundred that ran for man seconds.
To me, the fact that removing 12 or 15 tables from the query allowed it to
complete in real-time points back to the OPTIMIZATION HIGH versus LOW issue.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and
do not reflect on my employer, Advanced DataTools, the IIUG, nor any other
organization with which I am associated either explicitly, implicitly, or by
inference. Neither do those opinions reflect those of other individuals
affiliated with any entity with which I am affiliated nor those of the
entities themselves.
On Thu, Aug 12, 2010 at 1:21 PM, Wyza, Jonathon <wyzaj@bethelcollege.edu>wrote:
> I did manage to get it to run by eliminate some of the joins. The cost
> seems
> high for it, see below:
>
> QUERY: (OPTIMIZATION TIMESTAMP: 08-12-2010 13:18:23)
> ------
> select id_rec.id
>
> {
>
> id_rec.fullname,
>
> t_clinicals.crs_no,
>
> t_clinicals.sec,
>
> prog_enr_rec.cat,
>
> stu_acad_rec.sess,
>
> stu_acad_rec.yr,
>
> stu_acad_rec.major1,
>
> major_table.txt major_text,
>
> health_rec.physical,
>
> health_rec.sport_phys physical_date,
>
> health_rec.dr_name,
>
> t_cpr.due_date cpr_due,
>
> t_license.due_date license_due,
>
> t_handbook.handbook_completed,
>
> t_healthpr.healthpr_completed,
>
> t_crimhist.crimhist_completed,
>
> case when t_tetnus.stat = "W" then MDY(01,01,2100)
>
> else t_tetnus.immune_date
>
> end last_td_date,
>
> t_tb.immune_date last_tb_date,
>
> t_cxr.immune_date last_cxr_date,
>
> t_rube.immune_date rube_date,
>
> t_rub1.immune_date rub1_date,
>
> t_rub2.immune_date rub2_date,
>
> t_rubt.immune_date rubt_date,
>
> t_meas.immune_date meas_date,
>
> t_mumps.immune_date mumps_date,
>
> case when t_mmr.stat = "W" then MDY(01,01,2100)
>
> else t_mmr.immune_date
>
> end mmr_date,
>
> t_mmr1.immune_date mmr1_date,
>
> t_mmr2.immune_date mmr2_date,
>
> t_varv.immune_date varv_date,
>
> t_vard.immune_date vard_date,
>
> t_vart.immune_date vart_date,
>
> t_hep.immune_date hpbt_date,
>
> t_hep1.immune_date hpb1_date,
>
> t_hep2.immune_date hpb2_date,
>
> t_hep3.immune_date hpb3_date,
>
> today add_date,
>
> nvl(t_has_con.id,0) has_con
>
> }
> from id_rec,
>
> outer t_clinicals,
>
> stu_acad_rec,
>
> prog_enr_rec,
>
> outer t_has_con,
>
> major_table,
>
> outer health_rec,
>
> outer ctc_rec t_cpr,
>
> outer ctc_rec t_license,
>
> outer t_handbook,
>
> outer t_healthpr,
>
> outer t_crimhist ,
>
> outer immune_rec t_tetnus,
>
> outer immune_rec t_tb,
>
> outer immune_rec t_cxr,
>
> outer immune_rec t_rube,
>
> outer immune_rec t_rub1,
>
> outer immune_rec t_rub2,
>
> outer immune_rec t_rubt,
>
> outer immune_rec t_meas{,
>
> outer immune_rec t_mumps,
>
> outer immune_rec t_mmr,
>
> outer immune_rec t_mmr1,
>
> outer immune_rec t_mmr2,
>
> outer immune_rec t_varv,
>
> outer immune_rec t_vard,
>
> outer immune_rec t_vart,
>
> outer immune_rec t_hep,
>
> outer immune_rec t_hep1,
>
> outer immune_rec t_hep2,
>
> outer immune_rec t_hep3
>
> }
> where stu_acad_rec.sess = "FA"
>
> and stu_acad_rec.yr = 2010
>
> and stu_acad_rec.id = t_clinicals.id
>
> and stu_acad_rec.id = t_has_con.id
>
> and stu_acad_rec.major1 = major_table.major
>
> and stu_acad_rec.reg_stat = "C"
>
> and stu_acad_rec.id = id_rec.id
>
> and (stu_acad_rec.major1 = "47A" or "47A" = " ")
>
> and prog_enr_rec.id = stu_acad_rec.id
>
> and prog_enr_rec.prog = stu_acad_rec.prog
>
> and (major_table.dept = "NUR" or major_table.major = "305")
>
> and major_table.major[1,2] !="50"
>
> and health_rec.id = id_rec.id
>
> and t_cpr.id = id_rec.id
>
> and t_cpr.tick = "NUR"
>
> and t_cpr.resrc = "CPRCERT"
>
> and t_cpr.stat = "E"
>
> and t_license.id = id_rec.id
>
> and t_license.tick = "NUR"
>
> and t_license.resrc = "LICENSE"
>
> and t_license.stat = "E"
>
> and t_handbook.id = id_rec.id
>
> and t_healthpr.id = id_rec.id
>
> and t_crimhist.id = id_rec.id
>
> and t_tetnus.id = id_rec.id
>
> and t_tetnus.immune = "TD"
>
> and t_tetnus.stat in ("A", "W")
>
> and t_tb.id = id_rec.id
>
> and t_tb.immune = "TB"
>
> and t_tb.stat in ("A", "W")
>
> and t_cxr.id = id_rec.id
>
> and t_cxr.immune = "CXR"
>
> and t_cxr.stat in ("A", "W")
>
> and t_rube.id = id_rec.id
>
> and t_rube.immune = "RUBE"
>
> and t_rube.stat in ("A", "W")
>
> and t_rub1.id = id_rec.id
>
> and t_rub1.immune = "RUB1"
>
> and t_rub1.stat in ("A", "W")
>
> and t_rub2.id = id_rec.id
>
> and t_rub2.immune = "RUB2"
>
> and t_rub2.stat in ("A", "W")
>
> and t_rubt.id = id_rec.id
>
> and t_rubt.immune = "RUBT"
>
> and t_rubt.stat in ("A", "W")
>
> and t_meas.id = id_rec.id
>
> and t_meas.immune = "MEAS"
>
> and t_meas.stat in ("A", "W")
>
> {
>
> and t_mumps.id = id_rec.id
>
> and t_mumps.immune = "MUMPS"
>
> and t_mumps.stat in ("A", "W")
>
> and t_mmr.id = id_rec.id
>
> and t_mmr.immune = "MMR"
>
> and t_mmr.stat in ("A", "W")
>
> and t_mmr1.id = id_rec.id
>
> and t_mmr1.immune = "MMR1"
>
> and t_mmr1.stat in ("A", "W")
>
> and t_mmr2.id = id_rec.id
>
> and t_mmr2.immune = "MMR2"
>
> and t_mmr2.stat in ("A", "W")
>
> and t_varv.id = id_rec.id
>
> and t_varv.immune = "VARV"
>
> and t_varv.stat in ("A", "W")
>
> and t_vard.id = id_rec.id
>
> and t_vard.immune = "VARD"
>
> and t_vard.stat in ("A", "W")
>
> and t_vart.id = id_rec.id
>
> and t_vart.immune = "VART"
>
> and t_vart.stat in ("A", "W")
>
> and t_hep.id = id_rec.id
>
> and t_hep.immune = "HPBT"
>
> and t_hep.stat in ("A", "W")
>
> and t_hep1.id = id_rec.id
>
> and t_hep1.immune = "HPB1"
>
> and t_hep1.stat in ("A", "W")
>
> and t_hep2.id = id_rec.id
>
> and t_hep2.immune = "HPB2"
>
> and t_hep2.stat in ("A", "W")
>
> and t_hep3.id = id_rec.id
>
> and t_hep3.immune = "HPB3"
>
> and t_hep3.stat in ("A", "W")
>
> }
>
> Estimated Cost: 1775
> Estimated # of Rows Returned: 64
>
> 1) informix.major_table: INDEX PATH
>
> (1) Index Name: informix.tmajor_major
>
> Index Keys: major (Key-First) (Serial, fragments: ALL)
>
> Lower Index Filter: informix.major_table.major = '47A'
>
> PostIndex Filter:informix.major_table.dept = 'NUR'
>
> Index Key Filters: (informix.major_table.major[1,2] != '50' )
>
> (2) Index Name: informix.tmajor_major
>
> Index Keys: major (Key-First) (Serial, fragments: ALL)
>
> Lower Index Filter: informix.major_table.major = '305'
>
> PostIndex Filter:informix.major_table.major = '47A'
>
> Index Key Filters: (informix.major_table.major[1,2] != '50' )
>
> 2) informix.stu_acad_rec: INDEX PATH
>
> Filters: ((informix.stu_acad_rec.major1 = '47A' AND
> informix.stu_acad_rec.reg_stat = 'C' ) AND
> informix.stu_acad_rec.major1[1,2]
> != '50' )
>
> (1) Index Name: informix.stuac_key2
>
> Index Keys: yr sess id (Serial, fragments: ALL)
>
> Lower Index Filter: (informix.stu_acad_rec.yr = 2010 AND
> informix.stu_acad_rec.sess = 'FA' )
> NESTED LOOP JOIN
>
> 3) informix.prog_enr_rec: INDEX PATH
>
> (1) Index Name: informix.prog_prim
>
> Index Keys: id prog site (Key-Only) (Serial, fragments: ALL)
>
> Lower Index Filter: (informix.prog_enr_rec.id = informix.stu_acad_rec.idAND
> informix.prog_enr_rec.prog = informix.stu_acad_rec.prog )
> NESTED LOOP JOIN
>
> 4) informix.id_rec: INDEX PATH
>
> (1) Index Name: informix. 311_590
>
> Index Keys: id (Key-Only) (Serial, fragments: ALL)
>
> Lower Index Filter: informix.stu_acad_rec.id = informix.id_rec.id
> NESTED LOOP JOIN
>
> 5) wyzaj.t_meas: INDEX PATH
>
> Filters: wyzaj.t_meas.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_meas.id = informix.id_rec.id AND
> wyzaj.t_meas.immune = 'MEAS' )
> NESTED LOOP JOIN
>
> 6) wyzaj.t_has_con: AUTOINDEX PATH
>
> (1) Index Name: (Auto Index)
>
> Index Keys: id (Key-Only)
>
> Lower Index Filter: informix.stu_acad_rec.id = wyzaj.t_has_con.id
> NESTED LOOP JOIN
>
> 7) wyzaj.t_rub1: INDEX PATH
>
> Filters: wyzaj.t_rub1.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_rub1.id = informix.id_rec.id AND
> wyzaj.t_rub1.immune = 'RUB1' )
> NESTED LOOP JOIN
>
> 8) wyzaj.t_cxr: INDEX PATH
>
> Filters: wyzaj.t_cxr.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_cxr.id = informix.id_rec.id AND
> wyzaj.t_cxr.immune = 'CXR' )
> NESTED LOOP JOIN
>
> 9) wyzaj.t_rubt: INDEX PATH
>
> Filters: wyzaj.t_rubt.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_rubt.id = informix.id_rec.id AND
> wyzaj.t_rubt.immune = 'RUBT' )
> NESTED LOOP JOIN
>
> 10) wyzaj.t_rube: INDEX PATH
>
> Filters: wyzaj.t_rube.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_rube.id = informix.id_rec.id AND
> wyzaj.t_rube.immune = 'RUBE' )
> NESTED LOOP JOIN
>
> 11) wyzaj.t_rub2: INDEX PATH
>
> Filters: wyzaj.t_rub2.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_rub2.id = informix.id_rec.id AND
> wyzaj.t_rub2.immune = 'RUB2' )
> NESTED LOOP JOIN
>
> 12) wyzaj.t_tb: INDEX PATH
>
> Filters: wyzaj.t_tb.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_tb.id = informix.id_rec.id AND
> wyzaj.t_tb.immune
> = 'TB' )
> NESTED LOOP JOIN
>
> 13) wyzaj.t_tetnus: INDEX PATH
>
> Filters: wyzaj.t_tetnus.stat IN ('A' , 'W' )
>
> (1) Index Name: informix.immune_key1
>
> Index Keys: id immune (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_tetnus.id = informix.id_rec.id AND
> wyzaj.t_tetnus.immune = 'TD' )
> NESTED LOOP JOIN
>
> 14) wyzaj.t_cpr: INDEX PATH
>
> Filters: wyzaj.t_cpr.stat = 'E'
>
> (1) Index Name: informix.ctc_key2
>
> Index Keys: tick id cmpl_date resrc (Key-First) (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_cpr.id = informix.id_rec.id AND
> wyzaj.t_cpr.tick
> = 'NUR' )
>
> Index Key Filters: (wyzaj.t_cpr.resrc = 'CPRCERT' )
> NESTED LOOP JOIN
>
> 15) wyzaj.t_license: INDEX PATH
>
> Filters: wyzaj.t_license.stat = 'E'
>
> (1) Index Name: informix.ctc_key2
>
> Index Keys: tick id cmpl_date resrc (Key-First) (Serial, fragments: ALL)
>
> Lower Index Filter: (wyzaj.t_license.id = informix.id_rec.id AND
> wyzaj.t_license.tick = 'NUR' )
>
> Index Key Filters: (wyzaj.t_license.resrc = 'LICENSE' )
> NESTED LOOP JOIN
>
> 16) wyzaj.t_clinicals: SEQUENTIAL SCAN
>
> DYNAMIC HASH JOIN
>
> Dynamic Hash Filters: informix.stu_acad_rec.id = wyzaj.t_clinicals.id
>
> 17) informix.health_rec: INDEX PATH
>
> (1) Index Name: informix.health_id
>
> Index Keys: id (Key-Only) (Serial, fragments: ALL)
>
> Lower Index Filter: informix.health_rec.id = informix.id_rec.id
> NESTED LOOP JOIN
>
> 18) wyzaj.t_crimhist: SEQUENTIAL SCAN
>
> DYNAMIC HASH JOIN
>
> Dynamic Hash Filters: wyzaj.t_crimhist.id = informix.id_rec.id
>
> 19) wyzaj.t_handbook: SEQUENTIAL SCAN
>
> DYNAMIC HASH JOIN
>
> Dynamic Hash Filters: wyzaj.t_handbook.id = informix.id_rec.id
>
> 20) wyzaj.t_healthpr: SEQUENTIAL SCAN
>
> DYNAMIC HASH JOIN
>
> Dynamic Hash Filters: wyzaj.t_healthpr.id = informix.id_rec.id
>
> Query statistics:
> -----------------
>
> Table map :
> ----------------------------
> Internal name Table name
> ----------------------------
> t1 major_table
> t2 stu_acad_rec
> t3 prog_enr_rec
> t4 id_rec
> t5 t_meas
> t6 t_has_con
> t7 t_has_con
> t8 t_rub1
> t9 t_cxr
> t10 t_rubt
> t11 t_rube
> t12 t_rub2
> t13 t_tb
> t14 t_tetnus
> t15 t_cpr
> t16 t_license
> t17 t_clinicals
> t18 health_rec
> t19 t_crimhist
> t20 t_handbook
> t21 t_healthpr
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t1 1 1 2 00:00.00 2
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t2 57 69 1018 00:00.00 506
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 70 00:00.00 508
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t3 57 24555 57 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 64 00:00.00 550
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t4 57 340221 57 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 595
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t5 0 58 0 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 634
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t6 0 1 0 00:00.00 0
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t7 0 1 0 00:00.00 0
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 638
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t8 1 58 1 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 676
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t9 2 58 2 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 714
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t10 6 62 6 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 752
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t11 5 62 5 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 791
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t12 1 62 1 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 57 65 00:00.00 829
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t13 49 2578 49 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 59 65 00:00.00 867
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t14 49 3084 49 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 59 65 00:00.00 906
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t15 47 1 205 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 59 65 00:00.00 950
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t16 0 1 205 00:00.00 1
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 59 65 00:00.00 995
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t17 138 138 138 00:00.00 7
>
> type rows_prod est_rows rows_bld rows_prb novrflo time est_cost
>
> ------------------------------------------------------------------------------
> hjoin 59 65 138 59 0 00:00.01 1040
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t18 50 6854 50 00:00.00 0
>
> type rows_prod est_rows time est_cost
> -------------------------------------------------
> nljoin 59 65 00:00.01 1118
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t19 385 385 385 00:00.00 16
>
> type rows_prod est_rows rows_bld rows_prb novrflo time est_cost
>
> ------------------------------------------------------------------------------
> hjoin 59 65 385 59 0 00:00.01 1269
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t20 712 712 712 00:00.00 27
>
> type rows_prod est_rows rows_bld rows_prb novrflo time est_cost
>
> ------------------------------------------------------------------------------
> hjoin 59 65 712 59 0 00:00.01 1522
>
> type table rows_prod est_rows rows_scan time est_cost
> -------------------------------------------------------------------
> scan t21 715 715 715 00:00.00 27
>
> type rows_prod est_rows rows_bld rows_prb novrflo time est_cost
>
> ------------------------------------------------------------------------------
> hjoin 59 65 715 59 0 00:00.01 1776
>
> Jonathon Wyza
> CX & CBORD System Administrator
> CX Programmer/Analyst
> Administrative Computing
> Bethel College
> (574)-257-3381
> AIM: Iamwyza
> jonathon.wyza@bethelcollege.edu
> ==============================
> SLES 11x64 & IDS 11.50.FC6
>
> "Don't document the problem, fix it."
> - Atli Björgvin Oddsson
>
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> JACQUES
> RENAUT
> Sent: Thursday, August 12, 2010 1:19 PM
> To: ids@iiug.org
> Subject: Re: RE: Query works fine in IDS 10 but crashes.... [20895]
>
> Jonathon wrote:
> I'd post what the optimizer has to say, but sadly it never gets that far.
> It
> prints out the last query right before this one (which completes
> successfully)
> and then the oninit process hangs and the optimizer information for the
> query
> in question never gets printed to the file.
>
> Jonathon Wyza
> CX & CBORD System Administrator
> CX Programmer/Analyst
> Administrative Computing
> Bethel College
> (574)-257-3381
> AIM: Iamwyza
> jonathon.wyza@bethelcollege.edu
> ==============================
> SLES 11x64 & IDS 11.50.FC6
>
> "Don't document the problem, fix it."
> - Atli Björgvin Oddsson
>
> Response:
> You could try running the SQL "set optimization low" prior to executing the
> query. As it sounds like the optimizer is doing a too exhaustive search to
> find the best query plan, and so the sqlexec thread appears to be hanging
> because it's doing some sort of looping code over all the possible join
> plans
> for a 30+ table join. I believe it should have some sort of automatic kick
> into low optimization for really big joins so it should not be hanging the
> thread like that so you certainly should contact support and try and get
> them
> table schema to try to reproduce the problem to submit a defect. But in the
> mean time, set optimization low may help the optimizer rule out some join
> plans earlier on so it has less to consider when optimizing. It may or may
> not
> help, but it's probably worth trying.
>
> Jacques Renaut
> IBM Informix Advanced Support
> APD Team
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
>
> *******************************************************************************
> Forum Note: Use "Reply" to post a response in the discussion forum.
>
>
--0015175709609f9f12048da3c13d
Messages In This Thread
IDS Forum is maintained by Administrator with WebBBS 5.12.
|
|
